[PATCH RESEND v3 0/6] staging/rdma/hfi1: Driver cleanup and misc fixes

[PATCH RESEND v3 0/6] staging/rdma/hfi1: Driver cleanup and misc fixes

[Jubin John]

NOTE:
This is being resubmitted to linux-rdma (for patchworks) to aid Doug Ledford in
taking over the staging/rdma subtree.

https://www.mail-archive.com/linux-rdma%40vger.kernel.org/msg30860.html

This patch set is to follow: staging/rdma/hfi1: set Gen3 half-swing for integrated devices

Thanks,
Jubin

These patches were originally part of patch series:
http://driverdev.linuxdriverproject.org/pipermail/driverdev-devel/2015-November/081566.html
but did not apply cleanly to the staging-next branch.

Refreshed the remaining 6 patches against the latest staging-next.

Changes in v2:
	- 01/13: Updated commit message with more information about changes in patch
	- 04/13: Updated patch subject with "hfi1" instead of "hfi"
	- 07/13: Refreshed patch based on new hfi patches in staging-next
	- 12/13: Changed logic based on Dan's suggestions in
		 http://marc.info/?l=linux-driver-devel&m=144723149431368&w=2
Changes in v3:
	- Refreshed remaining 6 patches against latest staging-next

Dean Luick (1):
  staging/rdma/hfi1: Remove unneeded variable index

Harish Chegondi (1):
  staging/rdma/hfi1: Move s_sde to the read mostly portion of the
    hfi1_qp structure

Jubin John (2):
  staging/rdma/hfi1: Use BIT macro
  staging/rdma/hfi1: Change default krcvqs

Mark F. Brown (1):
  staging/rdma/hfi1: change krcvqs module parameter type from byte to
    uint

Vennila Megavannan (1):
  staging/rdma/hfi1: adding per SDMA engine stats to hfistats

 drivers/staging/rdma/hfi1/chip.c           |  127 +++++++++++++++++++++++++---
 drivers/staging/rdma/hfi1/chip.h           |   53 +++++++-----
 drivers/staging/rdma/hfi1/chip_registers.h |    1 +
 drivers/staging/rdma/hfi1/common.h         |    4 +-
 drivers/staging/rdma/hfi1/hfi.h            |   25 +++---
 drivers/staging/rdma/hfi1/init.c           |    6 +-
 drivers/staging/rdma/hfi1/mad.c            |    4 +-
 drivers/staging/rdma/hfi1/qp.h             |    2 +-
 drivers/staging/rdma/hfi1/qsfp.h           |   10 +-
 drivers/staging/rdma/hfi1/sdma.c           |   17 +++-
 drivers/staging/rdma/hfi1/sdma.h           |    7 ++
 drivers/staging/rdma/hfi1/verbs.h          |    2 +-
 12 files changed, 189 insertions(+), 69 deletions(-)

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH RESEND v3 1/6] staging/rdma/hfi1: Use BIT macro

[Jubin John]

This patch fixes the checkpatch issue:
CHECK: Prefer using the BIT macro

Use of BIT macro for HDRQ_INCREMENT in chip.h causes a change in
format specifier for error message in init.c in order to avoid a
build warning.

Reviewed-by: Dean Luick <dean.luick-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Reviewed-by: Ira Weiny <ira.weiny-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Reviewed-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Jubin John <jubin.john-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
Changes in v2:
	- Updated commit message with more information about changes in patch

Changes in v3:
	- Refreshed patch against latest staging-next

 drivers/staging/rdma/hfi1/chip.h   |   48 ++++++++++++++++++------------------
 drivers/staging/rdma/hfi1/common.h |    4 +-
 drivers/staging/rdma/hfi1/hfi.h    |   22 ++++++++--------
 drivers/staging/rdma/hfi1/init.c   |    2 +-
 drivers/staging/rdma/hfi1/mad.c    |    4 +-
 drivers/staging/rdma/hfi1/qp.h     |    2 +-
 drivers/staging/rdma/hfi1/qsfp.h   |   10 +++---
 drivers/staging/rdma/hfi1/sdma.c   |    8 +++---
 8 files changed, 50 insertions(+), 50 deletions(-)

diff --git a/drivers/staging/rdma/hfi1/chip.h b/drivers/staging/rdma/hfi1/chip.h
index 5b375dd..1368a44 100644
--- a/drivers/staging/rdma/hfi1/chip.h
+++ b/drivers/staging/rdma/hfi1/chip.h
@@ -242,18 +242,18 @@
 #define HCMD_SUCCESS 2
 
 /* DC_DC8051_DBG_ERR_INFO_SET_BY_8051.ERROR - error flags */
-#define SPICO_ROM_FAILED		    (1 <<  0)
-#define UNKNOWN_FRAME		    (1 <<  1)
-#define TARGET_BER_NOT_MET		    (1 <<  2)
-#define FAILED_SERDES_INTERNAL_LOOPBACK (1 <<  3)
-#define FAILED_SERDES_INIT		    (1 <<  4)
-#define FAILED_LNI_POLLING		    (1 <<  5)
-#define FAILED_LNI_DEBOUNCE		    (1 <<  6)
-#define FAILED_LNI_ESTBCOMM		    (1 <<  7)
-#define FAILED_LNI_OPTEQ		    (1 <<  8)
-#define FAILED_LNI_VERIFY_CAP1	    (1 <<  9)
-#define FAILED_LNI_VERIFY_CAP2	    (1 << 10)
-#define FAILED_LNI_CONFIGLT		    (1 << 11)
+#define SPICO_ROM_FAILED		BIT(0)
+#define UNKNOWN_FRAME			BIT(1)
+#define TARGET_BER_NOT_MET		BIT(2)
+#define FAILED_SERDES_INTERNAL_LOOPBACK	BIT(3)
+#define FAILED_SERDES_INIT		BIT(4)
+#define FAILED_LNI_POLLING		BIT(5)
+#define FAILED_LNI_DEBOUNCE		BIT(6)
+#define FAILED_LNI_ESTBCOMM		BIT(7)
+#define FAILED_LNI_OPTEQ		BIT(8)
+#define FAILED_LNI_VERIFY_CAP1		BIT(9)
+#define FAILED_LNI_VERIFY_CAP2		BIT(10)
+#define FAILED_LNI_CONFIGLT		BIT(11)
 
 #define FAILED_LNI (FAILED_LNI_POLLING | FAILED_LNI_DEBOUNCE \
 			| FAILED_LNI_ESTBCOMM | FAILED_LNI_OPTEQ \
@@ -262,16 +262,16 @@
 			| FAILED_LNI_CONFIGLT)
 
 /* DC_DC8051_DBG_ERR_INFO_SET_BY_8051.HOST_MSG - host message flags */
-#define HOST_REQ_DONE	   (1 << 0)
-#define BC_PWR_MGM_MSG	   (1 << 1)
-#define BC_SMA_MSG		   (1 << 2)
-#define BC_BCC_UNKOWN_MSG	   (1 << 3)
-#define BC_IDLE_UNKNOWN_MSG	   (1 << 4)
-#define EXT_DEVICE_CFG_REQ	   (1 << 5)
-#define VERIFY_CAP_FRAME	   (1 << 6)
-#define LINKUP_ACHIEVED	   (1 << 7)
-#define LINK_GOING_DOWN	   (1 << 8)
-#define LINK_WIDTH_DOWNGRADED  (1 << 9)
+#define HOST_REQ_DONE		BIT(0)
+#define BC_PWR_MGM_MSG		BIT(1)
+#define BC_SMA_MSG		BIT(2)
+#define BC_BCC_UNKNOWN_MSG	BIT(3)
+#define BC_IDLE_UNKNOWN_MSG	BIT(4)
+#define EXT_DEVICE_CFG_REQ	BIT(5)
+#define VERIFY_CAP_FRAME	BIT(6)
+#define LINKUP_ACHIEVED		BIT(7)
+#define LINK_GOING_DOWN		BIT(8)
+#define LINK_WIDTH_DOWNGRADED	BIT(9)
 
 /* DC_DC8051_CFG_EXT_DEV_1.REQ_TYPE - 8051 host requests */
 #define HREQ_LOAD_CONFIG	0x01
@@ -335,14 +335,14 @@
  * the CSR fields hold multiples of this value.
  */
 #define RCV_SHIFT 3
-#define RCV_INCREMENT (1 << RCV_SHIFT)
+#define RCV_INCREMENT BIT(RCV_SHIFT)
 
 /*
  * Receive header queue entry increment - the CSR holds multiples of
  * this value.
  */
 #define HDRQ_SIZE_SHIFT 5
-#define HDRQ_INCREMENT (1 << HDRQ_SIZE_SHIFT)
+#define HDRQ_INCREMENT BIT(HDRQ_SIZE_SHIFT)
 
 /*
  * Freeze handling flags
diff --git a/drivers/staging/rdma/hfi1/common.h b/drivers/staging/rdma/hfi1/common.h
index 5dd9272..e4b1dc6 100644
--- a/drivers/staging/rdma/hfi1/common.h
+++ b/drivers/staging/rdma/hfi1/common.h
@@ -349,10 +349,10 @@ struct hfi1_message_header {
 #define HFI1_QPN_MASK 0xFFFFFF
 #define HFI1_FECN_SHIFT 31
 #define HFI1_FECN_MASK 1
-#define HFI1_FECN_SMASK (1 << HFI1_FECN_SHIFT)
+#define HFI1_FECN_SMASK BIT(HFI1_FECN_SHIFT)
 #define HFI1_BECN_SHIFT 30
 #define HFI1_BECN_MASK 1
-#define HFI1_BECN_SMASK (1 << HFI1_BECN_SHIFT)
+#define HFI1_BECN_SMASK BIT(HFI1_BECN_SHIFT)
 #define HFI1_MULTICAST_LID_BASE 0xC000
 
 static inline __u64 rhf_to_cpu(const __le32 *rbuf)
diff --git a/drivers/staging/rdma/hfi1/hfi.h b/drivers/staging/rdma/hfi1/hfi.h
index 2611bb2..2dfd402 100644
--- a/drivers/staging/rdma/hfi1/hfi.h
+++ b/drivers/staging/rdma/hfi1/hfi.h
@@ -424,17 +424,17 @@ struct hfi1_sge_state;
 #define __HLS_GOING_OFFLINE_BP  9
 #define __HLS_LINK_COOLDOWN_BP 10
 
-#define HLS_UP_INIT	  (1 << __HLS_UP_INIT_BP)
-#define HLS_UP_ARMED	  (1 << __HLS_UP_ARMED_BP)
-#define HLS_UP_ACTIVE	  (1 << __HLS_UP_ACTIVE_BP)
-#define HLS_DN_DOWNDEF	  (1 << __HLS_DN_DOWNDEF_BP) /* link down default */
-#define HLS_DN_POLL	  (1 << __HLS_DN_POLL_BP)
-#define HLS_DN_DISABLE	  (1 << __HLS_DN_DISABLE_BP)
-#define HLS_DN_OFFLINE	  (1 << __HLS_DN_OFFLINE_BP)
-#define HLS_VERIFY_CAP	  (1 << __HLS_VERIFY_CAP_BP)
-#define HLS_GOING_UP	  (1 << __HLS_GOING_UP_BP)
-#define HLS_GOING_OFFLINE (1 << __HLS_GOING_OFFLINE_BP)
-#define HLS_LINK_COOLDOWN (1 << __HLS_LINK_COOLDOWN_BP)
+#define HLS_UP_INIT	  BIT(__HLS_UP_INIT_BP)
+#define HLS_UP_ARMED	  BIT(__HLS_UP_ARMED_BP)
+#define HLS_UP_ACTIVE	  BIT(__HLS_UP_ACTIVE_BP)
+#define HLS_DN_DOWNDEF	  BIT(__HLS_DN_DOWNDEF_BP) /* link down default */
+#define HLS_DN_POLL	  BIT(__HLS_DN_POLL_BP)
+#define HLS_DN_DISABLE	  BIT(__HLS_DN_DISABLE_BP)
+#define HLS_DN_OFFLINE	  BIT(__HLS_DN_OFFLINE_BP)
+#define HLS_VERIFY_CAP	  BIT(__HLS_VERIFY_CAP_BP)
+#define HLS_GOING_UP	  BIT(__HLS_GOING_UP_BP)
+#define HLS_GOING_OFFLINE BIT(__HLS_GOING_OFFLINE_BP)
+#define HLS_LINK_COOLDOWN BIT(__HLS_LINK_COOLDOWN_BP)
 
 #define HLS_UP (HLS_UP_INIT | HLS_UP_ARMED | HLS_UP_ACTIVE)
 
diff --git a/drivers/staging/rdma/hfi1/init.c b/drivers/staging/rdma/hfi1/init.c
index 4dd8051..35b5e41 100644
--- a/drivers/staging/rdma/hfi1/init.c
+++ b/drivers/staging/rdma/hfi1/init.c
@@ -260,7 +260,7 @@ struct hfi1_ctxtdata *hfi1_create_ctxtdata(struct hfi1_pportdata *ppd, u32 ctxt)
 		/* Validate and initialize Rcv Hdr Q variables */
 		if (rcvhdrcnt % HDRQ_INCREMENT) {
 			dd_dev_err(dd,
-				   "ctxt%u: header queue count %d must be divisible by %d\n",
+				   "ctxt%u: header queue count %d must be divisible by %lu\n",
 				   rcd->ctxt, rcvhdrcnt, HDRQ_INCREMENT);
 			goto bail;
 		}
diff --git a/drivers/staging/rdma/hfi1/mad.c b/drivers/staging/rdma/hfi1/mad.c
index 4f5dbd1..eeb8687 100644
--- a/drivers/staging/rdma/hfi1/mad.c
+++ b/drivers/staging/rdma/hfi1/mad.c
@@ -1782,7 +1782,7 @@ static int __subn_get_opa_cable_info(struct opa_smp *smp, u32 am, u8 *data,
 	u32 len = OPA_AM_CI_LEN(am) + 1;
 	int ret;
 
-#define __CI_PAGE_SIZE (1 << 7) /* 128 bytes */
+#define __CI_PAGE_SIZE BIT(7) /* 128 bytes */
 #define __CI_PAGE_MASK ~(__CI_PAGE_SIZE - 1)
 #define __CI_PAGE_NUM(a) ((a) & __CI_PAGE_MASK)
 
@@ -3402,7 +3402,7 @@ struct opa_led_info {
 };
 
 #define OPA_LED_SHIFT	31
-#define OPA_LED_MASK	(1 << OPA_LED_SHIFT)
+#define OPA_LED_MASK	BIT(OPA_LED_SHIFT)
 
 static int __subn_get_opa_led_info(struct opa_smp *smp, u32 am, u8 *data,
 				   struct ib_device *ibdev, u8 port,
diff --git a/drivers/staging/rdma/hfi1/qp.h b/drivers/staging/rdma/hfi1/qp.h
index 62a94c5..19b16a9 100644
--- a/drivers/staging/rdma/hfi1/qp.h
+++ b/drivers/staging/rdma/hfi1/qp.h
@@ -54,7 +54,7 @@
 #include "verbs.h"
 #include "sdma.h"
 
-#define QPN_MAX                 (1 << 24)
+#define QPN_MAX                 BIT(24)
 #define QPNMAP_ENTRIES          (QPN_MAX / PAGE_SIZE / BITS_PER_BYTE)
 
 /*
diff --git a/drivers/staging/rdma/hfi1/qsfp.h b/drivers/staging/rdma/hfi1/qsfp.h
index d30c2a6..16aebdc 100644
--- a/drivers/staging/rdma/hfi1/qsfp.h
+++ b/drivers/staging/rdma/hfi1/qsfp.h
@@ -59,11 +59,11 @@
  * Below are masks for QSFP pins.  Pins are the same for HFI0 and HFI1.
  * _N means asserted low
  */
-#define QSFP_HFI0_I2CCLK    (1 << 0)
-#define QSFP_HFI0_I2CDAT    (1 << 1)
-#define QSFP_HFI0_RESET_N   (1 << 2)
-#define QSFP_HFI0_INT_N	    (1 << 3)
-#define QSFP_HFI0_MODPRST_N (1 << 4)
+#define QSFP_HFI0_I2CCLK    BIT(0)
+#define QSFP_HFI0_I2CDAT    BIT(1)
+#define QSFP_HFI0_RESET_N   BIT(2)
+#define QSFP_HFI0_INT_N	    BIT(3)
+#define QSFP_HFI0_MODPRST_N BIT(4)
 
 /* QSFP is paged at 256 bytes */
 #define QSFP_PAGESIZE 256
diff --git a/drivers/staging/rdma/hfi1/sdma.c b/drivers/staging/rdma/hfi1/sdma.c
index 9a15f1f..1d38be5 100644
--- a/drivers/staging/rdma/hfi1/sdma.c
+++ b/drivers/staging/rdma/hfi1/sdma.c
@@ -112,10 +112,10 @@ MODULE_PARM_DESC(desct_intr, "Number of SDMA descriptor before interrupt");
 	| SD(ENG_ERR_STATUS_SDMA_HEADER_REQUEST_FIFO_UNC_ERR_SMASK))
 
 /* sdma_sendctrl operations */
-#define SDMA_SENDCTRL_OP_ENABLE    (1U << 0)
-#define SDMA_SENDCTRL_OP_INTENABLE (1U << 1)
-#define SDMA_SENDCTRL_OP_HALT      (1U << 2)
-#define SDMA_SENDCTRL_OP_CLEANUP   (1U << 3)
+#define SDMA_SENDCTRL_OP_ENABLE    BIT(0)
+#define SDMA_SENDCTRL_OP_INTENABLE BIT(1)
+#define SDMA_SENDCTRL_OP_HALT      BIT(2)
+#define SDMA_SENDCTRL_OP_CLEANUP   BIT(3)
 
 /* handle long defines */
 #define SDMA_EGRESS_PACKET_OCCUPANCY_SMASK \
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH RESEND v3 2/6] staging/rdma/hfi1: Move s_sde to the read mostly portion of the hfi1_qp structure

[Jubin John]

From: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

This would reduce L2 cache misses on s_sde in the _hfi1_schedule_send
function when invoked from post_send thereby improving performance of
post_send.

Reviewed-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Harish Chegondi <harish.chegondi-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Jubin John <jubin.john-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
Changes in v2:
	- None

Changes in v3:
	- Refreshed patch against latest staging-next

 drivers/staging/rdma/hfi1/verbs.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/staging/rdma/hfi1/verbs.h b/drivers/staging/rdma/hfi1/verbs.h
index 72106e5..d22db39 100644
--- a/drivers/staging/rdma/hfi1/verbs.h
+++ b/drivers/staging/rdma/hfi1/verbs.h
@@ -441,6 +441,7 @@ struct hfi1_qp {
 	struct hfi1_swqe *s_wq;  /* send work queue */
 	struct hfi1_mmap_info *ip;
 	struct ahg_ib_header *s_hdr;     /* next packet header to send */
+	struct sdma_engine *s_sde; /* current sde */
 	/* sc for UC/RC QPs - based on ah for UD */
 	u8 s_sc;
 	unsigned long timeout_jiffies;  /* computed from timeout */
@@ -506,7 +507,6 @@ struct hfi1_qp {
 	struct hfi1_swqe *s_wqe;
 	struct hfi1_sge_state s_sge;     /* current send request data */
 	struct hfi1_mregion *s_rdma_mr;
-	struct sdma_engine *s_sde; /* current sde */
 	u32 s_cur_size;         /* size of send packet in bytes */
 	u32 s_len;              /* total length of s_sge */
 	u32 s_rdma_read_len;    /* total length of s_rdma_read_sge */
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH RESEND v3 3/6] staging/rdma/hfi1: change krcvqs module parameter type from byte to uint

[Jubin John]

From: Mark F. Brown <mark.f.brown-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

The krcvqs parameter is displayed incorrectly in sysfs.
The workaround is to set the param type as uint.

Reviewed-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Reviewed-by: Mitko Haralanov <mitko.haralanov-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Mark F. Brown <mark.f.brown-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Jubin John <jubin.john-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
Changes in v2:
	- None

Changes in v3:
	- Refreshed patch against latest staging-next

 drivers/staging/rdma/hfi1/hfi.h  |    2 +-
 drivers/staging/rdma/hfi1/init.c |    4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/staging/rdma/hfi1/hfi.h b/drivers/staging/rdma/hfi1/hfi.h
index 2dfd402..d4a859f 100644
--- a/drivers/staging/rdma/hfi1/hfi.h
+++ b/drivers/staging/rdma/hfi1/hfi.h
@@ -1667,7 +1667,7 @@ extern unsigned int hfi1_cu;
 extern unsigned int user_credit_return_threshold;
 extern int num_user_contexts;
 extern unsigned n_krcvqs;
-extern u8 krcvqs[];
+extern uint krcvqs[];
 extern int krcvqsset;
 extern uint kdeth_qp;
 extern uint loopback;
diff --git a/drivers/staging/rdma/hfi1/init.c b/drivers/staging/rdma/hfi1/init.c
index 35b5e41..dbdc631 100644
--- a/drivers/staging/rdma/hfi1/init.c
+++ b/drivers/staging/rdma/hfi1/init.c
@@ -87,9 +87,9 @@ module_param_named(num_user_contexts, num_user_contexts, uint, S_IRUGO);
 MODULE_PARM_DESC(
 	num_user_contexts, "Set max number of user contexts to use");
 
-u8 krcvqs[RXE_NUM_DATA_VL];
+uint krcvqs[RXE_NUM_DATA_VL];
 int krcvqsset;
-module_param_array(krcvqs, byte, &krcvqsset, S_IRUGO);
+module_param_array(krcvqs, uint, &krcvqsset, S_IRUGO);
 MODULE_PARM_DESC(krcvqs, "Array of the number of non-control kernel receive queues by VL");
 
 /* computed based on above array */
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH RESEND v3 4/6] staging/rdma/hfi1: Change default krcvqs

[Jubin John]

Change the default number of krcvqs to number of numa nodes + 1
based on the performance data collected.

Reviewed-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
Signed-off-by: Jubin John <jubin.john@intel.com>
---
Changes in v2:
	- None

Changes in v3:
	- Refreshed patch against latest staging-next

 drivers/staging/rdma/hfi1/chip.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/drivers/staging/rdma/hfi1/chip.c b/drivers/staging/rdma/hfi1/chip.c
index bbe5ad8..503bfca 100644
--- a/drivers/staging/rdma/hfi1/chip.c
+++ b/drivers/staging/rdma/hfi1/chip.c
@@ -12445,7 +12445,7 @@ static int set_up_context_variables(struct hfi1_devdata *dd)
 		 */
 		num_kernel_contexts = n_krcvqs + MIN_KERNEL_KCTXTS - 1;
 	else
-		num_kernel_contexts = num_online_nodes();
+		num_kernel_contexts = num_online_nodes() + 1;
 	num_kernel_contexts =
 		max_t(int, MIN_KERNEL_KCTXTS, num_kernel_contexts);
 	/*
-- 
1.7.1

[PATCH RESEND v3 5/6] staging/rdma/hfi1: adding per SDMA engine stats to hfistats

[Jubin John]

From: Vennila Megavannan <vennila.megavannan-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

Added the following per sdma engine stats:
  - SendDmaDescFetchedCnt
  - software maintained count of SDMA interrupts
	 (SDmaInt, SDmaIdleInt, SDmaProgressInt)
  - software maintained counts of SDMA error cases

Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Vennila Megavannan <vennila.megavannan-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Jubin John <jubin.john-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
Changes in v2:
	- None

Changes in v3:
	- Refreshed patch against latest staging-next

 drivers/staging/rdma/hfi1/chip.c           |  110 +++++++++++++++++++++++++++-
 drivers/staging/rdma/hfi1/chip.h           |    5 +
 drivers/staging/rdma/hfi1/chip_registers.h |    1 +
 drivers/staging/rdma/hfi1/hfi.h            |    1 +
 drivers/staging/rdma/hfi1/sdma.c           |    9 ++-
 drivers/staging/rdma/hfi1/sdma.h           |    7 ++
 6 files changed, 129 insertions(+), 4 deletions(-)

diff --git a/drivers/staging/rdma/hfi1/chip.c b/drivers/staging/rdma/hfi1/chip.c
index 503bfca..f4f720d 100644
--- a/drivers/staging/rdma/hfi1/chip.c
+++ b/drivers/staging/rdma/hfi1/chip.c
@@ -1297,10 +1297,58 @@ static u64 dev_access_u32_csr(const struct cntr_entry *entry,
 			    void *context, int vl, int mode, u64 data)
 {
 	struct hfi1_devdata *dd = context;
+	u64 csr = entry->csr;
 
-	if (vl != CNTR_INVALID_VL)
-		return 0;
-	return read_write_csr(dd, entry->csr, mode, data);
+	if (entry->flags & CNTR_SDMA) {
+		if (vl == CNTR_INVALID_VL)
+			return 0;
+		csr += 0x100 * vl;
+	} else {
+		if (vl != CNTR_INVALID_VL)
+			return 0;
+	}
+	return read_write_csr(dd, csr, mode, data);
+}
+
+static u64 access_sde_err_cnt(const struct cntr_entry *entry,
+			      void *context, int idx, int mode, u64 data)
+{
+	struct hfi1_devdata *dd = (struct hfi1_devdata *)context;
+
+	if (dd->per_sdma && idx < dd->num_sdma)
+		return dd->per_sdma[idx].err_cnt;
+	return 0;
+}
+
+static u64 access_sde_int_cnt(const struct cntr_entry *entry,
+			      void *context, int idx, int mode, u64 data)
+{
+	struct hfi1_devdata *dd = (struct hfi1_devdata *)context;
+
+	if (dd->per_sdma && idx < dd->num_sdma)
+		return dd->per_sdma[idx].sdma_int_cnt;
+	return 0;
+}
+
+static u64 access_sde_idle_int_cnt(const struct cntr_entry *entry,
+				   void *context, int idx, int mode, u64 data)
+{
+	struct hfi1_devdata *dd = (struct hfi1_devdata *)context;
+
+	if (dd->per_sdma && idx < dd->num_sdma)
+		return dd->per_sdma[idx].idle_int_cnt;
+	return 0;
+}
+
+static u64 access_sde_progress_int_cnt(const struct cntr_entry *entry,
+				       void *context, int idx, int mode,
+				       u64 data)
+{
+	struct hfi1_devdata *dd = (struct hfi1_devdata *)context;
+
+	if (dd->per_sdma && idx < dd->num_sdma)
+		return dd->per_sdma[idx].progress_int_cnt;
+	return 0;
 }
 
 static u64 dev_access_u64_csr(const struct cntr_entry *entry, void *context,
@@ -4070,6 +4118,22 @@ static struct cntr_entry dev_cntrs[DEV_CNTR_LAST] = {
 			    access_sw_kmem_wait),
 [C_SW_SEND_SCHED] = CNTR_ELEM("SendSched", 0, 0, CNTR_NORMAL,
 			    access_sw_send_schedule),
+[C_SDMA_DESC_FETCHED_CNT] = CNTR_ELEM("SDEDscFdCn",
+				      SEND_DMA_DESC_FETCHED_CNT, 0,
+				      CNTR_NORMAL | CNTR_32BIT | CNTR_SDMA,
+				      dev_access_u32_csr),
+[C_SDMA_INT_CNT] = CNTR_ELEM("SDMAInt", 0, 0,
+			     CNTR_NORMAL | CNTR_32BIT | CNTR_SDMA,
+			     access_sde_int_cnt),
+[C_SDMA_ERR_CNT] = CNTR_ELEM("SDMAErrCt", 0, 0,
+			     CNTR_NORMAL | CNTR_32BIT | CNTR_SDMA,
+			     access_sde_err_cnt),
+[C_SDMA_IDLE_INT_CNT] = CNTR_ELEM("SDMAIdInt", 0, 0,
+				  CNTR_NORMAL | CNTR_32BIT | CNTR_SDMA,
+				  access_sde_idle_int_cnt),
+[C_SDMA_PROGRESS_INT_CNT] = CNTR_ELEM("SDMAPrIntCn", 0, 0,
+				      CNTR_NORMAL | CNTR_32BIT | CNTR_SDMA,
+				      access_sde_progress_int_cnt),
 /* MISC_ERR_STATUS */
 [C_MISC_PLL_LOCK_FAIL_ERR] = CNTR_ELEM("MISC_PLL_LOCK_FAIL_ERR", 0, 0,
 				CNTR_NORMAL,
@@ -5707,6 +5771,7 @@ static void handle_sdma_eng_err(struct hfi1_devdata *dd,
 	dd_dev_err(sde->dd, "CONFIG SDMA(%u) source: %u status 0x%llx\n",
 		   sde->this_idx, source, (unsigned long long)status);
 #endif
+	sde->err_cnt++;
 	sdma_engine_error(sde, status);
 
 	/*
@@ -11150,6 +11215,20 @@ u32 hfi1_read_cntrs(struct hfi1_devdata *dd, loff_t pos, char **namep,
 						dd->cntrs[entry->offset + j] =
 									    val;
 					}
+				} else if (entry->flags & CNTR_SDMA) {
+					hfi1_cdbg(CNTR,
+						  "\t Per SDMA Engine\n");
+					for (j = 0; j < dd->chip_sdma_engines;
+					     j++) {
+						val =
+						entry->rw_cntr(entry, dd, j,
+							       CNTR_MODE_R, 0);
+						hfi1_cdbg(CNTR,
+							  "\t\tRead 0x%llx for %d\n",
+							  val, j);
+						dd->cntrs[entry->offset + j] =
+									val;
+					}
 				} else {
 					val = entry->rw_cntr(entry, dd,
 							CNTR_INVALID_VL,
@@ -11553,6 +11632,21 @@ static int init_cntrs(struct hfi1_devdata *dd)
 				dd->ndevcntrs++;
 				index++;
 			}
+		} else if (dev_cntrs[i].flags & CNTR_SDMA) {
+			hfi1_dbg_early(
+				       "\tProcessing per SDE counters chip enginers %u\n",
+				       dd->chip_sdma_engines);
+			dev_cntrs[i].offset = index;
+			for (j = 0; j < dd->chip_sdma_engines; j++) {
+				memset(name, '\0', C_MAX_NAME);
+				snprintf(name, C_MAX_NAME, "%s%d",
+					 dev_cntrs[i].name, j);
+				sz += strlen(name);
+				sz++;
+				hfi1_dbg_early("\t\t%s\n", name);
+				dd->ndevcntrs++;
+				index++;
+			}
 		} else {
 			/* +1 for newline  */
 			sz += strlen(dev_cntrs[i].name) + 1;
@@ -11594,6 +11688,16 @@ static int init_cntrs(struct hfi1_devdata *dd)
 					p += strlen(name);
 					*p++ = '\n';
 				}
+			} else if (dev_cntrs[i].flags & CNTR_SDMA) {
+				for (j = 0; j < TXE_NUM_SDMA_ENGINES;
+				     j++) {
+					memset(name, '\0', C_MAX_NAME);
+					snprintf(name, C_MAX_NAME, "%s%d",
+						 dev_cntrs[i].name, j);
+					memcpy(p, name, strlen(name));
+					p += strlen(name);
+					*p++ = '\n';
+				}
 			} else {
 				memcpy(p, dev_cntrs[i].name,
 				       strlen(dev_cntrs[i].name));
diff --git a/drivers/staging/rdma/hfi1/chip.h b/drivers/staging/rdma/hfi1/chip.h
index 1368a44..b46ef66 100644
--- a/drivers/staging/rdma/hfi1/chip.h
+++ b/drivers/staging/rdma/hfi1/chip.h
@@ -787,6 +787,11 @@ enum {
 	C_SW_PIO_WAIT,
 	C_SW_KMEM_WAIT,
 	C_SW_SEND_SCHED,
+	C_SDMA_DESC_FETCHED_CNT,
+	C_SDMA_INT_CNT,
+	C_SDMA_ERR_CNT,
+	C_SDMA_IDLE_INT_CNT,
+	C_SDMA_PROGRESS_INT_CNT,
 /* MISC_ERR_STATUS */
 	C_MISC_PLL_LOCK_FAIL_ERR,
 	C_MISC_MBIST_FAIL_ERR,
diff --git a/drivers/staging/rdma/hfi1/chip_registers.h b/drivers/staging/rdma/hfi1/chip_registers.h
index 701e9e1..fb36541 100644
--- a/drivers/staging/rdma/hfi1/chip_registers.h
+++ b/drivers/staging/rdma/hfi1/chip_registers.h
@@ -1290,5 +1290,6 @@
 #define CCE_INT_BLOCKED (CCE + 0x000000110C00)
 #define SEND_DMA_IDLE_CNT (TXE + 0x000000200040)
 #define SEND_DMA_DESC_FETCHED_CNT (TXE + 0x000000200058)
+#define CCE_MSIX_PBA_OFFSET 0X0110000
 
 #endif          /* DEF_CHIP_REG */
diff --git a/drivers/staging/rdma/hfi1/hfi.h b/drivers/staging/rdma/hfi1/hfi.h
index d4a859f..d32ee84 100644
--- a/drivers/staging/rdma/hfi1/hfi.h
+++ b/drivers/staging/rdma/hfi1/hfi.h
@@ -490,6 +490,7 @@ struct hfi1_sge_state;
 #define CNTR_DISABLED		0x2 /* Disable this counter */
 #define CNTR_32BIT		0x4 /* Simulate 64 bits for this counter */
 #define CNTR_VL			0x8 /* Per VL counter */
+#define CNTR_SDMA              0x10
 #define CNTR_INVALID_VL		-1  /* Specifies invalid VL */
 #define CNTR_MODE_W		0x0
 #define CNTR_MODE_R		0x1
diff --git a/drivers/staging/rdma/hfi1/sdma.c b/drivers/staging/rdma/hfi1/sdma.c
index 1d38be5..cb66bd0 100644
--- a/drivers/staging/rdma/hfi1/sdma.c
+++ b/drivers/staging/rdma/hfi1/sdma.c
@@ -1061,7 +1061,6 @@ int sdma_init(struct hfi1_devdata *dd, u8 port)
 		sde->desc_avail = sdma_descq_freecnt(sde);
 		sde->sdma_shift = ilog2(descq_cnt);
 		sde->sdma_mask = (1 << sde->sdma_shift) - 1;
-		sde->descq_full_count = 0;
 
 		/* Create a mask for all 3 chip interrupt sources */
 		sde->imask = (u64)1 << (0*TXE_NUM_SDMA_ENGINES + this_idx)
@@ -1073,6 +1072,8 @@ int sdma_init(struct hfi1_devdata *dd, u8 port)
 		/* Create a mask specifically for sdma_progress */
 		sde->progress_mask =
 			(u64)1 << (TXE_NUM_SDMA_ENGINES + this_idx);
+		sde->int_mask =
+			(u64)1 << (0 * TXE_NUM_SDMA_ENGINES + this_idx);
 		spin_lock_init(&sde->tail_lock);
 		seqlock_init(&sde->head_lock);
 		spin_lock_init(&sde->senddmactrl_lock);
@@ -1552,6 +1553,12 @@ void sdma_engine_interrupt(struct sdma_engine *sde, u64 status)
 	trace_hfi1_sdma_engine_interrupt(sde, status);
 	write_seqlock(&sde->head_lock);
 	sdma_set_desc_cnt(sde, sdma_desct_intr);
+	if (status & sde->idle_mask)
+		sde->idle_int_cnt++;
+	else if (status & sde->progress_mask)
+		sde->progress_int_cnt++;
+	else if (status & sde->int_mask)
+		sde->sdma_int_cnt++;
 	sdma_make_progress(sde, status);
 	write_sequnlock(&sde->head_lock);
 }
diff --git a/drivers/staging/rdma/hfi1/sdma.h b/drivers/staging/rdma/hfi1/sdma.h
index da89e64..757017a 100644
--- a/drivers/staging/rdma/hfi1/sdma.h
+++ b/drivers/staging/rdma/hfi1/sdma.h
@@ -409,6 +409,7 @@ struct sdma_engine {
 	u64 imask;			/* clear interrupt mask */
 	u64 idle_mask;
 	u64 progress_mask;
+	u64 int_mask;
 	/* private: */
 	volatile __le64      *head_dma; /* DMA'ed by chip */
 	/* private: */
@@ -465,6 +466,12 @@ struct sdma_engine {
 	u16                   tx_head;
 	/* private: */
 	u64                   last_status;
+	/* private */
+	u64                     err_cnt;
+	/* private */
+	u64                     sdma_int_cnt;
+	u64                     idle_int_cnt;
+	u64                     progress_int_cnt;
 
 	/* private: */
 	struct list_head      dmawait;
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH RESEND v3 6/6] staging/rdma/hfi1: Remove unneeded variable index

[Jubin John]

From: Dean Luick <dean.luick-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>

The variable "index" increments the same as dd->ndevcntrs.
Just use the later.  Remove uneeded usage of "index" in the
fill loop - it is not used there or later in the function.

Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Dean Luick <dean.luick-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Signed-off-by: Jubin John <jubin.john-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
Changes in v2:
	- None

Changes in v3:
	- Refreshed patch against latest staging-next

 drivers/staging/rdma/hfi1/chip.c |   19 +++++++------------
 1 files changed, 7 insertions(+), 12 deletions(-)

diff --git a/drivers/staging/rdma/hfi1/chip.c b/drivers/staging/rdma/hfi1/chip.c
index f4f720d..1109049 100644
--- a/drivers/staging/rdma/hfi1/chip.c
+++ b/drivers/staging/rdma/hfi1/chip.c
@@ -11592,7 +11592,7 @@ mod_timer(&dd->synth_stats_timer, jiffies + HZ * SYNTH_CNT_TIME);
 #define C_MAX_NAME 13 /* 12 chars + one for /0 */
 static int init_cntrs(struct hfi1_devdata *dd)
 {
-	int i, rcv_ctxts, index, j;
+	int i, rcv_ctxts, j;
 	size_t sz;
 	char *p;
 	char name[C_MAX_NAME];
@@ -11609,7 +11609,6 @@ static int init_cntrs(struct hfi1_devdata *dd)
 	/* size names and determine how many we have*/
 	dd->ndevcntrs = 0;
 	sz = 0;
-	index = 0;
 
 	for (i = 0; i < DEV_CNTR_LAST; i++) {
 		hfi1_dbg_early("Init cntr %s\n", dev_cntrs[i].name);
@@ -11620,7 +11619,7 @@ static int init_cntrs(struct hfi1_devdata *dd)
 
 		if (dev_cntrs[i].flags & CNTR_VL) {
 			hfi1_dbg_early("\tProcessing VL cntr\n");
-			dev_cntrs[i].offset = index;
+			dev_cntrs[i].offset = dd->ndevcntrs;
 			for (j = 0; j < C_VL_COUNT; j++) {
 				memset(name, '\0', C_MAX_NAME);
 				snprintf(name, C_MAX_NAME, "%s%d",
@@ -11630,13 +11629,12 @@ static int init_cntrs(struct hfi1_devdata *dd)
 				sz++;
 				hfi1_dbg_early("\t\t%s\n", name);
 				dd->ndevcntrs++;
-				index++;
 			}
 		} else if (dev_cntrs[i].flags & CNTR_SDMA) {
 			hfi1_dbg_early(
 				       "\tProcessing per SDE counters chip enginers %u\n",
 				       dd->chip_sdma_engines);
-			dev_cntrs[i].offset = index;
+			dev_cntrs[i].offset = dd->ndevcntrs;
 			for (j = 0; j < dd->chip_sdma_engines; j++) {
 				memset(name, '\0', C_MAX_NAME);
 				snprintf(name, C_MAX_NAME, "%s%d",
@@ -11645,24 +11643,22 @@ static int init_cntrs(struct hfi1_devdata *dd)
 				sz++;
 				hfi1_dbg_early("\t\t%s\n", name);
 				dd->ndevcntrs++;
-				index++;
 			}
 		} else {
 			/* +1 for newline  */
 			sz += strlen(dev_cntrs[i].name) + 1;
+			dev_cntrs[i].offset = dd->ndevcntrs;
 			dd->ndevcntrs++;
-			dev_cntrs[i].offset = index;
-			index++;
 			hfi1_dbg_early("\tAdding %s\n", dev_cntrs[i].name);
 		}
 	}
 
 	/* allocate space for the counter values */
-	dd->cntrs = kcalloc(index, sizeof(u64), GFP_KERNEL);
+	dd->cntrs = kcalloc(dd->ndevcntrs, sizeof(u64), GFP_KERNEL);
 	if (!dd->cntrs)
 		goto bail;
 
-	dd->scntrs = kcalloc(index, sizeof(u64), GFP_KERNEL);
+	dd->scntrs = kcalloc(dd->ndevcntrs, sizeof(u64), GFP_KERNEL);
 	if (!dd->scntrs)
 		goto bail;
 
@@ -11674,7 +11670,7 @@ static int init_cntrs(struct hfi1_devdata *dd)
 		goto bail;
 
 	/* fill in the names */
-	for (p = dd->cntrnames, i = 0, index = 0; i < DEV_CNTR_LAST; i++) {
+	for (p = dd->cntrnames, i = 0; i < DEV_CNTR_LAST; i++) {
 		if (dev_cntrs[i].flags & CNTR_DISABLED) {
 			/* Nothing */
 		} else {
@@ -11704,7 +11700,6 @@ static int init_cntrs(struct hfi1_devdata *dd)
 				p += strlen(dev_cntrs[i].name);
 				*p++ = '\n';
 			}
-			index++;
 		}
 	}
 
-- 
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RESEND v3 5/6] staging/rdma/hfi1: adding per SDMA engine stats to hfistats

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 10936 bytes --]

On 01/11/2016 06:30 PM, Jubin John wrote:
> From: Vennila Megavannan <vennila.megavannan-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> 
> Added the following per sdma engine stats:
>   - SendDmaDescFetchedCnt
>   - software maintained count of SDMA interrupts
> 	 (SDmaInt, SDmaIdleInt, SDmaProgressInt)
>   - software maintained counts of SDMA error cases
> 
> Reviewed-by: Dennis Dalessandro <dennis.dalessandro-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Mike Marciniszyn <mike.marciniszyn-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Vennila Megavannan <vennila.megavannan-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> Signed-off-by: Jubin John <jubin.john-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
> Changes in v2:
> 	- None
> 
> Changes in v3:
> 	- Refreshed patch against latest staging-next
> 
>  drivers/staging/rdma/hfi1/chip.c           |  110 +++++++++++++++++++++++++++-
>  drivers/staging/rdma/hfi1/chip.h           |    5 +
>  drivers/staging/rdma/hfi1/chip_registers.h |    1 +
>  drivers/staging/rdma/hfi1/hfi.h            |    1 +
>  drivers/staging/rdma/hfi1/sdma.c           |    9 ++-
>  drivers/staging/rdma/hfi1/sdma.h           |    7 ++
>  6 files changed, 129 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/staging/rdma/hfi1/chip.c b/drivers/staging/rdma/hfi1/chip.c
> index 503bfca..f4f720d 100644
> --- a/drivers/staging/rdma/hfi1/chip.c
> +++ b/drivers/staging/rdma/hfi1/chip.c
> @@ -1297,10 +1297,58 @@ static u64 dev_access_u32_csr(const struct cntr_entry *entry,
>  			    void *context, int vl, int mode, u64 data)
>  {
>  	struct hfi1_devdata *dd = context;
> +	u64 csr = entry->csr;
>  
> -	if (vl != CNTR_INVALID_VL)
> -		return 0;
> -	return read_write_csr(dd, entry->csr, mode, data);
> +	if (entry->flags & CNTR_SDMA) {
> +		if (vl == CNTR_INVALID_VL)
> +			return 0;
> +		csr += 0x100 * vl;
> +	} else {
> +		if (vl != CNTR_INVALID_VL)
> +			return 0;
> +	}
> +	return read_write_csr(dd, csr, mode, data);
> +}
> +
> +static u64 access_sde_err_cnt(const struct cntr_entry *entry,
> +			      void *context, int idx, int mode, u64 data)
> +{
> +	struct hfi1_devdata *dd = (struct hfi1_devdata *)context;
> +
> +	if (dd->per_sdma && idx < dd->num_sdma)
> +		return dd->per_sdma[idx].err_cnt;
> +	return 0;
> +}
> +
> +static u64 access_sde_int_cnt(const struct cntr_entry *entry,
> +			      void *context, int idx, int mode, u64 data)
> +{
> +	struct hfi1_devdata *dd = (struct hfi1_devdata *)context;
> +
> +	if (dd->per_sdma && idx < dd->num_sdma)
> +		return dd->per_sdma[idx].sdma_int_cnt;
> +	return 0;
> +}
> +
> +static u64 access_sde_idle_int_cnt(const struct cntr_entry *entry,
> +				   void *context, int idx, int mode, u64 data)
> +{
> +	struct hfi1_devdata *dd = (struct hfi1_devdata *)context;
> +
> +	if (dd->per_sdma && idx < dd->num_sdma)
> +		return dd->per_sdma[idx].idle_int_cnt;
> +	return 0;
> +}
> +
> +static u64 access_sde_progress_int_cnt(const struct cntr_entry *entry,
> +				       void *context, int idx, int mode,
> +				       u64 data)
> +{
> +	struct hfi1_devdata *dd = (struct hfi1_devdata *)context;
> +
> +	if (dd->per_sdma && idx < dd->num_sdma)
> +		return dd->per_sdma[idx].progress_int_cnt;
> +	return 0;
>  }
>  
>  static u64 dev_access_u64_csr(const struct cntr_entry *entry, void *context,
> @@ -4070,6 +4118,22 @@ static struct cntr_entry dev_cntrs[DEV_CNTR_LAST] = {
>  			    access_sw_kmem_wait),
>  [C_SW_SEND_SCHED] = CNTR_ELEM("SendSched", 0, 0, CNTR_NORMAL,
>  			    access_sw_send_schedule),
> +[C_SDMA_DESC_FETCHED_CNT] = CNTR_ELEM("SDEDscFdCn",
> +				      SEND_DMA_DESC_FETCHED_CNT, 0,
> +				      CNTR_NORMAL | CNTR_32BIT | CNTR_SDMA,
> +				      dev_access_u32_csr),
> +[C_SDMA_INT_CNT] = CNTR_ELEM("SDMAInt", 0, 0,
> +			     CNTR_NORMAL | CNTR_32BIT | CNTR_SDMA,
> +			     access_sde_int_cnt),
> +[C_SDMA_ERR_CNT] = CNTR_ELEM("SDMAErrCt", 0, 0,
> +			     CNTR_NORMAL | CNTR_32BIT | CNTR_SDMA,
> +			     access_sde_err_cnt),
> +[C_SDMA_IDLE_INT_CNT] = CNTR_ELEM("SDMAIdInt", 0, 0,
> +				  CNTR_NORMAL | CNTR_32BIT | CNTR_SDMA,
> +				  access_sde_idle_int_cnt),
> +[C_SDMA_PROGRESS_INT_CNT] = CNTR_ELEM("SDMAPrIntCn", 0, 0,
> +				      CNTR_NORMAL | CNTR_32BIT | CNTR_SDMA,
> +				      access_sde_progress_int_cnt),
>  /* MISC_ERR_STATUS */
>  [C_MISC_PLL_LOCK_FAIL_ERR] = CNTR_ELEM("MISC_PLL_LOCK_FAIL_ERR", 0, 0,
>  				CNTR_NORMAL,
> @@ -5707,6 +5771,7 @@ static void handle_sdma_eng_err(struct hfi1_devdata *dd,
>  	dd_dev_err(sde->dd, "CONFIG SDMA(%u) source: %u status 0x%llx\n",
>  		   sde->this_idx, source, (unsigned long long)status);
>  #endif
> +	sde->err_cnt++;
>  	sdma_engine_error(sde, status);
>  
>  	/*
> @@ -11150,6 +11215,20 @@ u32 hfi1_read_cntrs(struct hfi1_devdata *dd, loff_t pos, char **namep,
>  						dd->cntrs[entry->offset + j] =
>  									    val;
>  					}
> +				} else if (entry->flags & CNTR_SDMA) {
> +					hfi1_cdbg(CNTR,
> +						  "\t Per SDMA Engine\n");
> +					for (j = 0; j < dd->chip_sdma_engines;
> +					     j++) {
> +						val =
> +						entry->rw_cntr(entry, dd, j,
> +							       CNTR_MODE_R, 0);
> +						hfi1_cdbg(CNTR,
> +							  "\t\tRead 0x%llx for %d\n",
> +							  val, j);
> +						dd->cntrs[entry->offset + j] =
> +									val;
> +					}
>  				} else {
>  					val = entry->rw_cntr(entry, dd,
>  							CNTR_INVALID_VL,
> @@ -11553,6 +11632,21 @@ static int init_cntrs(struct hfi1_devdata *dd)
>  				dd->ndevcntrs++;
>  				index++;
>  			}
> +		} else if (dev_cntrs[i].flags & CNTR_SDMA) {
> +			hfi1_dbg_early(
> +				       "\tProcessing per SDE counters chip enginers %u\n",
> +				       dd->chip_sdma_engines);
> +			dev_cntrs[i].offset = index;
> +			for (j = 0; j < dd->chip_sdma_engines; j++) {
> +				memset(name, '\0', C_MAX_NAME);
> +				snprintf(name, C_MAX_NAME, "%s%d",
> +					 dev_cntrs[i].name, j);
> +				sz += strlen(name);
> +				sz++;
> +				hfi1_dbg_early("\t\t%s\n", name);
> +				dd->ndevcntrs++;
> +				index++;
> +			}
>  		} else {
>  			/* +1 for newline  */
>  			sz += strlen(dev_cntrs[i].name) + 1;
> @@ -11594,6 +11688,16 @@ static int init_cntrs(struct hfi1_devdata *dd)
>  					p += strlen(name);
>  					*p++ = '\n';
>  				}
> +			} else if (dev_cntrs[i].flags & CNTR_SDMA) {
> +				for (j = 0; j < TXE_NUM_SDMA_ENGINES;
> +				     j++) {
> +					memset(name, '\0', C_MAX_NAME);
> +					snprintf(name, C_MAX_NAME, "%s%d",
> +						 dev_cntrs[i].name, j);
> +					memcpy(p, name, strlen(name));
> +					p += strlen(name);
> +					*p++ = '\n';
> +				}
>  			} else {
>  				memcpy(p, dev_cntrs[i].name,
>  				       strlen(dev_cntrs[i].name));
> diff --git a/drivers/staging/rdma/hfi1/chip.h b/drivers/staging/rdma/hfi1/chip.h
> index 1368a44..b46ef66 100644
> --- a/drivers/staging/rdma/hfi1/chip.h
> +++ b/drivers/staging/rdma/hfi1/chip.h
> @@ -787,6 +787,11 @@ enum {
>  	C_SW_PIO_WAIT,
>  	C_SW_KMEM_WAIT,
>  	C_SW_SEND_SCHED,
> +	C_SDMA_DESC_FETCHED_CNT,
> +	C_SDMA_INT_CNT,
> +	C_SDMA_ERR_CNT,
> +	C_SDMA_IDLE_INT_CNT,
> +	C_SDMA_PROGRESS_INT_CNT,
>  /* MISC_ERR_STATUS */
>  	C_MISC_PLL_LOCK_FAIL_ERR,
>  	C_MISC_MBIST_FAIL_ERR,
> diff --git a/drivers/staging/rdma/hfi1/chip_registers.h b/drivers/staging/rdma/hfi1/chip_registers.h
> index 701e9e1..fb36541 100644
> --- a/drivers/staging/rdma/hfi1/chip_registers.h
> +++ b/drivers/staging/rdma/hfi1/chip_registers.h
> @@ -1290,5 +1290,6 @@
>  #define CCE_INT_BLOCKED (CCE + 0x000000110C00)
>  #define SEND_DMA_IDLE_CNT (TXE + 0x000000200040)
>  #define SEND_DMA_DESC_FETCHED_CNT (TXE + 0x000000200058)
> +#define CCE_MSIX_PBA_OFFSET 0X0110000
>  
>  #endif          /* DEF_CHIP_REG */
> diff --git a/drivers/staging/rdma/hfi1/hfi.h b/drivers/staging/rdma/hfi1/hfi.h
> index d4a859f..d32ee84 100644
> --- a/drivers/staging/rdma/hfi1/hfi.h
> +++ b/drivers/staging/rdma/hfi1/hfi.h
> @@ -490,6 +490,7 @@ struct hfi1_sge_state;
>  #define CNTR_DISABLED		0x2 /* Disable this counter */
>  #define CNTR_32BIT		0x4 /* Simulate 64 bits for this counter */
>  #define CNTR_VL			0x8 /* Per VL counter */
> +#define CNTR_SDMA              0x10
>  #define CNTR_INVALID_VL		-1  /* Specifies invalid VL */
>  #define CNTR_MODE_W		0x0
>  #define CNTR_MODE_R		0x1
> diff --git a/drivers/staging/rdma/hfi1/sdma.c b/drivers/staging/rdma/hfi1/sdma.c
> index 1d38be5..cb66bd0 100644
> --- a/drivers/staging/rdma/hfi1/sdma.c
> +++ b/drivers/staging/rdma/hfi1/sdma.c
> @@ -1061,7 +1061,6 @@ int sdma_init(struct hfi1_devdata *dd, u8 port)
>  		sde->desc_avail = sdma_descq_freecnt(sde);
>  		sde->sdma_shift = ilog2(descq_cnt);
>  		sde->sdma_mask = (1 << sde->sdma_shift) - 1;
> -		sde->descq_full_count = 0;
>  
>  		/* Create a mask for all 3 chip interrupt sources */
>  		sde->imask = (u64)1 << (0*TXE_NUM_SDMA_ENGINES + this_idx)
> @@ -1073,6 +1072,8 @@ int sdma_init(struct hfi1_devdata *dd, u8 port)
>  		/* Create a mask specifically for sdma_progress */
>  		sde->progress_mask =
>  			(u64)1 << (TXE_NUM_SDMA_ENGINES + this_idx);
> +		sde->int_mask =
> +			(u64)1 << (0 * TXE_NUM_SDMA_ENGINES + this_idx);
                                   ^^^^^^^^^^^^^^^^^^^^^^^^
Why is that there?

>  		spin_lock_init(&sde->tail_lock);
>  		seqlock_init(&sde->head_lock);
>  		spin_lock_init(&sde->senddmactrl_lock);
> @@ -1552,6 +1553,12 @@ void sdma_engine_interrupt(struct sdma_engine *sde, u64 status)
>  	trace_hfi1_sdma_engine_interrupt(sde, status);
>  	write_seqlock(&sde->head_lock);
>  	sdma_set_desc_cnt(sde, sdma_desct_intr);
> +	if (status & sde->idle_mask)
> +		sde->idle_int_cnt++;
> +	else if (status & sde->progress_mask)
> +		sde->progress_int_cnt++;
> +	else if (status & sde->int_mask)
> +		sde->sdma_int_cnt++;
>  	sdma_make_progress(sde, status);
>  	write_sequnlock(&sde->head_lock);
>  }
> diff --git a/drivers/staging/rdma/hfi1/sdma.h b/drivers/staging/rdma/hfi1/sdma.h
> index da89e64..757017a 100644
> --- a/drivers/staging/rdma/hfi1/sdma.h
> +++ b/drivers/staging/rdma/hfi1/sdma.h
> @@ -409,6 +409,7 @@ struct sdma_engine {
>  	u64 imask;			/* clear interrupt mask */
>  	u64 idle_mask;
>  	u64 progress_mask;
> +	u64 int_mask;
>  	/* private: */
>  	volatile __le64      *head_dma; /* DMA'ed by chip */
>  	/* private: */
> @@ -465,6 +466,12 @@ struct sdma_engine {
>  	u16                   tx_head;
>  	/* private: */
>  	u64                   last_status;
> +	/* private */
> +	u64                     err_cnt;
> +	/* private */
> +	u64                     sdma_int_cnt;
> +	u64                     idle_int_cnt;
> +	u64                     progress_int_cnt;
>  
>  	/* private: */
>  	struct list_head      dmawait;
> 


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [PATCH RESEND v3 5/6] staging/rdma/hfi1: adding per SDMA engine stats to hfistats

[Jubin John]

> > diff --git a/drivers/staging/rdma/hfi1/sdma.c b/drivers/staging/rdma/hfi1/sdma.c
> > index 1d38be5..cb66bd0 100644
> > --- a/drivers/staging/rdma/hfi1/sdma.c
> > +++ b/drivers/staging/rdma/hfi1/sdma.c
> > @@ -1061,7 +1061,6 @@ int sdma_init(struct hfi1_devdata *dd, u8 port)
> >  		sde->desc_avail = sdma_descq_freecnt(sde);
> >  		sde->sdma_shift = ilog2(descq_cnt);
> >  		sde->sdma_mask = (1 << sde->sdma_shift) - 1;
> > -		sde->descq_full_count = 0;
> >  
> >  		/* Create a mask for all 3 chip interrupt sources */
> >  		sde->imask = (u64)1 << (0*TXE_NUM_SDMA_ENGINES + this_idx)
> > @@ -1073,6 +1072,8 @@ int sdma_init(struct hfi1_devdata *dd, u8 port)
> >  		/* Create a mask specifically for sdma_progress */
> >  		sde->progress_mask =
> >  			(u64)1 << (TXE_NUM_SDMA_ENGINES + this_idx);
> > +		sde->int_mask =
> > +			(u64)1 << (0 * TXE_NUM_SDMA_ENGINES + this_idx);
>                                    ^^^^^^^^^^^^^^^^^^^^^^^^
> Why is that there?
> 

Hi Doug,

The zero was intentionally added in an attempt to highlight that this
was for the first interrupt source, like we do in other places above this
where we have 1 * TXE_NUM_SDMA_ENGINES, 2 * TXE_NUM_SDMA_ENGINES etc.
The interrupt sources are separated by the number of SDMA engines
i.e TXE_NUM_SDMA_ENGINES. However, we agree that the code is not
consistent throughout. Would you like us to do a cleanup of this code
in a follow-on patch or rework this patch and resend the series?

Jubin
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH RESEND v3 5/6] staging/rdma/hfi1: adding per SDMA engine stats to hfistats

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1824 bytes --]

On 03/01/2016 03:27 PM, Jubin John wrote:
>>> diff --git a/drivers/staging/rdma/hfi1/sdma.c b/drivers/staging/rdma/hfi1/sdma.c
>>> index 1d38be5..cb66bd0 100644
>>> --- a/drivers/staging/rdma/hfi1/sdma.c
>>> +++ b/drivers/staging/rdma/hfi1/sdma.c
>>> @@ -1061,7 +1061,6 @@ int sdma_init(struct hfi1_devdata *dd, u8 port)
>>>  		sde->desc_avail = sdma_descq_freecnt(sde);
>>>  		sde->sdma_shift = ilog2(descq_cnt);
>>>  		sde->sdma_mask = (1 << sde->sdma_shift) - 1;
>>> -		sde->descq_full_count = 0;
>>>  
>>>  		/* Create a mask for all 3 chip interrupt sources */
>>>  		sde->imask = (u64)1 << (0*TXE_NUM_SDMA_ENGINES + this_idx)
>>> @@ -1073,6 +1072,8 @@ int sdma_init(struct hfi1_devdata *dd, u8 port)
>>>  		/* Create a mask specifically for sdma_progress */
>>>  		sde->progress_mask =
>>>  			(u64)1 << (TXE_NUM_SDMA_ENGINES + this_idx);
>>> +		sde->int_mask =
>>> +			(u64)1 << (0 * TXE_NUM_SDMA_ENGINES + this_idx);
>>                                    ^^^^^^^^^^^^^^^^^^^^^^^^
>> Why is that there?
>>
> 
> Hi Doug,
> 
> The zero was intentionally added in an attempt to highlight that this
> was for the first interrupt source, like we do in other places above this
> where we have 1 * TXE_NUM_SDMA_ENGINES, 2 * TXE_NUM_SDMA_ENGINES etc.
> The interrupt sources are separated by the number of SDMA engines
> i.e TXE_NUM_SDMA_ENGINES. However, we agree that the code is not
> consistent throughout. Would you like us to do a cleanup of this code
> in a follow-on patch or rework this patch and resend the series?
> 
> Jubin
> 

Thanks, the code is confusing to read, so a cleanup would be nice.  Just
send me something I can squash into this patch is good enough.

-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]

Re: [PATCH RESEND v3 5/6] staging/rdma/hfi1: adding per SDMA engine stats to hfistats

[Jubin John]

On Thu, Mar 03, 2016 at 10:50:41AM -0500, Doug Ledford wrote:
> On 03/01/2016 03:27 PM, Jubin John wrote:
> >>> diff --git a/drivers/staging/rdma/hfi1/sdma.c b/drivers/staging/rdma/hfi1/sdma.c
> >>> index 1d38be5..cb66bd0 100644
> >>> --- a/drivers/staging/rdma/hfi1/sdma.c
> >>> +++ b/drivers/staging/rdma/hfi1/sdma.c
> >>> @@ -1061,7 +1061,6 @@ int sdma_init(struct hfi1_devdata *dd, u8 port)
> >>>  		sde->desc_avail = sdma_descq_freecnt(sde);
> >>>  		sde->sdma_shift = ilog2(descq_cnt);
> >>>  		sde->sdma_mask = (1 << sde->sdma_shift) - 1;
> >>> -		sde->descq_full_count = 0;
> >>>  
> >>>  		/* Create a mask for all 3 chip interrupt sources */
> >>>  		sde->imask = (u64)1 << (0*TXE_NUM_SDMA_ENGINES + this_idx)
> >>> @@ -1073,6 +1072,8 @@ int sdma_init(struct hfi1_devdata *dd, u8 port)
> >>>  		/* Create a mask specifically for sdma_progress */
> >>>  		sde->progress_mask =
> >>>  			(u64)1 << (TXE_NUM_SDMA_ENGINES + this_idx);
> >>> +		sde->int_mask =
> >>> +			(u64)1 << (0 * TXE_NUM_SDMA_ENGINES + this_idx);
> >>                                    ^^^^^^^^^^^^^^^^^^^^^^^^
> >> Why is that there?
> >>
> > 
> > Hi Doug,
> > 
> > The zero was intentionally added in an attempt to highlight that this
> > was for the first interrupt source, like we do in other places above this
> > where we have 1 * TXE_NUM_SDMA_ENGINES, 2 * TXE_NUM_SDMA_ENGINES etc.
> > The interrupt sources are separated by the number of SDMA engines
> > i.e TXE_NUM_SDMA_ENGINES. However, we agree that the code is not
> > consistent throughout. Would you like us to do a cleanup of this code
> > in a follow-on patch or rework this patch and resend the series?
> > 
> > Jubin
> > 
> 
> Thanks, the code is confusing to read, so a cleanup would be nice.  Just
> send me something I can squash into this patch is good enough.

I have sent a cleanup patch to be squashed into this patch.
https://patchwork.kernel.org/patch/8497581/

Jubin
> 
> -- 
> Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
>               GPG KeyID: 0E572FDD
> 
> 


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html