[PATCH for-next 0/8] IB/hfi1, core, rdmavt: Driver fixes for 10/2/2017

[PATCH for-next 0/8] IB/hfi1, core, rdmavt: Driver fixes for 10/2/2017

[Dennis Dalessandro]

Hi Doug,
There are a couple fixes in here that would have been nice to get into the RC
cycle, including one marked stable. However I think you will find them to be
too many LOC for an rc-4 submission so I have sent them in one series for-next.
Patches 2,3,4 and 5 are the fixes. Patch 2 is small but it's not really that
important to the end user.

There are some clean ups in here from Don from the 16B changes. One takes care
of some sparse warnings and the other two are from a WARN_ON_ONCE that needed
special cased for OPA.

Patches can can also be found in my GitHub repo at:
https://github.com/ddalessa/kernel/tree/for-4.15

---

Don Hiatt (3):
      IB/core: Use __be32 for LIDs in opa_is_extended_lid
      IB/core: Do not warn on lid conversions for OPA
      IB/hfi1: Do not warn on lid conversions for OPA

Jakub Byczkowski (1):
      IB/hfi1: Add parsing for platform configuration format version 4

Michael J. Ruhl (1):
      IB/hfi1: Fix incorrect available receive user context count

Mike Marciniszyn (1):
      IB/hfi1: Fix output trace issues from 16B change

Sebastian Sanchez (2):
      IB/hfi1: Prevent LNI out of sync by resetting host interface version
      IB/rdmavt: Correct issues with read-mostly and send size cache lines


 drivers/infiniband/core/user_mad.c        |   11 +++
 drivers/infiniband/hw/hfi1/chip.c         |  112 +++++++++++++++++++----------
 drivers/infiniband/hw/hfi1/chip.h         |    1 
 drivers/infiniband/hw/hfi1/firmware.c     |   87 ++++++++++++++++++-----
 drivers/infiniband/hw/hfi1/hfi.h          |    3 +
 drivers/infiniband/hw/hfi1/mad.c          |    7 +-
 drivers/infiniband/hw/hfi1/sysfs.c        |    2 -
 drivers/infiniband/hw/hfi1/trace_ibhdrs.h |    5 +
 drivers/infiniband/hw/hfi1/vnic_main.c    |    7 +-
 include/rdma/opa_addr.h                   |    6 +-
 include/rdma/rdmavt_qp.h                  |    6 +-
 11 files changed, 177 insertions(+), 70 deletions(-)

--
-Denny

[PATCH for-next 3/8] IB/hfi1: Fix incorrect available receive user context count

[Dennis Dalessandro]

From: Michael J. Ruhl <michael.j.ruhl@intel.com>

The addition of the VNIC contexts to num_rcv_contexts changes the
meaning of the sysfs value nctxts from available user contexts, to
user contexts + reserved VNIC contexts.

User applications that use nctxts are now broken.

Update the calculation so that VNIC contexts are used only if there are
hardware contexts available, and do not silently affect nctxts.

Update code to use the calculated VNIC context number.

Update the sysfs value nctxts to be available user contexts only.

Fixes: 2280740f01ae ("IB/hfi1: Virtual Network Interface Controller (VNIC) HW support")
Reviewed-by: Ira Weiny <ira.weiny@intel.com>
Reviewed-by: Niranjana Vishwanathapura <Niranjana.Vishwanathapura@intel.com>
Reviewed-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
Cc: <Stable@vger.kernel.org> #v4.12+
Signed-off-by: Michael J. Ruhl <michael.j.ruhl@intel.com>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro@intel.com>
---
 drivers/infiniband/hw/hfi1/chip.c      |   35 +++++++++++++++++++-------------
 drivers/infiniband/hw/hfi1/hfi.h       |    2 ++
 drivers/infiniband/hw/hfi1/sysfs.c     |    2 +-
 drivers/infiniband/hw/hfi1/vnic_main.c |    7 +++++-
 4 files changed, 29 insertions(+), 17 deletions(-)

diff --git a/drivers/infiniband/hw/hfi1/chip.c b/drivers/infiniband/hw/hfi1/chip.c
index 680cfd4..764a066 100644
--- a/drivers/infiniband/hw/hfi1/chip.c
+++ b/drivers/infiniband/hw/hfi1/chip.c
@@ -13076,7 +13076,7 @@ static int request_msix_irqs(struct hfi1_devdata *dd)
 	first_sdma = last_general;
 	last_sdma = first_sdma + dd->num_sdma;
 	first_rx = last_sdma;
-	last_rx = first_rx + dd->n_krcv_queues + HFI1_NUM_VNIC_CTXT;
+	last_rx = first_rx + dd->n_krcv_queues + dd->num_vnic_contexts;
 
 	/* VNIC MSIx interrupts get mapped when VNIC contexts are created */
 	dd->first_dyn_msix_idx = first_rx + dd->n_krcv_queues;
@@ -13282,8 +13282,9 @@ static int set_up_interrupts(struct hfi1_devdata *dd)
 	 *		slow source, SDMACleanupDone)
 	 *	N interrupts - one per used SDMA engine
 	 *	M interrupt - one per kernel receive context
+	 *	V interrupt - one for each VNIC context
 	 */
-	total = 1 + dd->num_sdma + dd->n_krcv_queues + HFI1_NUM_VNIC_CTXT;
+	total = 1 + dd->num_sdma + dd->n_krcv_queues + dd->num_vnic_contexts;
 
 	/* ask for MSI-X interrupts */
 	request = request_msix(dd, total);
@@ -13344,10 +13345,12 @@ static int set_up_interrupts(struct hfi1_devdata *dd)
  *                             in array of contexts
  *	freectxts  - number of free user contexts
  *	num_send_contexts - number of PIO send contexts being used
+ *	num_vnic_contexts - number of contexts reserved for VNIC
  */
 static int set_up_context_variables(struct hfi1_devdata *dd)
 {
 	unsigned long num_kernel_contexts;
+	u16 num_vnic_contexts = HFI1_NUM_VNIC_CTXT;
 	int total_contexts;
 	int ret;
 	unsigned ngroups;
@@ -13381,6 +13384,14 @@ static int set_up_context_variables(struct hfi1_devdata *dd)
 			   num_kernel_contexts);
 		num_kernel_contexts = dd->chip_send_contexts - num_vls - 1;
 	}
+
+	/* Accommodate VNIC contexts if possible */
+	if ((num_kernel_contexts + num_vnic_contexts) > dd->chip_rcv_contexts) {
+		dd_dev_err(dd, "No receive contexts available for VNIC\n");
+		num_vnic_contexts = 0;
+	}
+	total_contexts = num_kernel_contexts + num_vnic_contexts;
+
 	/*
 	 * User contexts:
 	 *	- default to 1 user context per real (non-HT) CPU core if
@@ -13390,19 +13401,16 @@ static int set_up_context_variables(struct hfi1_devdata *dd)
 		num_user_contexts =
 			cpumask_weight(&node_affinity.real_cpu_mask);
 
-	total_contexts = num_kernel_contexts + num_user_contexts;
-
 	/*
 	 * Adjust the counts given a global max.
 	 */
-	if (total_contexts > dd->chip_rcv_contexts) {
+	if (total_contexts + num_user_contexts > dd->chip_rcv_contexts) {
 		dd_dev_err(dd,
 			   "Reducing # user receive contexts to: %d, from %d\n",
-			   (int)(dd->chip_rcv_contexts - num_kernel_contexts),
+			   (int)(dd->chip_rcv_contexts - total_contexts),
 			   (int)num_user_contexts);
-		num_user_contexts = dd->chip_rcv_contexts - num_kernel_contexts;
 		/* recalculate */
-		total_contexts = num_kernel_contexts + num_user_contexts;
+		num_user_contexts = dd->chip_rcv_contexts - total_contexts;
 	}
 
 	/* each user context requires an entry in the RMT */
@@ -13415,25 +13423,24 @@ static int set_up_context_variables(struct hfi1_devdata *dd)
 			   user_rmt_reduced);
 		/* recalculate */
 		num_user_contexts = user_rmt_reduced;
-		total_contexts = num_kernel_contexts + num_user_contexts;
 	}
 
-	/* Accommodate VNIC contexts */
-	if ((total_contexts + HFI1_NUM_VNIC_CTXT) <= dd->chip_rcv_contexts)
-		total_contexts += HFI1_NUM_VNIC_CTXT;
+	total_contexts += num_user_contexts;
 
 	/* the first N are kernel contexts, the rest are user/vnic contexts */
 	dd->num_rcv_contexts = total_contexts;
 	dd->n_krcv_queues = num_kernel_contexts;
 	dd->first_dyn_alloc_ctxt = num_kernel_contexts;
+	dd->num_vnic_contexts = num_vnic_contexts;
 	dd->num_user_contexts = num_user_contexts;
 	dd->freectxts = num_user_contexts;
 	dd_dev_info(dd,
-		    "rcv contexts: chip %d, used %d (kernel %d, user %d)\n",
+		    "rcv contexts: chip %d, used %d (kernel %d, vnic %u, user %u)\n",
 		    (int)dd->chip_rcv_contexts,
 		    (int)dd->num_rcv_contexts,
 		    (int)dd->n_krcv_queues,
-		    (int)dd->num_rcv_contexts - dd->n_krcv_queues);
+		    dd->num_vnic_contexts,
+		    dd->num_user_contexts);
 
 	/*
 	 * Receive array allocation:
diff --git a/drivers/infiniband/hw/hfi1/hfi.h b/drivers/infiniband/hw/hfi1/hfi.h
index 44a2c8f..2f3c830 100644
--- a/drivers/infiniband/hw/hfi1/hfi.h
+++ b/drivers/infiniband/hw/hfi1/hfi.h
@@ -1049,6 +1049,8 @@ struct hfi1_devdata {
 	u64 z_send_schedule;
 
 	u64 __percpu *send_schedule;
+	/* number of reserved contexts for VNIC usage */
+	u16 num_vnic_contexts;
 	/* number of receive contexts in use by the driver */
 	u32 num_rcv_contexts;
 	/* number of pio send contexts in use by the driver */
diff --git a/drivers/infiniband/hw/hfi1/sysfs.c b/drivers/infiniband/hw/hfi1/sysfs.c
index 6d2702e..25e8673 100644
--- a/drivers/infiniband/hw/hfi1/sysfs.c
+++ b/drivers/infiniband/hw/hfi1/sysfs.c
@@ -543,7 +543,7 @@ static ssize_t show_nctxts(struct device *device,
 	 * give a more accurate picture of total contexts available.
 	 */
 	return scnprintf(buf, PAGE_SIZE, "%u\n",
-			 min(dd->num_rcv_contexts - dd->first_dyn_alloc_ctxt,
+			 min(dd->num_user_contexts,
 			     (u32)dd->sc_sizes[SC_USER].count));
 }
 
diff --git a/drivers/infiniband/hw/hfi1/vnic_main.c b/drivers/infiniband/hw/hfi1/vnic_main.c
index f419cbb..1a17708 100644
--- a/drivers/infiniband/hw/hfi1/vnic_main.c
+++ b/drivers/infiniband/hw/hfi1/vnic_main.c
@@ -840,6 +840,9 @@ struct net_device *hfi1_vnic_alloc_rn(struct ib_device *device,
 	struct rdma_netdev *rn;
 	int i, size, rc;
 
+	if (!dd->num_vnic_contexts)
+		return ERR_PTR(-ENOMEM);
+
 	if (!port_num || (port_num > dd->num_pports))
 		return ERR_PTR(-EINVAL);
 
@@ -848,7 +851,7 @@ struct net_device *hfi1_vnic_alloc_rn(struct ib_device *device,
 
 	size = sizeof(struct opa_vnic_rdma_netdev) + sizeof(*vinfo);
 	netdev = alloc_netdev_mqs(size, name, name_assign_type, setup,
-				  dd->chip_sdma_engines, HFI1_NUM_VNIC_CTXT);
+				  dd->chip_sdma_engines, dd->num_vnic_contexts);
 	if (!netdev)
 		return ERR_PTR(-ENOMEM);
 
@@ -856,7 +859,7 @@ struct net_device *hfi1_vnic_alloc_rn(struct ib_device *device,
 	vinfo = opa_vnic_dev_priv(netdev);
 	vinfo->dd = dd;
 	vinfo->num_tx_q = dd->chip_sdma_engines;
-	vinfo->num_rx_q = HFI1_NUM_VNIC_CTXT;
+	vinfo->num_rx_q = dd->num_vnic_contexts;
 	vinfo->netdev = netdev;
 	rn->free_rdma_netdev = hfi1_vnic_free_rn;
 	rn->set_id = hfi1_vnic_set_vesw_id;

Re: [PATCH for-next 0/8] IB/hfi1, core, rdmavt: Driver fixes for 10/2/2017

[Doug Ledford]

On Mon, 2017-10-02 at 11:03 -0700, Dennis Dalessandro wrote:
> Hi Doug,
> There are a couple fixes in here that would have been nice to get
> into the RC
> cycle, including one marked stable. However I think you will find
> them to be
> too many LOC for an rc-4 submission so I have sent them in one series
> for-next.
> Patches 2,3,4 and 5 are the fixes. Patch 2 is small but it's not
> really that
> important to the end user.
> 
> There are some clean ups in here from Don from the 16B changes. One
> takes care
> of some sparse warnings and the other two are from a WARN_ON_ONCE
> that needed
> special cased for OPA.
> 
> Patches can can also be found in my GitHub repo at:
> https://github.com/ddalessa/kernel/tree/for-4.15

Hi Denny,

I didn't process that you mixed for-rc and for-next stuff in a single
thread before I had gone through and looked at the patches and
processed them.  So, this time they all went to for-next.  In the
future, you really need patches you want in for-rc separate from the
patches intended for for-next.

Anyway, applied.  And 2 and 5 are the only ones I would have been
likely to take to for-rc right now, and 2 isn't that important to the
end user as you say, so that just leaves 5, and it's sparse warnings,
so we'll just leave things as they are.

-- 
Doug Ledford <dledford@redhat.com>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

Re: [PATCH for-next 0/8] IB/hfi1, core, rdmavt: Driver fixes for 10/2/2017

[Dennis Dalessandro]

On 10/4/2017 3:44 PM, Doug Ledford wrote:
> On Mon, 2017-10-02 at 11:03 -0700, Dennis Dalessandro wrote:
>> Hi Doug,
>> There are a couple fixes in here that would have been nice to get
>> into the RC
>> cycle, including one marked stable. However I think you will find
>> them to be
>> too many LOC for an rc-4 submission so I have sent them in one series
>> for-next.
>> Patches 2,3,4 and 5 are the fixes. Patch 2 is small but it's not
>> really that
>> important to the end user.
>>
>> There are some clean ups in here from Don from the 16B changes. One
>> takes care
>> of some sparse warnings and the other two are from a WARN_ON_ONCE
>> that needed
>> special cased for OPA.
>>
>> Patches can can also be found in my GitHub repo at:
>> https://github.com/ddalessa/kernel/tree/for-4.15
> 
> Hi Denny,
> 
> I didn't process that you mixed for-rc and for-next stuff in a single
> thread before I had gone through and looked at the patches and
> processed them.  So, this time they all went to for-next.  In the
> future, you really need patches you want in for-rc separate from the
> patches intended for for-next.

Maybe I wasn't too clear, I didn't intend any of those to go for-rc. So 
yep for-next was the right target.

I would have liked to get the fixes into -rc but they were just too 
complex for this late in the game is all I meant.

-Denny

Re: [PATCH for-next 0/8] IB/hfi1, core, rdmavt: Driver fixes for 10/2/2017

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1817 bytes --]

On Wed, Oct 04, 2017 at 09:22:06PM -0400, Dennis Dalessandro wrote:
> On 10/4/2017 3:44 PM, Doug Ledford wrote:
> > On Mon, 2017-10-02 at 11:03 -0700, Dennis Dalessandro wrote:
> > > Hi Doug,
> > > There are a couple fixes in here that would have been nice to get
> > > into the RC
> > > cycle, including one marked stable. However I think you will find
> > > them to be
> > > too many LOC for an rc-4 submission so I have sent them in one series
> > > for-next.
> > > Patches 2,3,4 and 5 are the fixes. Patch 2 is small but it's not
> > > really that
> > > important to the end user.
> > >
> > > There are some clean ups in here from Don from the 16B changes. One
> > > takes care
> > > of some sparse warnings and the other two are from a WARN_ON_ONCE
> > > that needed
> > > special cased for OPA.
> > >
> > > Patches can can also be found in my GitHub repo at:
> > > https://github.com/ddalessa/kernel/tree/for-4.15
> >
> > Hi Denny,
> >
> > I didn't process that you mixed for-rc and for-next stuff in a single
> > thread before I had gone through and looked at the patches and
> > processed them.  So, this time they all went to for-next.  In the
> > future, you really need patches you want in for-rc separate from the
> > patches intended for for-next.
>
> Maybe I wasn't too clear, I didn't intend any of those to go for-rc. So yep
> for-next was the right target.
>
> I would have liked to get the fixes into -rc but they were just too complex
> for this late in the game is all I meant.

IMHO, the number of LOCs shouldn't be the gating factor for -rc, but the
severity of fixes yes.

Thanks

>
> -Denny
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH for-next 0/8] IB/hfi1, core, rdmavt: Driver fixes for 10/2/2017

[Dennis Dalessandro]

On 10/5/2017 3:04 AM, Leon Romanovsky wrote:
> On Wed, Oct 04, 2017 at 09:22:06PM -0400, Dennis Dalessandro wrote:
>> On 10/4/2017 3:44 PM, Doug Ledford wrote:
>>> On Mon, 2017-10-02 at 11:03 -0700, Dennis Dalessandro wrote:
>>>> Hi Doug,
>>>> There are a couple fixes in here that would have been nice to get
>>>> into the RC
>>>> cycle, including one marked stable. However I think you will find
>>>> them to be
>>>> too many LOC for an rc-4 submission so I have sent them in one series
>>>> for-next.
>>>> Patches 2,3,4 and 5 are the fixes. Patch 2 is small but it's not
>>>> really that
>>>> important to the end user.
>>>>
>>>> There are some clean ups in here from Don from the 16B changes. One
>>>> takes care
>>>> of some sparse warnings and the other two are from a WARN_ON_ONCE
>>>> that needed
>>>> special cased for OPA.
>>>>
>>>> Patches can can also be found in my GitHub repo at:
>>>> https://github.com/ddalessa/kernel/tree/for-4.15
>>>
>>> Hi Denny,
>>>
>>> I didn't process that you mixed for-rc and for-next stuff in a single
>>> thread before I had gone through and looked at the patches and
>>> processed them.  So, this time they all went to for-next.  In the
>>> future, you really need patches you want in for-rc separate from the
>>> patches intended for for-next.
>>
>> Maybe I wasn't too clear, I didn't intend any of those to go for-rc. So yep
>> for-next was the right target.
>>
>> I would have liked to get the fixes into -rc but they were just too complex
>> for this late in the game is all I meant.
> 
> IMHO, the number of LOCs shouldn't be the gating factor for -rc, but the
> severity of fixes yes.

Not the gating factor necessarily, but a factor that has to be be 
weighed against the severity of the bug being fixed. As we get into 
later -rc cycles the bar for risk vs reward gets raised, not that there 
should be a limit of X LOC for -rc whatever it's more of an overall 
assessment of the risk of causing more harm than good.

-Denny

Re: [PATCH for-next 0/8] IB/hfi1, core, rdmavt: Driver fixes for 10/2/2017

[Doug Ledford]

[-- Attachment #1.1: Type: text/plain, Size: 1167 bytes --]

On 10/5/2017 6:44 AM, Dennis Dalessandro wrote:
> On 10/5/2017 3:04 AM, Leon Romanovsky wrote:
>> On Wed, Oct 04, 2017 at 09:22:06PM -0400, Dennis Dalessandro wrote:
>>> Maybe I wasn't too clear, I didn't intend any of those to go for-rc.
>>> So yep
>>> for-next was the right target.

Ok.

>>> I would have liked to get the fixes into -rc but they were just too
>>> complex
>>> for this late in the game is all I meant.

Understood.

>> IMHO, the number of LOCs shouldn't be the gating factor for -rc, but the
>> severity of fixes yes.
> 
> Not the gating factor necessarily, but a factor that has to be be
> weighed against the severity of the bug being fixed. As we get into
> later -rc cycles the bar for risk vs reward gets raised, not that there
> should be a limit of X LOC for -rc whatever it's more of an overall
> assessment of the risk of causing more harm than good.

Exactly.  And given the rc stage we are at now, and the complexity of
the fixes, you made the right call.


-- 
Doug Ledford <dledford@redhat.com>
    GPG Key ID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 884 bytes --]