[PATCH net-next] cxgb4: Fix for SR-IOV VF initialization

[PATCH net-next] cxgb4: Fix for SR-IOV VF initialization

[Hariprasad Shenai]

Commit 35b1de5 ("rdma/cxgb4: Fixes cxgb4 probe failure in VM when PF is exposed
through PCI Passthrough") introduced a regression, where VF failed to
initialize. This commit fixes it.

Signed-off-by: Casey Leedom <leedom@chelsio.com>
Signed-off-by: Hariprasad Shenai <hariprasad@chelsio.com>
---
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c |   51 +++++++++++++++++++++--
 1 files changed, 47 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
index 8b46534..c78e6b2 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
+++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
@@ -6695,11 +6695,33 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 		attach_ulds(adapter);
 
 #ifdef CONFIG_PCI_IOV
-	if (func < ARRAY_SIZE(num_vf) && num_vf[func] > 0)
-		if (pci_enable_sriov(pdev, num_vf[func]) == 0)
-			dev_info(&pdev->dev,
+	for (func = 0; func < ARRAY_SIZE(num_vf); func++) {
+		struct pci_dev *pf;
+
+		if (num_vf[func] <= 0)
+			continue;
+
+		pf = pci_get_slot(pdev->bus,
+				  PCI_DEVFN(PCI_SLOT(pdev->devfn),
+					    func));
+		if (pf == NULL) {
+			dev_warn(&pdev->dev, "failed to find PF%d; not"
+				 " enabling %d virtual functions\n",
+				 func, num_vf[func]);
+			continue;
+		}
+		err = pci_enable_sriov(pf, num_vf[func]);
+		if (err < 0)
+			dev_warn(&pf->dev, "failed to instantiate %d"
+				 " virtual functions; err=%d\n",
+				 num_vf[func], err);
+		else
+			dev_info(&pf->dev,
 				 "instantiated %u virtual functions\n",
 				 num_vf[func]);
+
+		pci_dev_put(pf);
+	}
 #endif
 	return 0;
 
@@ -6725,8 +6747,29 @@ static void remove_one(struct pci_dev *pdev)
 	struct adapter *adapter = pci_get_drvdata(pdev);
 
 #ifdef CONFIG_PCI_IOV
-	pci_disable_sriov(pdev);
+	/* Loop accross SR-IOV PFs to see if any VFs need to be
+	 * uninstantiated.
+	 */
+	int func;
+
+	for (func = 0; func < ARRAY_SIZE(num_vf); func++) {
+		struct pci_dev *pf;
+
+		if (num_vf[func] <= 0)
+			continue;
 
+		pf = pci_get_slot(pdev->bus,
+				  PCI_DEVFN(PCI_SLOT(pdev->devfn),
+					    func));
+		if (pf == NULL) {
+			dev_warn(&pdev->dev, "failed to find PF%d; not"
+				 " disabling %d virtual functions\n",
+				 func, num_vf[func]);
+			continue;
+		}
+		pci_disable_sriov(pf);
+		pci_dev_put(pf);
+	}
 #endif
 
 	if (adapter) {
-- 
1.7.1

Re: [PATCH net-next] cxgb4: Fix for SR-IOV VF initialization

[David Miller]

From: Hariprasad Shenai <hariprasad@chelsio.com>
Date: Tue, 22 Jul 2014 16:26:20 +0530

> Commit 35b1de5 ("rdma/cxgb4: Fixes cxgb4 probe failure in VM when PF is exposed
> through PCI Passthrough") introduced a regression, where VF failed to
> initialize. This commit fixes it.
> 
> Signed-off-by: Casey Leedom <leedom@chelsio.com>
> Signed-off-by: Hariprasad Shenai <hariprasad@chelsio.com>

This commit message need to explain things better, how exactly was
the regression introduced, what's exactly wrong with the current code?

I actually can't figure it out myself, other than to say that maybe
replacing things with:

	func = PCI_FUNC(pdev->devfn);
 	if (func < ARRAY_SIZE(num_vf) && num_vf[func] > 0)
 		if (pci_enable_sriov(pdev, num_vf[func]) == 0)

would work equally as well.  That's precisely what the code was
doing before the mentioned commit.

Why do we have to iterate over _ALL_ functions of the PCI device,
rather than just directly enable SRIOV on the one function whether
it bet PCI_FUNC(pdev->devfn) or that WHOAMI value?

You need to explain this so that people understand the how and the
why of your changes.

Thanks.

Re: [PATCH net-next] cxgb4: Fix for SR-IOV VF initialization

[Casey Leedom]

   Sorry, Hari passed that patch off for review internally but I had 
just gotten back from a backpacking trip at the time.  I'll work with 
him to understand what the issue is.  Unfortunately I'm heading off on a 
second backpacking trip tomorrow so we probably won't be able to resolve 
this till next week.

Casey

On 07/22/14 15:34, David Miller wrote:
> From: Hariprasad Shenai <hariprasad@chelsio.com>
> Date: Tue, 22 Jul 2014 16:26:20 +0530
>
>> Commit 35b1de5 ("rdma/cxgb4: Fixes cxgb4 probe failure in VM when PF is exposed
>> through PCI Passthrough") introduced a regression, where VF failed to
>> initialize. This commit fixes it.
>>
>> Signed-off-by: Casey Leedom <leedom@chelsio.com>
>> Signed-off-by: Hariprasad Shenai <hariprasad@chelsio.com>
> This commit message need to explain things better, how exactly was
> the regression introduced, what's exactly wrong with the current code?
>
> I actually can't figure it out myself, other than to say that maybe
> replacing things with:
>
> 	func = PCI_FUNC(pdev->devfn);
>   	if (func < ARRAY_SIZE(num_vf) && num_vf[func] > 0)
>   		if (pci_enable_sriov(pdev, num_vf[func]) == 0)
>
> would work equally as well.  That's precisely what the code was
> doing before the mentioned commit.
>
> Why do we have to iterate over _ALL_ functions of the PCI device,
> rather than just directly enable SRIOV on the one function whether
> it bet PCI_FUNC(pdev->devfn) or that WHOAMI value?
>
> You need to explain this so that people understand the how and the
> why of your changes.
>
> Thanks.

Re: [PATCH net-next] cxgb4: Fix for SR-IOV VF initialization

[Alexander Duyck]

On 07/22/2014 03:34 PM, David Miller wrote:
> From: Hariprasad Shenai <hariprasad@chelsio.com>
> Date: Tue, 22 Jul 2014 16:26:20 +0530
> 
>> Commit 35b1de5 ("rdma/cxgb4: Fixes cxgb4 probe failure in VM when PF is exposed
>> through PCI Passthrough") introduced a regression, where VF failed to
>> initialize. This commit fixes it.
>>
>> Signed-off-by: Casey Leedom <leedom@chelsio.com>
>> Signed-off-by: Hariprasad Shenai <hariprasad@chelsio.com>
> 
> This commit message need to explain things better, how exactly was
> the regression introduced, what's exactly wrong with the current code?
> 
> I actually can't figure it out myself, other than to say that maybe
> replacing things with:
> 
> 	func = PCI_FUNC(pdev->devfn);
>  	if (func < ARRAY_SIZE(num_vf) && num_vf[func] > 0)
>  		if (pci_enable_sriov(pdev, num_vf[func]) == 0)
> 
> would work equally as well.  That's precisely what the code was
> doing before the mentioned commit.
> 
> Why do we have to iterate over _ALL_ functions of the PCI device,
> rather than just directly enable SRIOV on the one function whether
> it bet PCI_FUNC(pdev->devfn) or that WHOAMI value?
> 
> You need to explain this so that people understand the how and the
> why of your changes.
> 
> Thanks.

What it looks like it is doing is forcing the loop to iterate over
multiple PFs enabling SR-IOV on each one.  Same thing for disabling on
remove.  I would think this would fail for a multifunction device since
calling this a pci_enable_sriov a second time with values when SR-IOV is
enabled should return -EINVAL.

I thought the use of module parameters for SR-IOV had been deprecated in
favor of the PCI sysfs approach?  It seems like switching over might be
a better way to resolve whatever issue this was trying to address.

Thanks,

Alex