[PATCH V7 1/3] PCI: limit FLR wait time to 100ms maximum

[PATCH V7 1/3] PCI: limit FLR wait time to 100ms maximum

[Sinan Kaya]

Code is currently allowing PCIe devices to extend polling time up to 1
second. Reducing the wait time for virtual functions to 100ms maximum to
satisfy spec requirement mentioned in PCIe r3.1, sec 6.6.2. Function-Level
Reset (FLR).

SR-IOV r1.1, sec 2.2.2 also mentions that the virtual function's presence
from configuration space is not affected from FLR. There is no point in
polling the command register since it should always return success.

Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
---
 drivers/pci/pci.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index af0cc34..2ed604a 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -3812,7 +3812,7 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
 EXPORT_SYMBOL(pci_wait_for_pending_transaction);
 
 /*
- * We should only need to wait 100ms after FLR, but some devices take longer.
+ * We should only need to wait 100ms after FLR for virtual functions.
  * Wait for up to 1000ms for config space to return something other than -1.
  * Intel IGD requires this when an LCD panel is attached.  We read the 2nd
  * dword because VFs don't implement the 1st dword.
@@ -3822,6 +3822,11 @@ static void pci_flr_wait(struct pci_dev *dev)
 	int i = 0;
 	u32 id;
 
+	if (dev->is_virtfn) {
+		msleep(100);
+		return;
+	}
+
 	do {
 		msleep(100);
 		pci_read_config_dword(dev, PCI_COMMAND, &id);
-- 
1.9.1

[PATCH V7 2/3] PCI: handle CRS returned by device after FLR

[Sinan Kaya]

An endpoint is allowed to issue Configuration Request Retry Status (CRS)
following a Function Level Reset (FLR) request to indicate that it is
not ready to accept new requests. CRS is defined in PCIe r3.1, sec 2.3.1.
Request Handling Rules and CRS usage in FLR context is mentioned in
PCIe r3.1, sec 6.6.2. Function-Level Reset.

Adding a vendor ID read if this is a physical function before attempting
to read any other registers on the endpoint. A CRS indication will only
be given if the address to be read is vendor ID register.
pci_bus_read_dev_vendor_id() knows how to deal with CRS returned values.

Calling pci_bus_read_dev_vendor_id() with 60 seconds timeout to be
consistent with the rest of the kernel CRS timeout policy.

Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
---
 drivers/pci/pci.c | 27 +++++++++++----------------
 1 file changed, 11 insertions(+), 16 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 2ed604a..ad49e73 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -3813,30 +3813,25 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
 
 /*
  * We should only need to wait 100ms after FLR for virtual functions.
- * Wait for up to 1000ms for config space to return something other than -1.
- * Intel IGD requires this when an LCD panel is attached.  We read the 2nd
- * dword because VFs don't implement the 1st dword.
+ * Wait for up to 60s for config space to return something other than -1.
+ * Intel IGD requires 300ms when an LCD panel is attached.  We use
+ * pci_bus_read_dev_vendor_id() for reading the vendor ID as it handles
+ * CRS gracefully.
  */
 static void pci_flr_wait(struct pci_dev *dev)
 {
-	int i = 0;
 	u32 id;
+	bool ret;
 
-	if (dev->is_virtfn) {
-		msleep(100);
-		return;
-	}
+	/* don't touch the HW before waiting 100ms */
+	msleep(100);
 
-	do {
-		msleep(100);
-		pci_read_config_dword(dev, PCI_COMMAND, &id);
-	} while (i++ < 10 && id == ~0);
+	if (dev->is_virtfn)
+		return;
 
-	if (id == ~0)
+	ret = pci_bus_read_dev_vendor_id(dev->bus, dev->devfn, &id, 60000);
+	if (!ret)
 		dev_warn(&dev->dev, "Failed to return from FLR\n");
-	else if (i > 1)
-		dev_info(&dev->dev, "Required additional %dms to return from FLR\n",
-			 (i - 1) * 100);
 }
 
 /**
-- 
1.9.1

[PATCH V7 3/3] PCI: display not responding message while device is unreachable

[Sinan Kaya]

Adding a print statement into pci_bus_read_dev_vendor_id() so that
user observes the progress of device polling instead of silently
waiting for timeout to be reached.

Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
---
 drivers/pci/probe.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index c31310d..6e4a846 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1851,9 +1851,15 @@ bool pci_bus_read_dev_vendor_id(struct pci_bus *bus, int devfn, u32 *l,
 		delay *= 2;
 		if (pci_bus_read_config_dword(bus, devfn, PCI_VENDOR_ID, l))
 			return false;
+
+		if (delay > 1000)
+			pr_info("pci %04x:%02x:%02x.%d: not responding\n",
+				pci_domain_nr(bus), bus->number,
+				PCI_SLOT(devfn), PCI_FUNC(devfn));
+
 		/* Card hasn't responded in 60 seconds?  Must be stuck. */
 		if (delay > crs_timeout) {
-			printk(KERN_WARNING "pci %04x:%02x:%02x.%d: not responding\n",
+			pr_warn("pci %04x:%02x:%02x.%d: not responding timeout reached\n",
 			       pci_domain_nr(bus), bus->number, PCI_SLOT(devfn),
 			       PCI_FUNC(devfn));
 			return false;
-- 
1.9.1

Re: [PATCH V7 1/3] PCI: limit FLR wait time to 100ms maximum

[Bjorn Helgaas]

On Wed, Aug 02, 2017 at 03:15:15PM -0400, Sinan Kaya wrote:
> Code is currently allowing PCIe devices to extend polling time up to 1
> second. Reducing the wait time for virtual functions to 100ms maximum to
> satisfy spec requirement mentioned in PCIe r3.1, sec 6.6.2. Function-Level
> Reset (FLR).

This text suggests that the spec says we cannot wait more than 100ms.
That is not what the spec says.  There is no spec requirement to
*reduce* the wait time.

I haven't worked through all the details of what 6.6.2 says, but I
think it uses 100ms in the context of the *minimum* time software must
wait between initiating an FLR and initializing the function.

And of course, PCIe r3.1, sec 6.6.2, doesn't mention VFs at all.

> SR-IOV r1.1, sec 2.2.2 also mentions that the virtual function's presence
> from configuration space is not affected from FLR. There is no point in
> polling the command register since it should always return success.

My hypothesis was that CRS isn't useful on VFs because of sec 2.2.2
says "FLR ... does not affect its existence in PCI Configuration
Space".  But I think that hypothesis is wrong because sec 3.3.3.1
does talk about a VF returning CRS.

The SR-IOV spec (sec 3.4.1.1) says a VF's Vendor ID is read-only
0xffff.  But I expect CRS visibility (PCIe r3.1, sec 2.3.2) would work
normally and return a Vendor ID of 0x0001 to indicate CRS for a VF.

Of course, not all Root Ports support CRS software visibility, so
whatever we do has to work when it's absent.

> Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
> ---
>  drivers/pci/pci.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index af0cc34..2ed604a 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -3812,7 +3812,7 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
>  EXPORT_SYMBOL(pci_wait_for_pending_transaction);
>  
>  /*
> - * We should only need to wait 100ms after FLR, but some devices take longer.
> + * We should only need to wait 100ms after FLR for virtual functions.
>   * Wait for up to 1000ms for config space to return something other than -1.
>   * Intel IGD requires this when an LCD panel is attached.  We read the 2nd
>   * dword because VFs don't implement the 1st dword.
> @@ -3822,6 +3822,11 @@ static void pci_flr_wait(struct pci_dev *dev)
>  	int i = 0;
>  	u32 id;
>  
> +	if (dev->is_virtfn) {
> +		msleep(100);
> +		return;
> +	}
> +
>  	do {
>  		msleep(100);
>  		pci_read_config_dword(dev, PCI_COMMAND, &id);
> -- 
> 1.9.1
> 

Re: [PATCH V7 1/3] PCI: limit FLR wait time to 100ms maximum

[Sinan Kaya]

On 8/2/2017 4:27 PM, Bjorn Helgaas wrote:
> I haven't worked through all the details of what 6.6.2 says, but I
> think it uses 100ms in the context of the *minimum* time software must
> wait between initiating an FLR and initializing the function

Here is what spec says in 6.6.2

"After an FLR has been initiated by writing a 1b to the Initiate Function Level Reset bit,
the Function must complete the FLR within 100 ms."

I interpret this as maximum.

-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

Re: [PATCH V7 1/3] PCI: limit FLR wait time to 100ms maximum

[Sinan Kaya]

On 8/2/2017 4:27 PM, Bjorn Helgaas wrote:
> On Wed, Aug 02, 2017 at 03:15:15PM -0400, Sinan Kaya wrote:
> My hypothesis was that CRS isn't useful on VFs because of sec 2.2.2
> says "FLR ... does not affect its existence in PCI Configuration
> Space".  But I think that hypothesis is wrong because sec 3.3.3.1
> does talk about a VF returning CRS.
> 

hmm, good catch. I'll drop the first patch.

> The SR-IOV spec (sec 3.4.1.1) says a VF's Vendor ID is read-only
> 0xffff.  But I expect CRS visibility (PCIe r3.1, sec 2.3.2) would work
> normally and return a Vendor ID of 0x0001 to indicate CRS for a VF.
> 
> Of course, not all Root Ports support CRS software visibility, so
> whatever we do has to work when it's absent.

How about a mixture of old code and new code as follows?

static void pci_flr_wait(struct pci_dev *dev)
{
	u32 id;
	bool ret = false;
	int i = 0;

	if (CRS supported) {
		/* don't touch the HW before waiting 100ms */
		msleep(100);

		ret = pci_bus_read_dev_vendor_id(dev->bus, dev->devfn, &id, 60000);
		if (ret)
			return;
	}

	do {
		msleep(100);
		pci_read_config_dword(dev, PCI_COMMAND, &id);
	} while (i++ < 10 && id == ~0);
 
	if (id == ~0)
 		dev_warn(&dev->dev, "Failed to return from FLR\n");
	else if (i > 1)
		dev_info(&dev->dev, "Required additional %dms to return from FLR\n",
			 (i - 1) * 100);
}


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.