[PATCH V2 0/7] PCI: handle CRS response following Hot Reset and D3hot->D0

[PATCH V2 0/7] PCI: handle CRS response following Hot Reset and D3hot->D0

[Sinan Kaya]

Rev 3.1 Sec 2.3.1 Request Handling Rules:
Valid reset conditions after which a device is permitted to return CRS
are:
* Cold, Warm, and Hot Resets,
* FLR
* A reset initiated in response to a D3hot to D0 uninitialized

Try to reuse FLR implementation towards other reset types.

Changes from v1:
* PCI: add a return type for pci_reset_bridge_secondary_bus()
* set waittime as a global variable and have a kernel command line to
* override.
* get rid of initial sleep parameter

Sinan Kaya (7):
  PCI: protect restore with device lock to be consistent
  PCI: handle FLR failure and allow other reset types
  PCI: make pci_flr_wait() generic and rename to pci_dev_wait()
  PCI: wait device ready after pci_pm_reset()
  PCI: add a return type for pci_reset_bridge_secondary_bus()
  PCI: add device wait after slot and bus reset
  PCI: make reset poll time adjustable

 Documentation/admin-guide/kernel-parameters.txt |  2 +
 drivers/pci/pci.c                               | 72 ++++++++++++++++---------
 include/linux/pci.h                             |  4 +-
 3 files changed, 51 insertions(+), 27 deletions(-)

-- 
1.9.1

[PATCH V2 1/7] PCI: protect restore with device lock to be consistent

[Sinan Kaya]

Commit b014e96d1abb ("PCI: Protect pci_error_handlers->reset_notify() usage
with device_lock()") added protection around pci_dev_restore() function so
that device specific remove callback does not cause a race condition
against hotplug.

pci_dev_lock() usage has been forgotten in two different places in the
code. Adding locks for pci_slot_restore() and moving pci_dev_restore()
inside the locks for pci_try_reset_function().

Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
---
 drivers/pci/pci.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 6078dfc..23681f4 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -4344,9 +4344,9 @@ int pci_try_reset_function(struct pci_dev *dev)
 
 	pci_dev_save_and_disable(dev);
 	rc = __pci_reset_function_locked(dev);
+	pci_dev_restore(dev);
 	pci_dev_unlock(dev);
 
-	pci_dev_restore(dev);
 	return rc;
 }
 EXPORT_SYMBOL_GPL(pci_try_reset_function);
@@ -4546,7 +4546,9 @@ static void pci_slot_restore(struct pci_slot *slot)
 	list_for_each_entry(dev, &slot->bus->devices, bus_list) {
 		if (!dev->slot || dev->slot != slot)
 			continue;
+		pci_dev_lock(dev);
 		pci_dev_restore(dev);
+		pci_dev_unlock(dev);
 		if (dev->subordinate)
 			pci_bus_restore(dev->subordinate);
 	}
-- 
1.9.1

[PATCH V2 1/7] PCI: protect restore with device lock to be consistent

[Christoph Hellwig]

On Mon, Nov 27, 2017 at 01:20:22AM -0500, Sinan Kaya wrote:
> Commit b014e96d1abb ("PCI: Protect pci_error_handlers->reset_notify() usage
> with device_lock()") added protection around pci_dev_restore() function so
> that device specific remove callback does not cause a race condition
> against hotplug.
> 
> pci_dev_lock() usage has been forgotten in two different places in the
> code. Adding locks for pci_slot_restore() and moving pci_dev_restore()
> inside the locks for pci_try_reset_function().
> 
> Signed-off-by: Sinan Kaya <okaya@codeaurora.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

[PATCH V2 2/7] PCI: handle FLR failure and allow other reset types

[Sinan Kaya]

pci_flr_wait() and pci_af_flr() functions assume graceful return even
though the device is inaccessible under error conditions.

Return -ENOTTY in error cases so that __pci_reset_function_locked() can
try other reset types if AF_FLR/FLR reset fails.

Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
---
 drivers/pci/pci.c   | 18 ++++++++++--------
 include/linux/pci.h |  2 +-
 2 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 23681f4..27ec45d 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -3820,7 +3820,7 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
 }
 EXPORT_SYMBOL(pci_wait_for_pending_transaction);
 
-static void pci_flr_wait(struct pci_dev *dev)
+static int pci_flr_wait(struct pci_dev *dev)
 {
 	int delay = 1, timeout = 60000;
 	u32 id;
@@ -3849,7 +3849,7 @@ static void pci_flr_wait(struct pci_dev *dev)
 		if (delay > timeout) {
 			dev_warn(&dev->dev, "not ready %dms after FLR; giving up\n",
 				 100 + delay - 1);
-			return;
+			return -ENOTTY;
 		}
 
 		if (delay > 1000)
@@ -3863,6 +3863,8 @@ static void pci_flr_wait(struct pci_dev *dev)
 
 	if (delay > 1000)
 		dev_info(&dev->dev, "ready %dms after FLR\n", 100 + delay - 1);
+
+	return 0;
 }
 
 /**
@@ -3891,13 +3893,13 @@ static bool pcie_has_flr(struct pci_dev *dev)
  * device supports FLR before calling this function, e.g. by using the
  * pcie_has_flr() helper.
  */
-void pcie_flr(struct pci_dev *dev)
+int pcie_flr(struct pci_dev *dev)
 {
 	if (!pci_wait_for_pending_transaction(dev))
 		dev_err(&dev->dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
 
 	pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_BCR_FLR);
-	pci_flr_wait(dev);
+	return pci_flr_wait(dev);
 }
 EXPORT_SYMBOL_GPL(pcie_flr);
 
@@ -3930,8 +3932,7 @@ static int pci_af_flr(struct pci_dev *dev, int probe)
 		dev_err(&dev->dev, "timed out waiting for pending transaction; performing AF function level reset anyway\n");
 
 	pci_write_config_byte(dev, pos + PCI_AF_CTRL, PCI_AF_CTRL_FLR);
-	pci_flr_wait(dev);
-	return 0;
+	return pci_flr_wait(dev);
 }
 
 /**
@@ -4203,8 +4204,9 @@ int __pci_reset_function_locked(struct pci_dev *dev)
 	if (rc != -ENOTTY)
 		return rc;
 	if (pcie_has_flr(dev)) {
-		pcie_flr(dev);
-		return 0;
+		rc = pcie_flr(dev);
+		if (rc != -ENOTTY)
+			return rc;
 	}
 	rc = pci_af_flr(dev, 0);
 	if (rc != -ENOTTY)
diff --git a/include/linux/pci.h b/include/linux/pci.h
index d16a7c0..354b018 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1089,7 +1089,7 @@ static inline int pci_is_managed(struct pci_dev *pdev)
 int pcie_set_mps(struct pci_dev *dev, int mps);
 int pcie_get_minimum_link(struct pci_dev *dev, enum pci_bus_speed *speed,
 			  enum pcie_link_width *width);
-void pcie_flr(struct pci_dev *dev);
+int pcie_flr(struct pci_dev *dev);
 int __pci_reset_function(struct pci_dev *dev);
 int __pci_reset_function_locked(struct pci_dev *dev);
 int pci_reset_function(struct pci_dev *dev);
-- 
1.9.1

[PATCH V2 2/7] PCI: handle FLR failure and allow other reset types

[Christoph Hellwig]

On Mon, Nov 27, 2017 at 01:20:23AM -0500, Sinan Kaya wrote:
> pci_flr_wait() and pci_af_flr() functions assume graceful return even
> though the device is inaccessible under error conditions.
> 
> Return -ENOTTY in error cases so that __pci_reset_function_locked() can
> try other reset types if AF_FLR/FLR reset fails.

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

[PATCH V2 3/7] PCI: make pci_flr_wait() generic and rename to pci_dev_wait()

[Sinan Kaya]

Rev 3.1 Sec 2.3.1 Request Handling Rules:
Valid reset conditions after which a device is permitted to return CRS
are:
* Cold, Warm, and Hot Resets,
* FLR
* A reset initiated in response to a D3hot to D0 uninitialized

Try to reuse FLR implementation towards other reset types.

Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
---
 drivers/pci/pci.c | 47 ++++++++++++++++++++++++++++++-----------------
 1 file changed, 30 insertions(+), 17 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 27ec45d..ed3c3bc 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -125,6 +125,9 @@ static int __init pcie_port_pm_setup(char *str)
 }
 __setup("pcie_port_pm=", pcie_port_pm_setup);
 
+/* time to wait after a reset for device to become responsive */
+#define PCIE_RESET_READY_POLL_MS 60000
+
 /**
  * pci_bus_max_busnr - returns maximum PCI bus number of given bus' children
  * @bus: pointer to PCI bus structure to search
@@ -3820,20 +3823,13 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
 }
 EXPORT_SYMBOL(pci_wait_for_pending_transaction);
 
-static int pci_flr_wait(struct pci_dev *dev)
+static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
 {
-	int delay = 1, timeout = 60000;
+	int delay = 1;
 	u32 id;
 
 	/*
-	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within
-	 * 100ms, but may silently discard requests while the FLR is in
-	 * progress.  Wait 100ms before trying to access the device.
-	 */
-	msleep(100);
-
-	/*
-	 * After 100ms, the device should not silently discard config
+	 * After reset, the device should not silently discard config
 	 * requests, but it may still indicate that it needs more time by
 	 * responding to them with CRS completions.  The Root Port will
 	 * generally synthesize ~0 data to complete the read (except when
@@ -3847,14 +3843,14 @@ static int pci_flr_wait(struct pci_dev *dev)
 	pci_read_config_dword(dev, PCI_COMMAND, &id);
 	while (id == ~0) {
 		if (delay > timeout) {
-			dev_warn(&dev->dev, "not ready %dms after FLR; giving up\n",
-				 100 + delay - 1);
+			dev_warn(&dev->dev, "not ready %dms after %s; giving up\n",
+				 delay - 1, reset_type);
 			return -ENOTTY;
 		}
 
 		if (delay > 1000)
-			dev_info(&dev->dev, "not ready %dms after FLR; waiting\n",
-				 100 + delay - 1);
+			dev_info(&dev->dev, "not ready %dms after %s; waiting\n",
+				 delay - 1, reset_type);
 
 		msleep(delay);
 		delay *= 2;
@@ -3862,7 +3858,8 @@ static int pci_flr_wait(struct pci_dev *dev)
 	}
 
 	if (delay > 1000)
-		dev_info(&dev->dev, "ready %dms after FLR\n", 100 + delay - 1);
+		dev_info(&dev->dev, "ready %dms after %s\n", delay - 1,
+			 reset_type);
 
 	return 0;
 }
@@ -3899,7 +3896,15 @@ int pcie_flr(struct pci_dev *dev)
 		dev_err(&dev->dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
 
 	pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_BCR_FLR);
-	return pci_flr_wait(dev);
+
+	/*
+	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within
+	 * 100ms, but may silently discard requests while the FLR is in
+	 * progress.  Wait 100ms before trying to access the device.
+	 */
+	msleep(100);
+
+	return pci_dev_wait(dev, "FLR", PCIE_RESET_READY_POLL_MS);
 }
 EXPORT_SYMBOL_GPL(pcie_flr);
 
@@ -3932,7 +3937,15 @@ static int pci_af_flr(struct pci_dev *dev, int probe)
 		dev_err(&dev->dev, "timed out waiting for pending transaction; performing AF function level reset anyway\n");
 
 	pci_write_config_byte(dev, pos + PCI_AF_CTRL, PCI_AF_CTRL_FLR);
-	return pci_flr_wait(dev);
+
+	/*
+	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within
+	 * 100ms, but may silently discard requests while the FLR is in
+	 * progress.  Wait 100ms before trying to access the device.
+	 */
+	msleep(100);
+
+	return pci_dev_wait(dev, "AF_FLR", PCIE_RESET_READY_POLL_MS);
 }
 
 /**
-- 
1.9.1

[PATCH V2 3/7] PCI: make pci_flr_wait() generic and rename to pci_dev_wait()

[Christoph Hellwig]

On Mon, Nov 27, 2017 at 01:20:24AM -0500, Sinan Kaya wrote:
> Rev 3.1 Sec 2.3.1 Request Handling Rules:
> Valid reset conditions after which a device is permitted to return CRS
> are:
> * Cold, Warm, and Hot Resets,
> * FLR
> * A reset initiated in response to a D3hot to D0 uninitialized
> 
> Try to reuse FLR implementation towards other reset types.

Should we keep a helper for the FLR wait?  If not I guess the reference
for AF_FLR should be the PCI spec, not the PCIE spec.

[PATCH V2 3/7] PCI: make pci_flr_wait() generic and rename to pci_dev_wait()

[Sinan Kaya]

On 11/29/2017 12:38 PM, Christoph Hellwig wrote:
> On Mon, Nov 27, 2017 at 01:20:24AM -0500, Sinan Kaya wrote:
>> Rev 3.1 Sec 2.3.1 Request Handling Rules:
>> Valid reset conditions after which a device is permitted to return CRS
>> are:
>> * Cold, Warm, and Hot Resets,
>> * FLR
>> * A reset initiated in response to a D3hot to D0 uninitialized
>>
>> Try to reuse FLR implementation towards other reset types.
> 
> Should we keep a helper for the FLR wait?  If not I guess the reference
> for AF_FLR should be the PCI spec, not the PCIE spec.
> 

Bjorn,

what is your opinion?

Sinan

-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

[PATCH V2 3/7] PCI: make pci_flr_wait() generic and rename to pci_dev_wait()

[Bjorn Helgaas]

On Mon, Nov 27, 2017 at 01:20:24AM -0500, Sinan Kaya wrote:
> Rev 3.1 Sec 2.3.1 Request Handling Rules:
> Valid reset conditions after which a device is permitted to return CRS
> are:
> * Cold, Warm, and Hot Resets,
> * FLR
> * A reset initiated in response to a D3hot to D0 uninitialized
> 
> Try to reuse FLR implementation towards other reset types.
> 
> Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
> ---
>  drivers/pci/pci.c | 47 ++++++++++++++++++++++++++++++-----------------
>  1 file changed, 30 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index 27ec45d..ed3c3bc 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -125,6 +125,9 @@ static int __init pcie_port_pm_setup(char *str)
>  }
>  __setup("pcie_port_pm=", pcie_port_pm_setup);
>  
> +/* time to wait after a reset for device to become responsive */
> +#define PCIE_RESET_READY_POLL_MS 60000
> +
>  /**
>   * pci_bus_max_busnr - returns maximum PCI bus number of given bus' children
>   * @bus: pointer to PCI bus structure to search
> @@ -3820,20 +3823,13 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
>  }
>  EXPORT_SYMBOL(pci_wait_for_pending_transaction);
>  
> -static int pci_flr_wait(struct pci_dev *dev)
> +static int pci_dev_wait(struct pci_dev *dev, char *reset_type, int timeout)
>  {
> -	int delay = 1, timeout = 60000;
> +	int delay = 1;
>  	u32 id;
>  
>  	/*
> -	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within
> -	 * 100ms, but may silently discard requests while the FLR is in
> -	 * progress.  Wait 100ms before trying to access the device.
> -	 */
> -	msleep(100);
> -
> -	/*
> -	 * After 100ms, the device should not silently discard config
> +	 * After reset, the device should not silently discard config
>  	 * requests, but it may still indicate that it needs more time by
>  	 * responding to them with CRS completions.  The Root Port will
>  	 * generally synthesize ~0 data to complete the read (except when
> @@ -3847,14 +3843,14 @@ static int pci_flr_wait(struct pci_dev *dev)
>  	pci_read_config_dword(dev, PCI_COMMAND, &id);
>  	while (id == ~0) {
>  		if (delay > timeout) {
> -			dev_warn(&dev->dev, "not ready %dms after FLR; giving up\n",
> -				 100 + delay - 1);
> +			dev_warn(&dev->dev, "not ready %dms after %s; giving up\n",
> +				 delay - 1, reset_type);
>  			return -ENOTTY;
>  		}
>  
>  		if (delay > 1000)
> -			dev_info(&dev->dev, "not ready %dms after FLR; waiting\n",
> -				 100 + delay - 1);
> +			dev_info(&dev->dev, "not ready %dms after %s; waiting\n",
> +				 delay - 1, reset_type);
>  
>  		msleep(delay);
>  		delay *= 2;
> @@ -3862,7 +3858,8 @@ static int pci_flr_wait(struct pci_dev *dev)
>  	}
>  
>  	if (delay > 1000)
> -		dev_info(&dev->dev, "ready %dms after FLR\n", 100 + delay - 1);
> +		dev_info(&dev->dev, "ready %dms after %s\n", delay - 1,
> +			 reset_type);
>  
>  	return 0;
>  }
> @@ -3899,7 +3896,15 @@ int pcie_flr(struct pci_dev *dev)
>  		dev_err(&dev->dev, "timed out waiting for pending transaction; performing function level reset anyway\n");
>  
>  	pcie_capability_set_word(dev, PCI_EXP_DEVCTL, PCI_EXP_DEVCTL_BCR_FLR);
> -	return pci_flr_wait(dev);
> +
> +	/*
> +	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within
> +	 * 100ms, but may silently discard requests while the FLR is in
> +	 * progress.  Wait 100ms before trying to access the device.
> +	 */
> +	msleep(100);
> +
> +	return pci_dev_wait(dev, "FLR", PCIE_RESET_READY_POLL_MS);
>  }
>  EXPORT_SYMBOL_GPL(pcie_flr);
>  
> @@ -3932,7 +3937,15 @@ static int pci_af_flr(struct pci_dev *dev, int probe)
>  		dev_err(&dev->dev, "timed out waiting for pending transaction; performing AF function level reset anyway\n");
>  
>  	pci_write_config_byte(dev, pos + PCI_AF_CTRL, PCI_AF_CTRL_FLR);
> -	return pci_flr_wait(dev);
> +
> +	/*
> +	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within

I think this should reference the "Advanced Capabilities for
Conventional PCI" ECN, shouldn't it?  The one I see is dated 13 April
2006, updated 27 July 2006, and I don't see a PCI spec that includes
it.

> +	 * 100ms, but may silently discard requests while the FLR is in
> +	 * progress.  Wait 100ms before trying to access the device.
> +	 */
> +	msleep(100);
> +
> +	return pci_dev_wait(dev, "AF_FLR", PCIE_RESET_READY_POLL_MS);

CRS is not applicable to conventional PCI.  The ECN mentions waiting
100ms.  I don't see anything about polling after that, but I guess it
probably doesn't hurt anything.

>  }
>  
>  /**
> -- 
> 1.9.1
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH V2 3/7] PCI: make pci_flr_wait() generic and rename to pci_dev_wait()

[Sinan Kaya]

On 12/13/2017 5:43 PM, Bjorn Helgaas wrote:
>> +
>> +	/*
>> +	 * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within
> I think this should reference the "Advanced Capabilities for
> Conventional PCI" ECN, shouldn't it?  The one I see is dated 13 April
> 2006, updated 27 July 2006, and I don't see a PCI spec that includes
> it.

I'll add reference to PCI spec as well. Any other comments on the series?

> 
>> +	 * 100ms, but may silently discard requests while the FLR is in
>> +	 * progress.  Wait 100ms before trying to access the device.
>> +	 */
>> +	msleep(100);
>> +
>> +	return pci_dev_wait(dev, "AF_FLR", PCIE_RESET_READY_POLL_MS);
> CRS is not applicable to conventional PCI.  The ECN mentions waiting
> 100ms.  I don't see anything about polling after that, but I guess it
> probably doesn't hurt anything.

ok, I'll keep it as it is.

> 
>>  }


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

[PATCH V2 4/7] PCI: wait device ready after pci_pm_reset()

[Sinan Kaya]

Rev 3.1 Sec 2.3.1 Request Handling Rules says a device can issue CRS
following a D3hot->D0 transition. Add pci_dev_wait() call to see if
device is available before returning.

Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
---
 drivers/pci/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index ed3c3bc..87e4688 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -3990,7 +3990,7 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)
 	pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr);
 	pci_dev_d3_sleep(dev);
 
-	return 0;
+	return pci_dev_wait(dev, "PM D3->D0", PCIE_RESET_READY_POLL_MS);
 }
 
 void pci_reset_secondary_bus(struct pci_dev *dev)
-- 
1.9.1

[PATCH V2 4/7] PCI: wait device ready after pci_pm_reset()

[Christoph Hellwig]

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

[PATCH V2 5/7] PCI: add a return type for pci_reset_bridge_secondary_bus()

[Sinan Kaya]

Getting ready to return an error from pci_reset_bridge_secondary_bus() when
device is unreachable.

Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
---
 drivers/pci/pci.c   | 4 +++-
 include/linux/pci.h | 2 +-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 87e4688..0a9a696 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -4031,9 +4031,11 @@ void __weak pcibios_reset_secondary_bus(struct pci_dev *dev)
  * Use the bridge control register to assert reset on the secondary bus.
  * Devices on the secondary bus are left in power-on state.
  */
-void pci_reset_bridge_secondary_bus(struct pci_dev *dev)
+int pci_reset_bridge_secondary_bus(struct pci_dev *dev)
 {
 	pcibios_reset_secondary_bus(dev);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(pci_reset_bridge_secondary_bus);
 
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 354b018..539b3af 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1103,7 +1103,7 @@ int pcie_get_minimum_link(struct pci_dev *dev, enum pci_bus_speed *speed,
 int pci_try_reset_bus(struct pci_bus *bus);
 void pci_reset_secondary_bus(struct pci_dev *dev);
 void pcibios_reset_secondary_bus(struct pci_dev *dev);
-void pci_reset_bridge_secondary_bus(struct pci_dev *dev);
+int pci_reset_bridge_secondary_bus(struct pci_dev *dev);
 void pci_update_resource(struct pci_dev *dev, int resno);
 int __must_check pci_assign_resource(struct pci_dev *dev, int i);
 int __must_check pci_reassign_resource(struct pci_dev *dev, int i, resource_size_t add_size, resource_size_t align);
-- 
1.9.1

[PATCH V2 5/7] PCI: add a return type for pci_reset_bridge_secondary_bus()

[Christoph Hellwig]

On Mon, Nov 27, 2017 at 01:20:26AM -0500, Sinan Kaya wrote:
> Getting ready to return an error from pci_reset_bridge_secondary_bus() when
> device is unreachable.
> 
> Signed-off-by: Sinan Kaya <okaya@codeaurora.org>

Looks ok, but I would just merge it into the next patch.

Reviewed-by: Christoph Hellwig <hch@lst.de>

[PATCH V2 6/7] PCI: add device wait after slot and bus reset

[Sinan Kaya]

Rev 3.1 Sec 2.3.1 Request Handling Rules indicates that a device can issue
CRS following secondary bus reset. Handle device presence gracefully.

Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
---
 drivers/pci/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 0a9a696..8472c24 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -4035,7 +4035,7 @@ int pci_reset_bridge_secondary_bus(struct pci_dev *dev)
 {
 	pcibios_reset_secondary_bus(dev);
 
-	return 0;
+	return pci_dev_wait(dev, "bus reset", PCIE_RESET_READY_POLL_MS);
 }
 EXPORT_SYMBOL_GPL(pci_reset_bridge_secondary_bus);
 
-- 
1.9.1

[PATCH V2 6/7] PCI: add device wait after slot and bus reset

[Christoph Hellwig]

On Mon, Nov 27, 2017 at 01:20:27AM -0500, Sinan Kaya wrote:
> Rev 3.1 Sec 2.3.1 Request Handling Rules indicates that a device can issue
> CRS following secondary bus reset. Handle device presence gracefully.
> 
> Signed-off-by: Sinan Kaya <okaya@codeaurora.org>

Looks good,

Reviewed-by: Christoph Hellwig <hch@lst.de>

[PATCH V2 7/7] PCI: make reset poll time adjustable

[Sinan Kaya]

Introduce pci=resetpolltime= argument to override 60 seconds poll time in
units of milliseconds.

Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
---
 Documentation/admin-guide/kernel-parameters.txt |  2 ++
 drivers/pci/pci.c                               | 13 ++++++++-----
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 0549662..a07d4f5 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -3071,6 +3071,8 @@
 		pcie_scan_all	Scan all possible PCIe devices.  Otherwise we
 				only look for one device below a PCIe downstream
 				port.
+		resetpolltime=	Adjusts the default poll time following hot reset
+				and D3->D0 transition.
 
 	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
 			Management.
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index 8472c24..a6c3e25 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -127,7 +127,7 @@ static int __init pcie_port_pm_setup(char *str)
 
 /* time to wait after a reset for device to become responsive */
 #define PCIE_RESET_READY_POLL_MS 60000
-
+static unsigned long pci_reset_polltime = PCIE_RESET_READY_POLL_MS;
 /**
  * pci_bus_max_busnr - returns maximum PCI bus number of given bus' children
  * @bus: pointer to PCI bus structure to search
@@ -3904,7 +3904,7 @@ int pcie_flr(struct pci_dev *dev)
 	 */
 	msleep(100);
 
-	return pci_dev_wait(dev, "FLR", PCIE_RESET_READY_POLL_MS);
+	return pci_dev_wait(dev, "FLR", pci_reset_polltime);
 }
 EXPORT_SYMBOL_GPL(pcie_flr);
 
@@ -3945,7 +3945,7 @@ static int pci_af_flr(struct pci_dev *dev, int probe)
 	 */
 	msleep(100);
 
-	return pci_dev_wait(dev, "AF_FLR", PCIE_RESET_READY_POLL_MS);
+	return pci_dev_wait(dev, "AF_FLR", pci_reset_polltime);
 }
 
 /**
@@ -3990,7 +3990,7 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)
 	pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr);
 	pci_dev_d3_sleep(dev);
 
-	return pci_dev_wait(dev, "PM D3->D0", PCIE_RESET_READY_POLL_MS);
+	return pci_dev_wait(dev, "PM D3->D0", pci_reset_polltime);
 }
 
 void pci_reset_secondary_bus(struct pci_dev *dev)
@@ -4035,7 +4035,7 @@ int pci_reset_bridge_secondary_bus(struct pci_dev *dev)
 {
 	pcibios_reset_secondary_bus(dev);
 
-	return pci_dev_wait(dev, "bus reset", PCIE_RESET_READY_POLL_MS);
+	return pci_dev_wait(dev, "bus reset", pci_reset_polltime);
 }
 EXPORT_SYMBOL_GPL(pci_reset_bridge_secondary_bus);
 
@@ -5528,6 +5528,9 @@ static int __init pci_setup(char *str)
 				pcie_bus_config = PCIE_BUS_PEER2PEER;
 			} else if (!strncmp(str, "pcie_scan_all", 13)) {
 				pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
+			} else if (!strncmp(str, "resetpolltime=", 14)) {
+				pci_reset_polltime =
+					simple_strtoul(str + 14, &str, 0);
 			} else {
 				printk(KERN_ERR "PCI: Unknown option `%s'\n",
 						str);
-- 
1.9.1

[PATCH V2 7/7] PCI: make reset poll time adjustable

[Bjorn Helgaas]

On Mon, Nov 27, 2017 at 01:20:28AM -0500, Sinan Kaya wrote:
> Introduce pci=resetpolltime= argument to override 60 seconds poll time in
> units of milliseconds.

I resist adding kernel parameters because they really complicate the
user experience.  Obviously you added this for a reason, but I don't
know what that is.  If we really need it, is there any way we could
automatically figure out in the kernel when we need different poll
times?

> Signed-off-by: Sinan Kaya <okaya@codeaurora.org>
> ---
>  Documentation/admin-guide/kernel-parameters.txt |  2 ++
>  drivers/pci/pci.c                               | 13 ++++++++-----
>  2 files changed, 10 insertions(+), 5 deletions(-)
> 
> diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> index 0549662..a07d4f5 100644
> --- a/Documentation/admin-guide/kernel-parameters.txt
> +++ b/Documentation/admin-guide/kernel-parameters.txt
> @@ -3071,6 +3071,8 @@
>  		pcie_scan_all	Scan all possible PCIe devices.  Otherwise we
>  				only look for one device below a PCIe downstream
>  				port.
> +		resetpolltime=	Adjusts the default poll time following hot reset
> +				and D3->D0 transition.
>  
>  	pcie_aspm=	[PCIE] Forcibly enable or disable PCIe Active State Power
>  			Management.
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index 8472c24..a6c3e25 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -127,7 +127,7 @@ static int __init pcie_port_pm_setup(char *str)
>  
>  /* time to wait after a reset for device to become responsive */
>  #define PCIE_RESET_READY_POLL_MS 60000
> -
> +static unsigned long pci_reset_polltime = PCIE_RESET_READY_POLL_MS;
>  /**
>   * pci_bus_max_busnr - returns maximum PCI bus number of given bus' children
>   * @bus: pointer to PCI bus structure to search
> @@ -3904,7 +3904,7 @@ int pcie_flr(struct pci_dev *dev)
>  	 */
>  	msleep(100);
>  
> -	return pci_dev_wait(dev, "FLR", PCIE_RESET_READY_POLL_MS);
> +	return pci_dev_wait(dev, "FLR", pci_reset_polltime);
>  }
>  EXPORT_SYMBOL_GPL(pcie_flr);
>  
> @@ -3945,7 +3945,7 @@ static int pci_af_flr(struct pci_dev *dev, int probe)
>  	 */
>  	msleep(100);
>  
> -	return pci_dev_wait(dev, "AF_FLR", PCIE_RESET_READY_POLL_MS);
> +	return pci_dev_wait(dev, "AF_FLR", pci_reset_polltime);
>  }
>  
>  /**
> @@ -3990,7 +3990,7 @@ static int pci_pm_reset(struct pci_dev *dev, int probe)
>  	pci_write_config_word(dev, dev->pm_cap + PCI_PM_CTRL, csr);
>  	pci_dev_d3_sleep(dev);
>  
> -	return pci_dev_wait(dev, "PM D3->D0", PCIE_RESET_READY_POLL_MS);
> +	return pci_dev_wait(dev, "PM D3->D0", pci_reset_polltime);
>  }
>  
>  void pci_reset_secondary_bus(struct pci_dev *dev)
> @@ -4035,7 +4035,7 @@ int pci_reset_bridge_secondary_bus(struct pci_dev *dev)
>  {
>  	pcibios_reset_secondary_bus(dev);
>  
> -	return pci_dev_wait(dev, "bus reset", PCIE_RESET_READY_POLL_MS);
> +	return pci_dev_wait(dev, "bus reset", pci_reset_polltime);
>  }
>  EXPORT_SYMBOL_GPL(pci_reset_bridge_secondary_bus);
>  
> @@ -5528,6 +5528,9 @@ static int __init pci_setup(char *str)
>  				pcie_bus_config = PCIE_BUS_PEER2PEER;
>  			} else if (!strncmp(str, "pcie_scan_all", 13)) {
>  				pci_add_flags(PCI_SCAN_ALL_PCIE_DEVS);
> +			} else if (!strncmp(str, "resetpolltime=", 14)) {
> +				pci_reset_polltime =
> +					simple_strtoul(str + 14, &str, 0);
>  			} else {
>  				printk(KERN_ERR "PCI: Unknown option `%s'\n",
>  						str);
> -- 
> 1.9.1
> 

[PATCH V2 7/7] PCI: make reset poll time adjustable

[Sinan Kaya]

On 11/28/2017 9:20 AM, Bjorn Helgaas wrote:
>> Introduce pci=resetpolltime= argument to override 60 seconds poll time in
>> units of milliseconds.
> I resist adding kernel parameters because they really complicate the
> user experience.  Obviously you added this for a reason, but I don't
> know what that is.  If we really need it, is there any way we could
> automatically figure out in the kernel when we need different poll
> times?
> 

No, we can drop this patch if we want to . This was mostly a
scalability concern. 

I worried adding 60 seconds would be too much for some use case. We can
hold onto this change until that use case happens if you want.

-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.