[PATCH] PCI/AER: Prevent runtime power management during recovery

[PATCH] PCI/AER: Prevent runtime power management during recovery

[Keith Busch]

A bridge that supports D3 but not hotplug will be subject to runtime
power management placing it in a non-operation power state if it doesn't
have any devices attached. This patch will prevent this power management
during error recovery so that the rescan at the end may be successful.

Cc: Oza Pawandeep <poza@codeaurora.org>
Signed-off-by: Keith Busch <keith.busch@intel.com>
---
 drivers/pci/pcie/err.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/pci/pcie/err.c b/drivers/pci/pcie/err.c
index f7ce0cb0b0b7..247b6ce14f0d 100644
--- a/drivers/pci/pcie/err.c
+++ b/drivers/pci/pcie/err.c
@@ -16,6 +16,7 @@
 #include <linux/kernel.h>
 #include <linux/errno.h>
 #include <linux/aer.h>
+#include <linux/pm_runtime.h>
 #include "portdrv.h"
 #include "../pci.h"
 
@@ -294,6 +295,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service)
 		udev = dev->bus->self;
 
 	parent = udev->subordinate;
+	pm_runtime_forbid(&udev->dev);
 	pci_lock_rescan_remove();
 	list_for_each_entry_safe_reverse(pdev, temp, &parent->devices,
 					 bus_list) {
@@ -329,6 +331,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service)
 	}
 
 	pci_unlock_rescan_remove();
+	pm_runtime_allow(&udev->dev);
 }
 
 /**
-- 
2.14.3

Re: [PATCH] PCI/AER: Prevent runtime power management during recovery

[Sinan Kaya]

On 6/11/2018 6:29 PM, Keith Busch wrote:
> A bridge that supports D3 but not hotplug will be subject to runtime
> power management placing it in a non-operation power state if it doesn't
> have any devices attached. This patch will prevent this power management
> during error recovery so that the rescan at the end may be successful.

If there is no card connected, why would the bridge observe a fatal error?


-- 
Sinan Kaya
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.

Re: [PATCH] PCI/AER: Prevent runtime power management during recovery

[poza]

On 2018-06-12 10:10, Sinan Kaya wrote:
> On 6/11/2018 6:29 PM, Keith Busch wrote:
>> A bridge that supports D3 but not hotplug will be subject to runtime
>> power management placing it in a non-operation power state if it 
>> doesn't
>> have any devices attached. This patch will prevent this power 
>> management
>> during error recovery so that the rescan at the end may be successful.
> 
> If there is no card connected, why would the bridge observe a fatal 
> error?

fatal error could be coming from anywhere let us say below RootPort..
and RP observes it and decides to take the tree down....
Is that the case Keith is talking about ?

Why will re-enumeration be a problem even if runtime PM is active ?
I assume that enumeration will get bridge out of D3.

Re: [PATCH] PCI/AER: Prevent runtime power management during recovery

[Keith Busch]

On Tue, Jun 12, 2018 at 01:21:53PM +0530, poza@codeaurora.org wrote:
> On 2018-06-12 10:10, Sinan Kaya wrote:
> > On 6/11/2018 6:29 PM, Keith Busch wrote:
> > > A bridge that supports D3 but not hotplug will be subject to runtime
> > > power management placing it in a non-operation power state if it
> > > doesn't
> > > have any devices attached. This patch will prevent this power
> > > management
> > > during error recovery so that the rescan at the end may be successful.
> > 
> > If there is no card connected, why would the bridge observe a fatal
> > error?
> 
> fatal error could be coming from anywhere let us say below RootPort..
> and RP observes it and decides to take the tree down....
> Is that the case Keith is talking about ?

Right, the err fatal handling removes all the devices below a bridge,
making that bridge allowed for run time d3.
 
> Why will re-enumeration be a problem even if runtime PM is active ?
> I assume that enumeration will get bridge out of D3.

That doesn't seem to be the case.

Re: [PATCH] PCI/AER: Prevent runtime power management during recovery

[poza]

On 2018-06-12 20:14, Keith Busch wrote:
> On Tue, Jun 12, 2018 at 01:21:53PM +0530, poza@codeaurora.org wrote:
>> On 2018-06-12 10:10, Sinan Kaya wrote:
>> > On 6/11/2018 6:29 PM, Keith Busch wrote:
>> > > A bridge that supports D3 but not hotplug will be subject to runtime
>> > > power management placing it in a non-operation power state if it
>> > > doesn't
>> > > have any devices attached. This patch will prevent this power
>> > > management
>> > > during error recovery so that the rescan at the end may be successful.
>> >
>> > If there is no card connected, why would the bridge observe a fatal
>> > error?
>> 
>> fatal error could be coming from anywhere let us say below RootPort..
>> and RP observes it and decides to take the tree down....
>> Is that the case Keith is talking about ?
> 
> Right, the err fatal handling removes all the devices below a bridge,
> making that bridge allowed for run time d3.
> 
>> Why will re-enumeration be a problem even if runtime PM is active ?
>> I assume that enumeration will get bridge out of D3.
> 
> That doesn't seem to be the case.

reset_link(udev, service); will initiate SBR, and which should do hot 
reset the bridge and bring it out of D3 !
but it seems even SBR is not doing it for you.

Although I am not sure if the EP has to be designed to reset its config 
space upon SBR.. even if they are, some just might not do it all 
correctly !

The code looks okay to me anyway.
Reviewed-by: Oza Pawandeep <poza@codeaurora.org>

Re: [PATCH] PCI/AER: Prevent runtime power management during recovery

[Bjorn Helgaas]

[+cc Rafael, linux-pm for a PM perspective]

On Mon, Jun 11, 2018 at 04:29:18PM -0600, Keith Busch wrote:
> A bridge that supports D3 but not hotplug will be subject to runtime
> power management placing it in a non-operation power state if it doesn't
> have any devices attached. This patch will prevent this power management
> during error recovery so that the rescan at the end may be successful.

I assume you're seeing a problem that's fixed by this patch?

I'm not very familiar with the pm_runtime_*() interfaces.  The fact
that drivers/pci/ only contains two calls to pm_runtime_forbid() and
two to pm_runtime_allow() (one in a quirk) makes me wonder if this is
the right approach.

Do we need something similar in other enumeration paths, e.g.,
boot-time enumeration, hot-add, sysfs "rescan" entry, etc?

> Cc: Oza Pawandeep <poza@codeaurora.org>
> Signed-off-by: Keith Busch <keith.busch@intel.com>
> ---
>  drivers/pci/pcie/err.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/pci/pcie/err.c b/drivers/pci/pcie/err.c
> index f7ce0cb0b0b7..247b6ce14f0d 100644
> --- a/drivers/pci/pcie/err.c
> +++ b/drivers/pci/pcie/err.c
> @@ -16,6 +16,7 @@
>  #include <linux/kernel.h>
>  #include <linux/errno.h>
>  #include <linux/aer.h>
> +#include <linux/pm_runtime.h>
>  #include "portdrv.h"
>  #include "../pci.h"
>  
> @@ -294,6 +295,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service)
>  		udev = dev->bus->self;
>  
>  	parent = udev->subordinate;
> +	pm_runtime_forbid(&udev->dev);
>  	pci_lock_rescan_remove();
>  	list_for_each_entry_safe_reverse(pdev, temp, &parent->devices,
>  					 bus_list) {
> @@ -329,6 +331,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service)
>  	}
>  
>  	pci_unlock_rescan_remove();
> +	pm_runtime_allow(&udev->dev);
>  }
>  
>  /**
> -- 
> 2.14.3
> 

Re: [PATCH] PCI/AER: Prevent runtime power management during recovery

[Lukas Wunner]

On Sat, Jun 30, 2018 at 02:58:27PM -0500, Bjorn Helgaas wrote:
> On Mon, Jun 11, 2018 at 04:29:18PM -0600, Keith Busch wrote:
> > A bridge that supports D3 but not hotplug will be subject to runtime
> > power management placing it in a non-operation power state if it doesn't
> > have any devices attached. This patch will prevent this power management
> > during error recovery so that the rescan at the end may be successful.
> 
> I assume you're seeing a problem that's fixed by this patch?
> 
> I'm not very familiar with the pm_runtime_*() interfaces.  The fact
> that drivers/pci/ only contains two calls to pm_runtime_forbid() and
> two to pm_runtime_allow() (one in a quirk) makes me wonder if this is
> the right approach.

The use of pm_runtime_forbid() / pm_runtime_allow() is wrong here.
The user may forbid and allow runtime PM on a device by writing "on"
or "auto" to its power/control file in sysfs.  The two functions have
the same effect, but the user may override them at any time.  If the
intention is to ensure the bridge is runtime resumed while a rescan
is in progess, the appropriate functions to call are pm_runtime_get_sync()
and pm_runtime_put().


> Do we need something similar in other enumeration paths, e.g.,
> boot-time enumeration, hot-add, sysfs "rescan" entry, etc?

Definitely not on boot-time enumeration.  PCI devices are kept in D0
if not bound to a driver, and the PCI core first scans all devices,
only then attaches drivers.

As for hot-add, hotplug bridges are currently always kept in D0,
but I've posted a patch series two weeks ago to add runtime PM.
Did you get a chance to look at it?  Patch [28/32] of that series
resumes the bridge to D0 on hot-add ("PCI: pciehp: Resume to D0 on
enable/disable").

As for sysfs "rescan", this does lack runtime PM calls currently,
as do many other sysfs accessors.  The above-mentioned series
adds runtime PM to sysfs "reset" in patch [30/32] ("PCI: sysfs:
Resume to D0 on function reset").  The patch does not depend on the
preceding patches in the series, it could be applied out of order.
Amending all the other accessors would be another big project.
It's not as trivial as sprinkling a few runtime PM calls, it requires
careful thought.  E.g. if only config space is accessed, it's
sufficient to resume the parent with pci_config_pm_runtime_get()
instead of the device itself with pm_runtime_get_sync().  In other
cases the information is read from config space even though it's cached,
e.g. secondary_bus_number_show().  In those cases it might be better
to always return the cached data instead of adding runtime PM calls.

FWIW, from a quick look proc.c seems fine as far as runtime PM is
concerned.

Thanks,

Lukas

> 
> > Cc: Oza Pawandeep <poza@codeaurora.org>
> > Signed-off-by: Keith Busch <keith.busch@intel.com>
> > ---
> >  drivers/pci/pcie/err.c | 3 +++
> >  1 file changed, 3 insertions(+)
> > 
> > diff --git a/drivers/pci/pcie/err.c b/drivers/pci/pcie/err.c
> > index f7ce0cb0b0b7..247b6ce14f0d 100644
> > --- a/drivers/pci/pcie/err.c
> > +++ b/drivers/pci/pcie/err.c
> > @@ -16,6 +16,7 @@
> >  #include <linux/kernel.h>
> >  #include <linux/errno.h>
> >  #include <linux/aer.h>
> > +#include <linux/pm_runtime.h>
> >  #include "portdrv.h"
> >  #include "../pci.h"
> >  
> > @@ -294,6 +295,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service)
> >  		udev = dev->bus->self;
> >  
> >  	parent = udev->subordinate;
> > +	pm_runtime_forbid(&udev->dev);
> >  	pci_lock_rescan_remove();
> >  	list_for_each_entry_safe_reverse(pdev, temp, &parent->devices,
> >  					 bus_list) {
> > @@ -329,6 +331,7 @@ void pcie_do_fatal_recovery(struct pci_dev *dev, u32 service)
> >  	}
> >  
> >  	pci_unlock_rescan_remove();
> > +	pm_runtime_allow(&udev->dev);
> >  }
> >  
> >  /**
> > -- 
> > 2.14.3