[PATCH v3 0/3] powerpc/eeh: Release EEH device state synchronously

[PATCH v3 0/3] powerpc/eeh: Release EEH device state synchronously

[Sam Bobroff]

Hi everyone,

Here are some fixes and cleanups that have come from other work but that I
think stand on their own.

Only one patch ("Release EEH device state synchronously", suggested by Oliver
O'Halloran) is a significant change: it moves the cleanup of some EEH device
data out of the (possibly asynchronous) device release handler and into the
(synchronously called) bus notifier. This is useful for future work as it makes
it easier to reason about the lifetimes of EEH structures.

Note that I've left a few WARN_ON_ONCEs in the code because I'm paranoid, but I
have not been able to hit them during testing.

Cheers,
Sam.

Notes for v3:
I've tweaked the fix for pseries_eeh_configure_bridge() to return the correct
error code (even though it's not used) by calling an already present RTAS
function, rtas_error_rc(). However, I had to make another change to export that
function and while it does seem like the right thing to do, but I'm concerned
it's a bit out of scope for such a small fix.

Notes for v2:

I've dropped both cleanup patches (3/4, 4/4) because that type of cleanup
(replacing a call to eeh_rmv_from_parent_pe() with one to eeh_remove_device())
is incorrect: if called during recovery, it will cause edev->pe to remain set
when it would have been cleared previously. This would lead to stale
information in the edev. I think there should be a way to simplify the code
around EEH_PE_KEEP but I'll look at that separately.

Patch set changelog follows:

Patch set v3: 
Patch 1/3 (new in this version): powerpc/rtas: Export rtas_error_rc
Patch 2/3 (was 1/2): powerpc/eeh: fix pseries_eeh_configure_bridge()
Patch 3/3 (was 2/2): powerpc/eeh: Release EEH device state synchronously

Patch set v2: 
Patch 1/2: powerpc/eeh: fix pseries_eeh_configure_bridge()
Patch 2/2: powerpc/eeh: Release EEH device state synchronously
- Added comment explaining why the add case can't be handled similarly to the remove case.
Dropped (was 3/4) powerpc/eeh: Remove workaround from eeh_add_device_late()
Dropped (was 4/4) powerpc/eeh: Clean up edev cleanup for VFs

Patch set v1:
Patch 1/4: powerpc/eeh: fix pseries_eeh_configure_bridge()
Patch 2/4: powerpc/eeh: Release EEH device state synchronously
Patch 3/4: powerpc/eeh: Remove workaround from eeh_add_device_late()
Patch 4/4: powerpc/eeh: Clean up edev cleanup for VFs

Sam Bobroff (3):
  powerpc/rtas: Export rtas_error_rc
  powerpc/eeh: fix pseries_eeh_configure_bridge()
  powerpc/eeh: Release EEH device state synchronously

 arch/powerpc/include/asm/rtas.h              |  1 +
 arch/powerpc/kernel/eeh.c                    | 31 ++++++++++++++++++++
 arch/powerpc/kernel/pci-hotplug.c            |  2 --
 arch/powerpc/kernel/rtas.c                   |  3 +-
 arch/powerpc/platforms/pseries/eeh_pseries.c |  4 ++-
 5 files changed, 37 insertions(+), 4 deletions(-)

-- 
2.22.0.216.g00a2a96fc9

[PATCH v3 1/3] powerpc/rtas: Export rtas_error_rc

[Sam Bobroff]

Export rtas_error_rc() so that it can be used by other users of
rtas_call() (which is already exported).

Signed-off-by: Sam Bobroff <sbobroff@linux.ibm.com>
---
v3 * New in this version.

 arch/powerpc/include/asm/rtas.h | 1 +
 arch/powerpc/kernel/rtas.c      | 3 ++-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/include/asm/rtas.h b/arch/powerpc/include/asm/rtas.h
index 3c1887351c71..7c9e4d3635cf 100644
--- a/arch/powerpc/include/asm/rtas.h
+++ b/arch/powerpc/include/asm/rtas.h
@@ -379,6 +379,7 @@ extern int rtas_set_rtc_time(struct rtc_time *rtc_time);
 
 extern unsigned int rtas_busy_delay_time(int status);
 extern unsigned int rtas_busy_delay(int status);
+extern int rtas_error_rc(int rtas_rc);
 
 extern int early_init_dt_scan_rtas(unsigned long node,
 		const char *uname, int depth, void *data);
diff --git a/arch/powerpc/kernel/rtas.c b/arch/powerpc/kernel/rtas.c
index c5fa251b8950..238bf112d29a 100644
--- a/arch/powerpc/kernel/rtas.c
+++ b/arch/powerpc/kernel/rtas.c
@@ -518,7 +518,7 @@ unsigned int rtas_busy_delay(int status)
 }
 EXPORT_SYMBOL(rtas_busy_delay);
 
-static int rtas_error_rc(int rtas_rc)
+int rtas_error_rc(int rtas_rc)
 {
 	int rc;
 
@@ -546,6 +546,7 @@ static int rtas_error_rc(int rtas_rc)
 	}
 	return rc;
 }
+EXPORT_SYMBOL(rtas_error_rc);
 
 int rtas_get_power_level(int powerdomain, int *level)
 {
-- 
2.22.0.216.g00a2a96fc9

Re: [PATCH v3 1/3] powerpc/rtas: Export rtas_error_rc

[Nathan Lynch]

Sam Bobroff <sbobroff@linux.ibm.com> writes:
> Export rtas_error_rc() so that it can be used by other users of
> rtas_call() (which is already exported).

This will do the right thing for your ibm,configure-pe use case in patch
2, but the -900x => errno translations in rtas_error_rc() appear
tailored for the indicator- and sensor-related calls that currently use
it. From my reading of PAPR+, the meaning of a -900x RTAS status word
depends on the call. For example, -9002 commonly means "not authorized",
which we would typically translate to -EPERM, but rtas_error_rc() would
translate it to -ENODEV.

Also the semantics of -9001 as a return value seem to vary a bit.

So I don't think rtas_error_rc() should be advertised as a generically
useful facility in its current form.

(I have had some thoughts about how firmware/hypervisor call status can
be translated to meaningful Linux error values without tedious switch
statements, which I'm happy to expand on if anyone is interested, but I
don't want to hijack your submission for that discussion.)

Re: [PATCH v3 1/3] powerpc/rtas: Export rtas_error_rc

[Sam Bobroff]

[-- Attachment #1: Type: text/plain, Size: 1261 bytes --]

On Fri, Apr 24, 2020 at 11:07:43AM -0500, Nathan Lynch wrote:
> Sam Bobroff <sbobroff@linux.ibm.com> writes:
> > Export rtas_error_rc() so that it can be used by other users of
> > rtas_call() (which is already exported).
> 
> This will do the right thing for your ibm,configure-pe use case in patch
> 2, but the -900x => errno translations in rtas_error_rc() appear
> tailored for the indicator- and sensor-related calls that currently use
> it. From my reading of PAPR+, the meaning of a -900x RTAS status word
> depends on the call. For example, -9002 commonly means "not authorized",
> which we would typically translate to -EPERM, but rtas_error_rc() would
> translate it to -ENODEV.
> 
> Also the semantics of -9001 as a return value seem to vary a bit.
> 
> So I don't think rtas_error_rc() should be advertised as a generically
> useful facility in its current form.
> 
> (I have had some thoughts about how firmware/hypervisor call status can
> be translated to meaningful Linux error values without tedious switch
> statements, which I'm happy to expand on if anyone is interested, but I
> don't want to hijack your submission for that discussion.)

Ah, interesting.

I'll do another version as you suggest.

Cheers,
Sam.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH v3 1/3] powerpc/rtas: Export rtas_error_rc

[Michael Ellerman]

Sam Bobroff <sbobroff@linux.ibm.com> writes:
> Export rtas_error_rc() so that it can be used by other users of
> rtas_call() (which is already exported).
>
> Signed-off-by: Sam Bobroff <sbobroff@linux.ibm.com>
> ---
> v3 * New in this version.
>
>  arch/powerpc/include/asm/rtas.h | 1 +
>  arch/powerpc/kernel/rtas.c      | 3 ++-
>  2 files changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/include/asm/rtas.h b/arch/powerpc/include/asm/rtas.h
> index 3c1887351c71..7c9e4d3635cf 100644
> --- a/arch/powerpc/include/asm/rtas.h
> +++ b/arch/powerpc/include/asm/rtas.h
> @@ -379,6 +379,7 @@ extern int rtas_set_rtc_time(struct rtc_time *rtc_time);
>  
>  extern unsigned int rtas_busy_delay_time(int status);
>  extern unsigned int rtas_busy_delay(int status);
> +extern int rtas_error_rc(int rtas_rc);
>  
>  extern int early_init_dt_scan_rtas(unsigned long node,
>  		const char *uname, int depth, void *data);
> diff --git a/arch/powerpc/kernel/rtas.c b/arch/powerpc/kernel/rtas.c
> index c5fa251b8950..238bf112d29a 100644
> --- a/arch/powerpc/kernel/rtas.c
> +++ b/arch/powerpc/kernel/rtas.c
> @@ -518,7 +518,7 @@ unsigned int rtas_busy_delay(int status)
>  }
>  EXPORT_SYMBOL(rtas_busy_delay);
>  
> -static int rtas_error_rc(int rtas_rc)
> +int rtas_error_rc(int rtas_rc)
>  {
>  	int rc;
>  
> @@ -546,6 +546,7 @@ static int rtas_error_rc(int rtas_rc)
>  	}
>  	return rc;
>  }
> +EXPORT_SYMBOL(rtas_error_rc);

Will it be used in a module somewhere?

AFAICS the only caller you add is built-in.

cheers

[PATCH v3 2/3] powerpc/eeh: fix pseries_eeh_configure_bridge()

[Sam Bobroff]

If a device is hot unplgged during EEH recovery, it's possible for the
RTAS call to ibm,configure-pe in pseries_eeh_configure() to return
parameter error (-3), however negative return values are not checked
for and this leads to an infinite loop.

Fix this by correctly bailing out on negative values.

Signed-off-by: Sam Bobroff <sbobroff@linux.ibm.com>
---
 arch/powerpc/platforms/pseries/eeh_pseries.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c
index 893ba3f562c4..9ea1c06a78cd 100644
--- a/arch/powerpc/platforms/pseries/eeh_pseries.c
+++ b/arch/powerpc/platforms/pseries/eeh_pseries.c
@@ -607,6 +607,8 @@ static int pseries_eeh_configure_bridge(struct eeh_pe *pe)
 
 		if (!ret)
 			return ret;
+		if (ret < 0)
+			break;
 
 		/*
 		 * If RTAS returns a delay value that's above 100ms, cut it
@@ -627,7 +629,7 @@ static int pseries_eeh_configure_bridge(struct eeh_pe *pe)
 
 	pr_warn("%s: Unable to configure bridge PHB#%x-PE#%x (%d)\n",
 		__func__, pe->phb->global_number, pe->addr, ret);
-	return ret;
+	return rtas_error_rc(ret);
 }
 
 /**
-- 
2.22.0.216.g00a2a96fc9

Re: [PATCH v3 2/3] powerpc/eeh: fix pseries_eeh_configure_bridge()

[Nathan Lynch]

Sam Bobroff <sbobroff@linux.ibm.com> writes:
> If a device is hot unplgged during EEH recovery, it's possible for the
> RTAS call to ibm,configure-pe in pseries_eeh_configure() to return
> parameter error (-3), however negative return values are not checked
> for and this leads to an infinite loop.
>
> Fix this by correctly bailing out on negative values.
>
> Signed-off-by: Sam Bobroff <sbobroff@linux.ibm.com>
> ---
>  arch/powerpc/platforms/pseries/eeh_pseries.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/platforms/pseries/eeh_pseries.c b/arch/powerpc/platforms/pseries/eeh_pseries.c
> index 893ba3f562c4..9ea1c06a78cd 100644
> --- a/arch/powerpc/platforms/pseries/eeh_pseries.c
> +++ b/arch/powerpc/platforms/pseries/eeh_pseries.c
> @@ -607,6 +607,8 @@ static int pseries_eeh_configure_bridge(struct eeh_pe *pe)
>  
>  		if (!ret)
>  			return ret;
> +		if (ret < 0)
> +			break;
>  
>  		/*
>  		 * If RTAS returns a delay value that's above 100ms, cut it
> @@ -627,7 +629,7 @@ static int pseries_eeh_configure_bridge(struct eeh_pe *pe)
>  
>  	pr_warn("%s: Unable to configure bridge PHB#%x-PE#%x (%d)\n",
>  		__func__, pe->phb->global_number, pe->addr, ret);
> -	return ret;
> +	return rtas_error_rc(ret);

See my response to patch #1. I think the best you can do here right now
is return -EINVAL for -3, -EIO for any other (unarchitected) negative
value.

[PATCH v3 3/3] powerpc/eeh: Release EEH device state synchronously

[Sam Bobroff]

EEH device state is currently removed (by eeh_remove_device()) during
the device release handler, which is invoked as the device's reference
count drops to zero. This may take some time, or forever, as other
threads may hold references.

However, the PCI device state is released synchronously by
pci_stop_and_remove_bus_device(). This mismatch causes problems, for
example the device may be re-discovered as a new device before the
release handler has been called, leaving the PCI and EEH state
mismatched.

So instead, call eeh_remove_device() from the bus device removal
handlers, which are called synchronously in the removal path.

Signed-off-by: Sam Bobroff <sbobroff@linux.ibm.com>
---
 arch/powerpc/kernel/eeh.c         | 31 +++++++++++++++++++++++++++++++
 arch/powerpc/kernel/pci-hotplug.c |  2 --
 2 files changed, 31 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/kernel/eeh.c b/arch/powerpc/kernel/eeh.c
index 17cb3e9b5697..64361311bc8e 100644
--- a/arch/powerpc/kernel/eeh.c
+++ b/arch/powerpc/kernel/eeh.c
@@ -1106,6 +1106,37 @@ static int eeh_init(void)
 
 core_initcall_sync(eeh_init);
 
+static int eeh_device_notifier(struct notifier_block *nb,
+			       unsigned long action, void *data)
+{
+	struct device *dev = data;
+
+	switch (action) {
+	/*
+	 * Note: It's not possible to perform EEH device addition (i.e.
+	 * {pseries,pnv}_pcibios_bus_add_device()) here because it depends on
+	 * the device's resources, which have not yet been set up.
+	 */
+	case BUS_NOTIFY_DEL_DEVICE:
+		eeh_remove_device(to_pci_dev(dev));
+		break;
+	default:
+		break;
+	}
+	return NOTIFY_DONE;
+}
+
+static struct notifier_block eeh_device_nb = {
+	.notifier_call = eeh_device_notifier,
+};
+
+static __init int eeh_set_bus_notifier(void)
+{
+	bus_register_notifier(&pci_bus_type, &eeh_device_nb);
+	return 0;
+}
+arch_initcall(eeh_set_bus_notifier);
+
 /**
  * eeh_add_device_early - Enable EEH for the indicated device node
  * @pdn: PCI device node for which to set up EEH
diff --git a/arch/powerpc/kernel/pci-hotplug.c b/arch/powerpc/kernel/pci-hotplug.c
index d6a67f814983..28e9aa274f64 100644
--- a/arch/powerpc/kernel/pci-hotplug.c
+++ b/arch/powerpc/kernel/pci-hotplug.c
@@ -57,8 +57,6 @@ void pcibios_release_device(struct pci_dev *dev)
 	struct pci_controller *phb = pci_bus_to_host(dev->bus);
 	struct pci_dn *pdn = pci_get_pdn(dev);
 
-	eeh_remove_device(dev);
-
 	if (phb->controller_ops.release_device)
 		phb->controller_ops.release_device(dev);
 
-- 
2.22.0.216.g00a2a96fc9