[RFC] powerpc/powernv/mce: Don't silently restart the machine

[RFC] powerpc/powernv/mce: Don't silently restart the machine

[Balbir Singh]

On MCE the current code will restart the machine with
ppc_md.restart(). This case was extremely unlikely since
prior to that a skiboot call is made and that resulted in
a checkstop for analysis.

With newer skiboots, on P9 we don't checkstop the box by
default, instead we return back to the kernel to extract
useful information at the time of the MCE. While we still
get this information, this patch converts the restart to
a panic(), so that if configured a dump can be taken and
we can track and probably debug the potential issue causing
the MCE.

Signed-off-by: Balbir Singh <bsingharora@gmail.com>
---
 arch/powerpc/platforms/powernv/opal.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/powerpc/platforms/powernv/opal.c b/arch/powerpc/platforms/powernv/opal.c
index 69b5263fc9e3..b510a6f41b00 100644
--- a/arch/powerpc/platforms/powernv/opal.c
+++ b/arch/powerpc/platforms/powernv/opal.c
@@ -500,9 +500,12 @@ void pnv_platform_error_reboot(struct pt_regs *regs, const char *msg)
 	 *    opal to trigger checkstop explicitly for error analysis.
 	 *    The FSP PRD component would have already got notified
 	 *    about this error through other channels.
+	 * 4. We are running on a newer skiboot that by default does
+	 *    not cause a checkstop, drops us back to the kernel to
+	 *    extract context and state at the time of the error.
 	 */
 
-	ppc_md.restart(NULL);
+	panic("PowerNV Unrecovered Machine Check");
 }
 
 int opal_machine_check(struct pt_regs *regs)
-- 
2.13.6

Re: [RFC] powerpc/powernv/mce: Don't silently restart the machine

[Nicholas Piggin]

On Wed, 21 Feb 2018 12:01:11 +1100
Balbir Singh <bsingharora@gmail.com> wrote:

> On MCE the current code will restart the machine with
> ppc_md.restart(). This case was extremely unlikely since
> prior to that a skiboot call is made and that resulted in
> a checkstop for analysis.
> 
> With newer skiboots, on P9 we don't checkstop the box by
> default, instead we return back to the kernel to extract
> useful information at the time of the MCE. While we still
> get this information, this patch converts the restart to
> a panic(), so that if configured a dump can be taken and
> we can track and probably debug the potential issue causing
> the MCE.
> 
> Signed-off-by: Balbir Singh <bsingharora@gmail.com>

Seems like something we should be doing.

Reviewed-by: Nicholas Piggin <npiggin@gmail.com>

> ---
>  arch/powerpc/platforms/powernv/opal.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/powerpc/platforms/powernv/opal.c b/arch/powerpc/platforms/powernv/opal.c
> index 69b5263fc9e3..b510a6f41b00 100644
> --- a/arch/powerpc/platforms/powernv/opal.c
> +++ b/arch/powerpc/platforms/powernv/opal.c
> @@ -500,9 +500,12 @@ void pnv_platform_error_reboot(struct pt_regs *regs, const char *msg)
>  	 *    opal to trigger checkstop explicitly for error analysis.
>  	 *    The FSP PRD component would have already got notified
>  	 *    about this error through other channels.
> +	 * 4. We are running on a newer skiboot that by default does
> +	 *    not cause a checkstop, drops us back to the kernel to
> +	 *    extract context and state at the time of the error.
>  	 */
>  
> -	ppc_md.restart(NULL);
> +	panic("PowerNV Unrecovered Machine Check");
>  }
>  
>  int opal_machine_check(struct pt_regs *regs)

Re: [RFC] powerpc/powernv/mce: Don't silently restart the machine

[Stewart Smith]

Balbir Singh <bsingharora@gmail.com> writes:
> On MCE the current code will restart the machine with
> ppc_md.restart(). This case was extremely unlikely since
> prior to that a skiboot call is made and that resulted in
> a checkstop for analysis.
>
> With newer skiboots, on P9 we don't checkstop the box by
> default, instead we return back to the kernel to extract
> useful information at the time of the MCE. While we still
> get this information, this patch converts the restart to
> a panic(), so that if configured a dump can be taken and
> we can track and probably debug the potential issue causing
> the MCE.

I agree with the patch, although I'd be nervous stating that skiboot is
going to keep this behaviour. In *theory* we should only ever get a
platform error when there's actually something that isn't the kernel's
fault.

Like any firmware promise though, it's slightly less reliable than one
from a politician.

I'd say that in this case deferring to policy on what to do in event of
panic() is the right thing.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: [RFC] powerpc/powernv/mce: Don't silently restart the machine

[Balbir Singh]

On Wed, Feb 21, 2018 at 3:54 PM, Stewart Smith
<stewart@linux.vnet.ibm.com> wrote:
> Balbir Singh <bsingharora@gmail.com> writes:
>> On MCE the current code will restart the machine with
>> ppc_md.restart(). This case was extremely unlikely since
>> prior to that a skiboot call is made and that resulted in
>> a checkstop for analysis.
>>
>> With newer skiboots, on P9 we don't checkstop the box by
>> default, instead we return back to the kernel to extract
>> useful information at the time of the MCE. While we still
>> get this information, this patch converts the restart to
>> a panic(), so that if configured a dump can be taken and
>> we can track and probably debug the potential issue causing
>> the MCE.
>
> I agree with the patch, although I'd be nervous stating that skiboot is
> going to keep this behaviour. In *theory* we should only ever get a
> platform error when there's actually something that isn't the kernel's
> fault.
>
> Like any firmware promise though, it's slightly less reliable than one
> from a politician.
>
> I'd say that in this case deferring to policy on what to do in event of
> panic() is the right thing.
>

Your right, except that with NPUs and coherent device memory, things
change. It is a platform error on the device side, the large issue is
that the CPU touched this memory. Ideally we want the device and
device driver to handle this error, but its racy as the CPU now treats
it as a platform error on the system side. IOW, the definition of
platform has grown and so has the definition of platform error and it
can no longer be solely contained inside of one boxes firmware.

Balbir Singh.