[PATCH v2] powerpc/xmon: Wait for secondaries before IPI'ing on system reset

[PATCH v2] powerpc/xmon: Wait for secondaries before IPI'ing on system reset

[Michael Ellerman]

From: Nicholas Piggin <npiggin@gmail.com>

An externally triggered system reset (e.g., via QEMU nmi command, or pseries
reset button) can cause system reset interrupts on all CPUs. In case this causes
xmon to be entered, it is undesirable for the primary (first) CPU into xmon to
trigger an NMI IPI to others, because this may cause a nested system reset
interrupt.

So spin for a time waiting for secondaries to join xmon before performing the
NMI IPI, similarly to what the crash dump code does.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
[mpe: Only do it when we come in from system reset, not via sysrq etc.]
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
---
 arch/powerpc/xmon/xmon.c | 34 ++++++++++++++++++++++++++--------
 1 file changed, 26 insertions(+), 8 deletions(-)

v2: Only wait before sending the IPI if we came from a system reset. Otherwise
    any entry into xmon, from eg. sysrq, will pause before entering which is
    undesirable.

diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
index 682506821bba..5c184b059a3d 100644
--- a/arch/powerpc/xmon/xmon.c
+++ b/arch/powerpc/xmon/xmon.c
@@ -430,6 +430,21 @@ static inline int unrecoverable_excp(struct pt_regs *regs)
 #endif
 }
 
+static bool wait_for_other_cpus(int ncpus)
+{
+	unsigned long timeout;
+
+	/* We wait for 2s, which is a metric "little while" */
+	for (timeout = 20000; timeout != 0; --timeout) {
+		if (cpumask_weight(&cpus_in_xmon) >= ncpus)
+			return true;
+		udelay(100);
+		barrier();
+	}
+
+	return false;
+}
+
 static int xmon_core(struct pt_regs *regs, int fromipi)
 {
 	int cmd = 0;
@@ -440,7 +455,6 @@ static int xmon_core(struct pt_regs *regs, int fromipi)
 #ifdef CONFIG_SMP
 	int cpu;
 	int secondary;
-	unsigned long timeout;
 #endif
 
 	local_irq_save(flags);
@@ -527,13 +541,17 @@ static int xmon_core(struct pt_regs *regs, int fromipi)
 		xmon_owner = cpu;
 		mb();
 		if (ncpus > 1) {
-			smp_send_debugger_break();
-			/* wait for other cpus to come in */
-			for (timeout = 100000000; timeout != 0; --timeout) {
-				if (cpumask_weight(&cpus_in_xmon) >= ncpus)
-					break;
-				barrier();
-			}
+			/*
+			 * A system reset (trap == 0x100) can be triggered on
+			 * all CPUs, so when we come in via 0x100 try waiting
+			 * for the other CPUs to come in before we send the
+			 * debugger break (IPI). This is similar to
+			 * crash_kexec_secondary().
+			 */
+			if (TRAP(regs) != 0x100 || !wait_for_other_cpus(ncpus))
+				smp_send_debugger_break();
+
+			wait_for_other_cpus(ncpus);
 		}
 		remove_bpts();
 		disable_surveillance();
-- 
2.7.4

Re: [PATCH v2] powerpc/xmon: Wait for secondaries before IPI'ing on system reset

[kbuild test robot]

[-- Attachment #1: Type: text/plain, Size: 2114 bytes --]

Hi Nicholas,

[auto build test ERROR on powerpc/next]
[also build test ERROR on v4.11-rc8 next-20170428]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Michael-Ellerman/powerpc-xmon-Wait-for-secondaries-before-IPI-ing-on-system-reset/20170429-143734
base:   https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next
config: powerpc-holly_defconfig (attached as .config)
compiler: powerpc-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
        wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # save the attached .config to linux build tree
        make.cross ARCH=powerpc 

All errors (new ones prefixed by >>):

   arch/powerpc/xmon/xmon.c: In function 'wait_for_other_cpus':
>> arch/powerpc/xmon/xmon.c:439:23: error: 'cpus_in_xmon' undeclared (first use in this function)
      if (cpumask_weight(&cpus_in_xmon) >= ncpus)
                          ^~~~~~~~~~~~
   arch/powerpc/xmon/xmon.c:439:23: note: each undeclared identifier is reported only once for each function it appears in
   At top level:
>> arch/powerpc/xmon/xmon.c:433:13: error: 'wait_for_other_cpus' defined but not used [-Werror=unused-function]
    static bool wait_for_other_cpus(int ncpus)
                ^~~~~~~~~~~~~~~~~~~
   cc1: all warnings being treated as errors

vim +/cpus_in_xmon +439 arch/powerpc/xmon/xmon.c

   427		return 0;
   428	#else
   429		return ((regs->msr & MSR_RI) == 0);
   430	#endif
   431	}
   432	
 > 433	static bool wait_for_other_cpus(int ncpus)
   434	{
   435		unsigned long timeout;
   436	
   437		/* We wait for 2s, which is a metric "little while" */
   438		for (timeout = 20000; timeout != 0; --timeout) {
 > 439			if (cpumask_weight(&cpus_in_xmon) >= ncpus)
   440				return true;
   441			udelay(100);
   442			barrier();

---
0-DAY kernel test infrastructure                Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all                   Intel Corporation

[-- Attachment #2: .config.gz --]
[-- Type: application/gzip, Size: 13182 bytes --]

Re: [PATCH v2] powerpc/xmon: Wait for secondaries before IPI'ing on system reset

[Anton Blanchard]

Hi Nick,

> An externally triggered system reset (e.g., via QEMU nmi command, or
> pseries reset button) can cause system reset interrupts on all CPUs.
> In case this causes xmon to be entered, it is undesirable for the
> primary (first) CPU into xmon to trigger an NMI IPI to others,
> because this may cause a nested system reset interrupt.
> 
> So spin for a time waiting for secondaries to join xmon before
> performing the NMI IPI, similarly to what the crash dump code does.

That reminds me of similar delays in our crash path:

/*
 * The primary CPU waits a while for all secondary CPUs to enter. This is to
 * avoid sending an IPI if the secondary CPUs are entering
 * crash_kexec_secondary on their own (eg via a system reset).
 *
 * The secondary timeout has to be longer than the primary. Both timeouts are
 * in milliseconds.
 */
#define PRIMARY_TIMEOUT         500
#define SECONDARY_TIMEOUT       1000

...

        /*
         * If we came in via system reset, wait a while for the secondary
         * CPUs to enter.
         */
        if (TRAP(regs) == 0x100)
                mdelay(PRIMARY_TIMEOUT);

We might want to consolidate the juggling we do. Not sure if many people use
it, but kdb and kgdb may benefit if we make it common.

Anton

Re: [PATCH v2] powerpc/xmon: Wait for secondaries before IPI'ing on system reset

[Nicholas Piggin]

On Sun, 30 Apr 2017 22:27:58 +1000
Anton Blanchard <anton@samba.org> wrote:

> Hi Nick,
> 
> > An externally triggered system reset (e.g., via QEMU nmi command, or
> > pseries reset button) can cause system reset interrupts on all CPUs.
> > In case this causes xmon to be entered, it is undesirable for the
> > primary (first) CPU into xmon to trigger an NMI IPI to others,
> > because this may cause a nested system reset interrupt.
> > 
> > So spin for a time waiting for secondaries to join xmon before
> > performing the NMI IPI, similarly to what the crash dump code does.  
> 
> That reminds me of similar delays in our crash path:
> 
> /*
>  * The primary CPU waits a while for all secondary CPUs to enter. This is to
>  * avoid sending an IPI if the secondary CPUs are entering
>  * crash_kexec_secondary on their own (eg via a system reset).
>  *
>  * The secondary timeout has to be longer than the primary. Both timeouts are
>  * in milliseconds.
>  */
> #define PRIMARY_TIMEOUT         500
> #define SECONDARY_TIMEOUT       1000
> 
> ...
> 
>         /*
>          * If we came in via system reset, wait a while for the secondary
>          * CPUs to enter.
>          */
>         if (TRAP(regs) == 0x100)
>                 mdelay(PRIMARY_TIMEOUT);
> 
> We might want to consolidate the juggling we do. Not sure if many people use
> it, but kdb and kgdb may benefit if we make it common.

Good point, yes we definitely should consolidate these. I had a few misc patches
that were supposed to improve the crash paths to go after the NMI stacks and
IPIs series. I'll look at consolidating some of this timeout logic there, if
nobody beats me to it. Thanks.