crash on shutdown on rs/6000 powerpc

crash on shutdown on rs/6000 powerpc

[Paul Mackerras]

I have just seen a crash at shutdown with 2.6.23-rc9 on an RS/6000
powerpc box (POWER3, 64-bit).  It crashed immediately after printing
"Disabling non-boot CPUs", so I tried reverting 4047727e and that
fixed it.  It's unfortunate that that commit added the
disable_nonboot_cpus for all architectures, when only x86[-64] needs
it.  (At least it sounds like only x86[-64] needs it; the commit
message seems quite x86-centric.)

In case anyone on the linuxppc-dev list wants to chase this further,
the crash looks like this:

Disabling non-boot CPUs ...
cpu 0x1: Vector: 300 (Data Access) at [c00000003f923950]
    pc: c00000000003bf98: .xics_set_cpu_priority+0x58/0x78
    lr: c00000000003bff4: .xics_migrate_irqs_away+0x3c/0x20c
    sp: c00000003f923bd0
   msr: a000000000001032
   dar: 4
 dsisr: 42000000
  current = 0xc0000000bfb6b060
  paca    = 0xc00000000071c600
    pid   = 2610, comm = kstopmachine
enter ? for help
[c00000003f923c40] c00000000003bff4 .xics_migrate_irqs_away+0x3c/0x20c
[c00000003f923d00] c000000000040d54 .pseries_cpu_disable+0x98/0xb4
[c00000003f923d80] c000000000028e4c .__cpu_disable+0x44/0x58
[c00000003f923df0] c00000000007e204 .take_cpu_down+0x34/0x60
[c00000003f923e70] c00000000008ba3c .do_stop+0x144/0x1e4
[c00000003f923f00] c00000000006fd74 .kthread+0x78/0xc4
[c00000003f923f90] c0000000000272a8 .kernel_thread+0x4c/0x68

I don't have time tonight or tomorrow to track this down further.  I'm
taking the kids to the coast tommorow. :)

Paul.

Re: crash on shutdown on rs/6000 powerpc

[Joel Schopp]

> [c00000003f923c40] c00000000003bff4 .xics_migrate_irqs_away+0x3c/0x20c
> [c00000003f923d00] c000000000040d54 .pseries_cpu_disable+0x98/0xb4
> [c00000003f923d80] c000000000028e4c .__cpu_disable+0x44/0x58
> [c00000003f923df0] c00000000007e204 .take_cpu_down+0x34/0x60
> [c00000003f923e70] c00000000008ba3c .do_stop+0x144/0x1e4
> [c00000003f923f00] c00000000006fd74 .kthread+0x78/0xc4
> [c00000003f923f90] c0000000000272a8 .kernel_thread+0x4c/0x68
> 
> I don't have time tonight or tomorrow to track this down further.  I'm
> taking the kids to the coast tommorow. :)

This looks like the problem Olof and Milton sent out patches to fix.  The 
Power3 shouldn't be doing cpu hotplug.

Re: crash on shutdown on rs/6000 powerpc

[Milton Miller]

Joel Schoopp wrote:
>Paulus wrote:
>> [c00000003f923c40] c00000000003bff4 .xics_migrate_irqs_away+0x3c/0x20c
>> [c00000003f923d00] c000000000040d54 .pseries_cpu_disable+0x98/0xb4
>> [c00000003f923d80] c000000000028e4c .__cpu_disable+0x44/0x58
>> [c00000003f923df0] c00000000007e204 .take_cpu_down+0x34/0x60
>> [c00000003f923e70] c00000000008ba3c .do_stop+0x144/0x1e4
>> [c00000003f923f00] c00000000006fd74 .kthread+0x78/0xc4
>> [c00000003f923f90] c0000000000272a8 .kernel_thread+0x4c
>>
>> I don't have time tonight or tomorrow to track this down further.  I'm
>> taking the kids to the coast tommorow. :)
>>
> This looks like the problem Olof and Milton sent out patches to fix.  The 
> Power3 shouldn't be doing cpu hotplug.

Paul's comment prompted both of us.

Actually, cpu_hotplug (offline calling rtas stop-self, not dlpar
remove) should be supported on POWER3 and RS64, its just that the
pseries disable code calls over into xics directly.  offline support
would help preserve system uptime under a "failing processor"
scenerio.

To fix that, we need a mpic version of migrate_irqs_away.   powermac
is using the "bury head in sand" approach for that right now (ie
if you bind an interrupt and offline all the cpus in the mask, the
interrupt won't be delivered until you reset the affinity).  Its also
a broken in that bringing a cpu back online adds the cpu to all
interrupts not just those who have the cpu mask bit set.

milton