Re: Strange panic as soon as timer interrupts are enabled (recent 2.5)

[Andrew Morton <akpm@digeo.com>]

"Martin J. Bligh" wrote:
> 
> ...
> ---------------------
> 
> BOOT CPU:
> 
> init
>         smp_prepare_cpus
>                 smp_boot_cpus
>                         setup_local_APIC
>                         foreach cpu
>                                 do_boot_cpu
>                                         wakeup_secondary_via_(NMI|INIT)
>                                         {set bit in cpu_callout_map}
>                                         {spin on cpu_callin_map}
>                         setup_IO_APIC
>                         synchronize_tsc_bp
>         smp_init
>                 foreach cpu
>                         cpu_up
>                                 notifier_call_chain(CPU_UP_PREPARE)
>                                 __cpu_up
>                                         {set bit in smp_commenced_mask}
>                                         {spin on cpu_online_map}
>                                 notifier_call_chain(CPU_ONLINE)
>                 smp_cpus_done
>                 smp_commence

Nice!

> ------------------------
> 
> SECONDARY CPU:
> 
> start_secondary
>         cpu_init
>         smp_callin
>                 {spin on cpu_callout_map}
>                 setup_local_APIC
>                 local_irq_enable
>                 calibrate_delay
>                 disable_APIC_timer
>                 {set our bit in cpu_callin_map}
>                 synchronise_tsc_api
>         {spin on smp_commenced_mask}
>         enable_APIC_timer
>         {set our bit in cpu_online_map}
>         idle

So this is the bug, isn't it?  Can the calibrate_delay stuff be moved
until _after_ the bit has been set in smp_commenced_mask??
 
> 
> 
> > In this case I'd say "all interrupts".  The secondary really
> > should be 100% dormant until all CPU_UP_PREPARE callouts have
> > been run and have returned NOTIFY_OK.
> >
> > At least, that's how I'd have designed it.
> 
> Well that makes sense to me. Apart from when I started doing it,
> it seems that in smp_callin we do calibrate_delay, which looks
> like it needs interrupts enabled (I could be wrong). I suppose
> I could just disable them at the end of smp_callin again, but
> it's all rather ugly. Maybe after we do disable_APIC_timer in
> smp_callin.

Maybe you could copy the boot cpu's calibrate_delay result
over to all cpu_possible secondaries, then redo the calibration
for real once the secondary is actually legally up and running.