Re: Commit 78eef01b0fae087c5fadbd85dd4fe2918c3a015f (on_each_cpu(): disable local interrupts) Breaks SGI IP32

[Kumba <kumba@gentoo.org>]

Andrew Morton wrote:
> 
> Yup.  But again, the reasons for that change to on_each_cpu() were to make
> all instances of the the callback function run under the same environment
> in all cases.  That's a good change.
> 
> If the platform _knows_ that it's safe to do normally-unsafe things then as
> I say, it shold special-case that case.
> 
> Here, if the call is in O2-only code then we don't need on_each_cpu() at
> all - just call the function instead.
> 
> If the call is in board-neutral MIPS code then things get more complicated.
> Sure, the code is safe on UP, but it might be deadlocky on SMP.  It needs
> to be thought about and a suitable UP&&SMP fix needs to be found.
> 
> Can someone point me at the code we're talking about here?  file-n-line?

The farthest I can track the lock up is to line ~290 in mm/bootmem.c, in 
__alloc_bootmem_core.  It's when it goes off to try and invoke that memset() 
function that the machine hangs, and gets put into a state only a cold boot can 
recover.

Since discovering the this change that introduces this oddity, I think it's what 
explains the hang happening in the "dark void between function calls" -- control 
attempts to pass to the memset(), but never reaches it (a printk() before that 
memset() will get called, but a printk inside the memset() never gets called, 
even if it's the very first instruction inside the function).  Turning off the 
IRQs, then calling func() while 'info' contains the pointer to the memset() just 
goes *poof* as Ralf mentioned.

I tried putting printk()'s in the on_each_cpu() macro, before all three 
instructions, but none of those fired.  So whether control even reaches that 
point, I'm not sure.  Ralf mentioned (I think) it could be a cache flush 
happening after IRQs are disabled that knocks the machine out or something to 
that effect.

It's tricky to say -- I ran into the exact same hang back in ~2.6.15.2, but it 
silently vanished before I could pin it down, so I don't even know if it's 
related.  I'm good at tracking down some problems, but even this one's done a 
fine job of eluding me up until now :)


--Kumba

-- 
Gentoo/MIPS Team Lead
Gentoo Foundation Board of Trustees

"Such is oft the course of deeds that move the wheels of the world: small hands 
do them because they must, while the eyes of the great are elsewhere."  --Elrond