Re: BUG_ON(!cpus_equal(cpumask, tmp));

["Martin J. Bligh" <mbligh@aracnet.com>]

>> > I'll just say that kexec fails without this patch and works with
>> > it applied, so I'd like to see it merged.  If this patch isn't
>> > acceptable, let's find out why and try to make one that is.
>> > 
>> > Thanks for the patch, Hari.
>> 
>> > From discussions with Andy, it seems this still has the same race as before
>> just smaller. I don't see how we can fix this properly without having some
>> locking on cpu_online_map .... probably RCU as it's massively read-biased
>> and we don't want to pay a spinlock cost to read it.
> 
> We do want to avoid adding stuff to the IPI path.

OK, but my thinking was that the readlock of RCU is free (pretty much).

> If the going-away CPU still responds to IPIs after it has gone away 
> then do we actually need to do anything?  For x86, at least?

Eeek, you *really* don't want it responding to IPIs after it's been shut
down. It's dead, dead, dead, and we don't want it rolling over in it's
gasping throes, and stabbing us in the back. Supposing we've kexeced or 
otherwise rebooted the machine? If we've shut down the other cpus, we
should be able to reprogram the IDT, drop back out of paging mode to 
reboot into the BIOS, or basically anything. Nothing short of an NMI or 
a power cycle (aka Startup / INIT sequence) should arouse it.

And if we don't take IPIs, I really think we're playing russian roulette 
here ... I was under the *impression* that if you sent an IPI to a cpu 
that was down and didn't ack it, we'd hang. Forever. At least on a P3 
style apic bus, when we're sending individual interrupts to each CPU, 
not broadcast.

I made a similar patch, but I don't see how we can really fix it without
providing locking on cpu_online_map. Or else you have to do something 
gross like mark the CPU offline, spin for 5 seconds, then take it offline.
Ick. Maybe we could be smart with a 2-stage commit with 2 bitmaps or
something, but I think that basically evolves into RCU anyway, and just
reimplements it ;-( 

Shared global data needs some form of locking, even if "oh we don't
write to that very often, I think we'll get away with it" ;-)

M.