Re: Inquiry: Should we remove "isolcpus= kernel boot option? (may have realtime uses)

[Max Krasnyansky <maxk@qualcomm.com>]

Dimitri Sivanich wrote:
> On Mon, Jun 02, 2008 at 11:39:34AM -0700, Max Krasnyansky wrote:
>> Ah, I know exactly what you're talking about.  However this is non-issue these
>> days. In order to clear cpuN from all the timers and other things all you need
>> to do is to bring that cpu off-line
>> 	echo 0 > /sys/devices/cpu/cpuN/online
>> and then bring it back online
>> 	echo 1 > /sys/devices/cpu/cpuN/online
> 
> Although it seemed like something of a hack, we experimented with this
> previously and found that it didn't work reliably.  I'm sure things
> have gotten better, but will need to revisit.
Yes it used to be somewhat unstable. These days it solid. I'm using it on a
wide range of systems: uTCA Core2Duo, NUMA dual-Opteron, 8way Core2, etc. And
things work as expected.
I forgot to mention that it's not just timers. There are also work queues and
delayed work that have similar side effects (ie they stick to the CPU they
were originally scheduled on). Hotplug cleans all that stuff very nicely.

btw I would not call it a hack. ie Using cpu hotplug for isolation purposes.
By definition hotplug must be able to migrate _everything_ running on the cpuN
when it goes off-line, otherwise it simply won't work. And that's exactly what
we need for the isolation too (migrate everything running on a cpuN to other
cpus).

>> There are currently a couple of issues with scheduler domains and hotplug
>> event handling. I do have the fix for them, and Paul had already acked it.
> 
> Until a proven reliable method for doing this is firmly in place (as
> firmly as anything is, anyway),  I don't think we should be removing
> the alternative.
Agree. That's why I submitted the removal patch along with those fixes ;-).

>> initialization). See my latest "default IRQ affinity" patch.
> Nice idea.
Thanx.

>> Also isolcpus= conflicts with the scheduler domains created by the cpusets.
> 
> What sort of conflict are we talking about? I assume once you've begun
> setting up cpusets that include those cpus that you're intention is to change
> the original behavior.
That exactly where the conflict is. Lets say you boot with isolcpus=2 (ie cpu2
is not load balanced), then you add cpu2 along with cpu3 to cpuset N and
enable load balancing in cpusetN. In that case cpu2 will still remain
unbalanced which is definitely a wrong behaviour.

Max