Re: [PATCH v2 0/7] CPU hotplug, cpusets: Fix issues with cpusets handling upon CPU hotplug

[Nishanth Aravamudan <nacc@linux.vnet.ibm.com>]

On 04.05.2012 [23:34:25 +0200], Peter Zijlstra wrote:
> On Fri, 2012-05-04 at 14:27 -0700, Nishanth Aravamudan wrote:
> > >  - if you retain it for cpuset but not others that's confusing (too);
> > 
> > That's a good point.
> > 
> > Related, possibly counter-example, and perhaps I'm wrong about it.  When
> > we hot-unplug a CPU, and a task's scheduler affinity (via
> > sched_setaffinity) refers to that CPU only, do we kill that task? Can
> > you sched_setaffinity a task to a CPU that is offline (alone or in a
> > group of possible CPUs)? Or is it allowed to run anywhere? Do we destroy
> > its affinity policy when that situation is run across?
> 
> See a few emails back, we destroy the affinity. Current cpuset behaviour
> can be said to match that.

Ah you're right, sorry for glossing over that case. Does that also
happen if you affinitize it to a group of CPUs?

Seems not, we "remember" the original mask in that case:

# taskset -p f $$
pid 1424's current affinity mask: ff
pid 1424's new affinity mask: f
# grep Cpus_allowed /proc/self/status
Cpus_allowed:   0000000f
Cpus_allowed_list:      0-3
# echo 0 > /sys/devices/system/cpu/cpu2/online
# grep Cpus_allowed /proc/self/status
Cpus_allowed:   0000000f
Cpus_allowed_list:      0-3

So ... it seems like we come to a crossroads of sorts? I would think
cpusets and sched_setaffinity should behave the same in terms of
hotplug. 

*Maybe* a compromise is that we remember cpuset information up to the
empty cpuset, once you empty a cpuset, you forget everything? That
roughly corresponds to your and my test-case results?

Maybe that's more work than it's worth. It seems like, though, they
should have some similarity in functionality.

> >  Or do we restore the task to the CPU again when we re-plug it?
> 
> Nope that information is lost forever from the kernels pov.
> 
> Keeping this information around for the off-chance of needing it is
> rather expensive (512 bytes per task for your regular distro kernel that
> has NR_CPUS=4096).

Yep, that's another good point.

Thanks,
Nish

-- 
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center