cpusets vs cpu-hotplug interaction is broken?

[Oleg Nesterov <oleg@tv-sign.ru>]

(cpu-hotplug experts cc'ed)

On 08/25, Oleg Nesterov wrote:
>
> After the brief look at kernel/cpuset.c, it seems that attach_task() should
> guarantee that the task can't use CPUs outside of cpuset->cpus_allowed.
> 
> But this looks racy wrt sched_setaffinity() which does
> 
> 	cpus_allowed = cpuset_cpus_allowed(p);
> 	// callback_mutex is free
> 	set_cpus_allowed(p);
> 
> What if attach_task()->set_cpus_allowed() happens in between?

Actually, I think there is another problem, and cpuset_cpus_allowed() is
just broken wrt CONFIG_HOTPLUG_CPU.

Suppose that CONFIG_CPUSETS is true, but we don't use cpusets. In that
case all tasks in system belong to the top_cpuset (btw, why cpuset_init()
sets init_task.cpuset? this was already done by cpuset_init_early()), and
we should have the same behaviour as without CONFIG_CPUSETS.

By default, all tasks have ->cpus_allowed = CPU_MASK_ALL inherited from
kernel_init(). This means that the task can use the new CPU right after
cpu_up().

Now let's suppose that some task does sched_setaffinity(0, CPU_MASK_ALL).
In that case, cpuset_cpus_allowed() sets ->cpus_allowed = cpu_online_map,
and I think this is just wrong. Now that task doesn't see the new CPUs.

Of course, we have the similar problem with cpusets other than top_cpuset.

In short, unless I missed something, top_cpuset.cpus_allowed should be
cpu_possible_map, guarantee_online_cpus() shouldn't mix "allowed" and
"online" masks.

Oleg.