Re: [Lse-tech] [PATCH] cpusets+hotplug+preepmt broken

[Paul Jackson <pj@sgi.com>]

pj wrote:
> Could you explain why this is -- what is the deadlock?

On further reading, I see it.  You're right.

Deep in the bowels of the hotplug code, when removing a dead cpu, while
holding the runqueue lock (task_rq_lock), the hotplug code might need to
walk up the cpuset hierarchy, to find the nearest enclosing cpuset that
still has online cpus, as part of figuring where to run a task that is
being kicked off the dead cpu.  The runqueue lock is atomic, but getting
the cpuset_sem (needed to walk up the cpuset hierarchy) can sleep.  So
you need to get the cpuset_sem before calling task_rq_lock, so as to
avoid the "scheduling while atomic" oops that you reported.  Therefore
the hotplug code, and anyone else calling cpuset_cpus_allowed(), which
means sched_setaffinity(), needs to first grab cpuset_sem, so that they
can grab any atomic locks needed, after getting cpuset_sem, not before.

Acked-by: Paul Jackson <pj@sgi.com>

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@engr.sgi.com> 1.650.933.1373, 1.925.600.0401