cgroup scheduling: Adding kthreadd to a non-RT cgroup can deadlock the kernel

cgroup scheduling: Adding kthreadd to a non-RT cgroup can deadlock the kernel

[Nelson Elhage]

Hi,

I've found a bug where, on CONFIG_RT_GROUP_SCHED systems, adding the kthreadd
task to a cgroup with cpu.rt_runtime_us = 0 (as some cgroup configuration
scripts do, when they move all processes into a default cgroup), can result in
deadlocks in the kernel.

On 2.6.37, the problem can be triggered via CPU hotplug. The following sequence
of events will deadlock on an SMP system:

1. Add kthreadd to a cpu cgroup with rt_runtime_us = 0
2. echo 0 > /sys/devices/system/cpu/cpu1/online
3. echo 1 > /sys/devices/system/cpu/cpu1/online
4. echo 0 > /sys/devices/system/cpu/cpu1/online
5. echo 1 > /sys/devices/system/cpu/cpu1/online

In line (3), the CPU hotplug will cause us to create a new ksoftirqd/1
thread. Since that thread is forked from kthreadd, it will end up in the same
cgroup, also without any realtime access.

In step (4), cpu_callback in softirq.c will attempt to kill ksoftirqd by setting
it to SCHED_FIFO and using kthread_stop(). It does this with
'sched_setscheduler_nocheck', which bypasses the usual checks that prevent
setting a process to an SCHED_FIFO if it is in a cgroup that would prevent it
from running.

Thus, ksoftirqd ends up at SCHED_FIFO but with a zero rt_runtime_us, and is
never scheduled again, and kthread_stop blocks waiting on it.

In (5), we try to call the CPU notifier chain again, but it is still locked from
(4), and we deadlock.

For reasons I don't fully understand, just adding ksoftirqd/1 to a cgroup and
then taking CPU 1 offline doesn't result in a hang, so I think there may be some
detail of this situation I don't fully understand, but I'm pretty confident in
the general analysis.

Before 2.6.34, we can trigger a similar problem just by adding kthreadd to a
cgroup and then calling stop_machine (e.g. by removing a module), since
stop_machine created a new RT workqueue on each invocation. This is how I first
found this problem: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/693594

- Nelson

Re: cgroup scheduling: Adding kthreadd to a non-RT cgroup can deadlock the kernel

[Mike Galbraith]

On Tue, 2011-01-04 at 23:54 -0500, Nelson Elhage wrote:
> Hi,

Greetings,

> I've found a bug where, on CONFIG_RT_GROUP_SCHED systems, adding the kthreadd
> task to a cgroup with cpu.rt_runtime_us = 0 (as some cgroup configuration
> scripts do, when they move all processes into a default cgroup), can result in
> deadlocks in the kernel.
> 
> On 2.6.37, the problem can be triggered via CPU hotplug. The following sequence
> of events will deadlock on an SMP system:
> 
> 1. Add kthreadd to a cpu cgroup with rt_runtime_us = 0
> 2. echo 0 > /sys/devices/system/cpu/cpu1/online
> 3. echo 1 > /sys/devices/system/cpu/cpu1/online
> 4. echo 0 > /sys/devices/system/cpu/cpu1/online
> 5. echo 1 > /sys/devices/system/cpu/cpu1/online
> 
> In line (3), the CPU hotplug will cause us to create a new ksoftirqd/1
> thread. Since that thread is forked from kthreadd, it will end up in the same
> cgroup, also without any realtime access.
> 
> In step (4), cpu_callback in softirq.c will attempt to kill ksoftirqd by setting
> it to SCHED_FIFO and using kthread_stop(). It does this with
> 'sched_setscheduler_nocheck', which bypasses the usual checks that prevent
> setting a process to an SCHED_FIFO if it is in a cgroup that would prevent it
> from running.
> 
> Thus, ksoftirqd ends up at SCHED_FIFO but with a zero rt_runtime_us, and is
> never scheduled again, and kthread_stop blocks waiting on it.
> 
> In (5), we try to call the CPU notifier chain again, but it is still locked from
> (4), and we deadlock.

Hm.  Seems to me this is just another of the myriad ways a privileged
user can shoot himself in the foot.

	-Mike

Re: cgroup scheduling: Adding kthreadd to a non-RT cgroup can deadlock the kernel

[Peter Zijlstra]

On Tue, 2011-01-04 at 23:54 -0500, Nelson Elhage wrote:
> Hi,
> 
> I've found a bug where, on CONFIG_RT_GROUP_SCHED systems, adding the kthreadd
> task to a cgroup with cpu.rt_runtime_us = 0 (as some cgroup configuration
> scripts do, when they move all processes into a default cgroup), can result in
> deadlocks in the kernel.

IMHO its a bug to move kthreadd into a cgroup. Simply don't do that.

Re: cgroup scheduling: Adding kthreadd to a non-RT cgroup can deadlock the kernel

[Nelson Elhage]

Ok. I bothered reporting this here since the kernel /does/ try to prevent you
from shooting yourself in the foot in related ways, by moving existing
SCHED_FIFO tasks into a cgroup without any RT runtime, so I figured it might
make sense to add a check here. libcgroup, for example, will try to move all
processes into a default cgroup, ignoring errors, and relies on the kernel to
prevent it from shooting itself in the foot.

It sounds like you consider that behavior a bug, though, so I'll go report this
bug there.

- Nelson

On Wed, Jan 5, 2011 at 4:44 AM, Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> On Tue, 2011-01-04 at 23:54 -0500, Nelson Elhage wrote:
>> Hi,
>>
>> I've found a bug where, on CONFIG_RT_GROUP_SCHED systems, adding the kthreadd
>> task to a cgroup with cpu.rt_runtime_us = 0 (as some cgroup configuration
>> scripts do, when they move all processes into a default cgroup), can result in
>> deadlocks in the kernel.
>
> IMHO its a bug to move kthreadd into a cgroup. Simply don't do that.
>
>
>

Re: cgroup scheduling: Adding kthreadd to a non-RT cgroup can deadlock the kernel

[Peter Zijlstra]

On Wed, 2011-01-05 at 10:02 -0500, Nelson Elhage wrote:
> Ok. I bothered reporting this here since the kernel /does/ try to prevent you
> from shooting yourself in the foot in related ways, by moving existing
> SCHED_FIFO tasks into a cgroup without any RT runtime, so I figured it might
> make sense to add a check here. libcgroup, for example, will try to move all
> processes into a default cgroup, ignoring errors, and relies on the kernel to
> prevent it from shooting itself in the foot.
> 
> It sounds like you consider that behavior a bug, though, so I'll go report this
> bug there.

Right, so we try to catch obvious cases, but its near impossible to
catch all cases.

And its not only the scheduler controller, I bet you can get into
trouble with some of the other controllers as well.