From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756188Ab1ILXMQ (ORCPT ); Mon, 12 Sep 2011 19:12:16 -0400 Received: from mail-vw0-f52.google.com ([209.85.212.52]:45745 "EHLO mail-vw0-f52.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756096Ab1ILXML (ORCPT ); Mon, 12 Sep 2011 19:12:11 -0400 From: Frederic Weisbecker To: Andrew Morton Cc: LKML , Frederic Weisbecker , Containers , Paul Menage , Li Zefan , Johannes Weiner , Aditya Kali , Oleg Nesterov , Kay Sievers , Tim Hockin , Tejun Heo Subject: [PATCH 09/11] cgroups: Add a task counter subsystem Date: Tue, 13 Sep 2011 01:11:29 +0200 Message-Id: <1315869091-18933-10-git-send-email-fweisbec@gmail.com> X-Mailer: git-send-email 1.7.5.4 In-Reply-To: <1315869091-18933-1-git-send-email-fweisbec@gmail.com> References: <1315869091-18933-1-git-send-email-fweisbec@gmail.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Add a new subsystem to limit the number of running tasks, similar to the NR_PROC rlimit but in the scope of a cgroup. This is a step to be able to isolate a bit more a cgroup against the rest of the system and limit the global impact of a fork bomb inside a given cgroup. Signed-off-by: Frederic Weisbecker Cc: Paul Menage Cc: Li Zefan Cc: Johannes Weiner Cc: Aditya Kali Cc: Oleg Nesterov Cc: Andrew Morton Cc: Kay Sievers Cc: Tim Hockin Cc: Tejun Heo --- Documentation/cgroups/task_counter.txt | 126 +++++++++++++++++ include/linux/cgroup.h | 9 ++ include/linux/cgroup_subsys.h | 8 + init/Kconfig | 9 ++ kernel/Makefile | 1 + kernel/cgroup_task_counter.c | 237 ++++++++++++++++++++++++++++++++ kernel/fork.c | 4 + 7 files changed, 394 insertions(+), 0 deletions(-) create mode 100644 Documentation/cgroups/task_counter.txt create mode 100644 kernel/cgroup_task_counter.c diff --git a/Documentation/cgroups/task_counter.txt b/Documentation/cgroups/task_counter.txt new file mode 100644 index 0000000..e93760a --- /dev/null +++ b/Documentation/cgroups/task_counter.txt @@ -0,0 +1,126 @@ +Task counter subsystem + +1. Description + +The task counter subsystem limits the number of tasks running +inside a given cgroup. It behaves like the NR_PROC rlimit but in +the scope of a cgroup instead of a user. + +It has two typical usecases, although more can probably be found: + +- Protect against forkbombs that explode inside a container when +that container is implemented using a cgroup. The NR_PROC rlimit +is not efficient for that because if we have several containers +running in parallel under the same user, one container could starve +all the others by spawning a high number of tasks close to the +rlimit boundary. So in this case we need this limitation to be +done in a per cgroup granularity. + +- Kill all tasks inside a cgroup without races. By setting the limit +of running tasks to 0, one can prevent from any further fork inside a +cgroup and then kill all of its tasks without the need to retry an +unbound amount of time due to races between kills and forks running +in parallel (more details in "Kill a cgroup safely" paragraph). + + +2. Interface + +When a hierarchy is mounted with the task counter subsystem binded, it +adds two files into the cgroups directories, except the root one: + +- tasks.usage contains the number of tasks running inside a cgroup and +its children in the hierarchy (see paragraph about Inheritance). + +- tasks.limit contains the maximum number of tasks that can run inside +a cgroup. We check this limit when a task forks or when it is migrated +to a cgroup. + +Note that the tasks.limit value can be forced below tasks.usage, in which +case any new task in the cgroup will be rejected until the tasks.usage +value goes below tasks.limit. + +For optimization reasons, the root directory of a hierarchy doesn't have +a task counter. + + +3. Inheritance + +When a task is added to a cgroup, by way of a cgroup migration or a fork, +it increases the task counter of that cgroup and of all its ancestors. +Hence a cgroup is also subject to the limit of its ancestors. + +In the following hierarchy: + + + A + | + B + / \ + C D + + +We have 1 task running in B, one running in C and none running in D. +It means we have tasks.usage = 1 in C and tasks.usage = 2 in B because +B counts its task and those of its children. + +Now lets set tasks.limit = 2 in B and tasks.limit = 1 in D. +If we move a new task in D, it will be refused because the limit in B has +been reached already. + + +4. Kill a cgroup safely + +As explained in the description, this subsystem is also helpful to +kill all tasks in a cgroup safely, after setting tasks.limit to 0, +so that we don't race against parallel forks in an unbound numbers +of kill iterations. + +But there is a small detail to be aware of to use this feature that +way. + +Some typical way to proceed would be: + + echo 0 > tasks.limit + for TASK in $(cat cgroup.procs) + do + kill -KILL $TASK + done + +However there is a small race window where a task can be in the way to +be forked but hasn't enough completed the fork to have the PID of the +fork appearing in the cgroup.procs file. + +The only way to get it right is to run a loop that reads tasks.usage, kill +all the tasks in cgroup.procs and exit the loop only if the value in +tasks.usage was the same than the number of tasks that were in cgroup.procs, +ie: the number of tasks that were killed. + +It works because the new child appears in tasks.usage right before we check, +in the fork path, whether the parent has a pending signal, in which case the +fork is cancelled anyway. So relying on tasks.usage is fine and non-racy. + +This race window is tiny and unlikely to happen, so most of the time a single +kill iteration should be enough. But it's worth knowing about that corner +case spotted by Oleg Nesterov. + +Example of safe use would be: + + echo 0 > tasks.limit + END=false + + while [ $END == false ] + do + NR_TASKS=$(cat tasks.usage) + NR_KILLED=0 + + for TASK in $(cat cgroup.procs) + do + let NR_KILLED=NR_KILLED+1 + kill -KILL $TASK + done + + if [ "$NR_TASKS" = "$NR_KILLED" ] + then + END=true + fi + done diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index b62cf5e..3f132f5 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -661,4 +661,13 @@ static inline int cgroup_attach_task_current_cg(struct task_struct *t) #endif /* !CONFIG_CGROUPS */ +#ifdef CONFIG_CGROUP_TASK_COUNTER +int cgroup_task_counter_fork(struct task_struct *child); +#else +static inline int cgroup_task_counter_fork(struct task_struct *child) +{ + return 0; +} +#endif /* CONFIG_CGROUP_TASK_COUNTER */ + #endif /* _LINUX_CGROUP_H */ diff --git a/include/linux/cgroup_subsys.h b/include/linux/cgroup_subsys.h index ac663c1..5425822 100644 --- a/include/linux/cgroup_subsys.h +++ b/include/linux/cgroup_subsys.h @@ -59,8 +59,16 @@ SUBSYS(net_cls) SUBSYS(blkio) #endif +/* */ + #ifdef CONFIG_CGROUP_PERF SUBSYS(perf) #endif /* */ + +#ifdef CONFIG_CGROUP_TASK_COUNTER +SUBSYS(tasks) +#endif + +/* */ diff --git a/init/Kconfig b/init/Kconfig index d627783..7410b05 100644 --- a/init/Kconfig +++ b/init/Kconfig @@ -690,6 +690,15 @@ config CGROUP_MEM_RES_CTLR_SWAP_ENABLED select this option (if, for some reason, they need to disable it then swapaccount=0 does the trick). +config CGROUP_TASK_COUNTER + bool "Control number of tasks in a cgroup" + depends on RESOURCE_COUNTERS + help + Let the user set up an upper boundary of the allowed number of tasks + running in a cgroup. When a task forks or is migrated to a cgroup that + has this subsystem binded, the limit is checked to either accept or + reject the fork/migration. + config CGROUP_PERF bool "Enable perf_event per-cpu per-container group (cgroup) monitoring" depends on PERF_EVENTS && CGROUPS diff --git a/kernel/Makefile b/kernel/Makefile index eca595e..5598a7f 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -60,6 +60,7 @@ obj-$(CONFIG_BACKTRACE_SELF_TEST) += backtracetest.o obj-$(CONFIG_COMPAT) += compat.o obj-$(CONFIG_CGROUPS) += cgroup.o obj-$(CONFIG_CGROUP_FREEZER) += cgroup_freezer.o +obj-$(CONFIG_CGROUP_TASK_COUNTER) += cgroup_task_counter.o obj-$(CONFIG_CPUSETS) += cpuset.o obj-$(CONFIG_UTS_NS) += utsname.o obj-$(CONFIG_USER_NS) += user_namespace.o diff --git a/kernel/cgroup_task_counter.c b/kernel/cgroup_task_counter.c new file mode 100644 index 0000000..8882552 --- /dev/null +++ b/kernel/cgroup_task_counter.c @@ -0,0 +1,237 @@ +/* + * Limits on number of tasks subsystem for cgroups + * + * Copyright (C) 2011 Red Hat, Inc., Frederic Weisbecker + * + * Thanks to Andrew Morton, Johannes Weiner, Li Zefan, Oleg Nesterov and + * Paul Menage for their suggestions. + * + */ + +#include +#include +#include + + +struct task_counter { + struct res_counter res; + struct cgroup_subsys_state css; +}; + +/* + * The root task counter doesn't exist because it's not part of the + * whole task counting. We want to optimize the trivial case of only + * one root cgroup living. + */ +static struct cgroup_subsys_state root_css; + + +static inline struct task_counter *cgroup_task_counter(struct cgroup *cgrp) +{ + if (!cgrp->parent) + return NULL; + + return container_of(cgroup_subsys_state(cgrp, tasks_subsys_id), + struct task_counter, css); +} + +static inline struct res_counter *cgroup_task_res_counter(struct cgroup *cgrp) +{ + struct task_counter *cnt; + + cnt = cgroup_task_counter(cgrp); + if (!cnt) + return NULL; + + return &cnt->res; +} + +static struct cgroup_subsys_state * +task_counter_create(struct cgroup_subsys *ss, struct cgroup *cgrp) +{ + struct task_counter *cnt; + struct res_counter *parent_res; + + if (!cgrp->parent) + return &root_css; + + cnt = kzalloc(sizeof(*cnt), GFP_KERNEL); + if (!cnt) + return ERR_PTR(-ENOMEM); + + parent_res = cgroup_task_res_counter(cgrp->parent); + + res_counter_init(&cnt->res, parent_res); + + return &cnt->css; +} + +/* + * Inherit the limit value of the parent. This is not really to enforce + * a limit below or equal to the one of the parent which can be changed + * concurrently anyway. This is just to honour the clone flag. + */ +static void task_counter_post_clone(struct cgroup_subsys *ss, + struct cgroup *cgrp) +{ + /* cgrp can't be root, so cgroup_task_res_counter() can't return NULL */ + res_counter_inherit(cgroup_task_res_counter(cgrp), RES_LIMIT); +} + +static void task_counter_destroy(struct cgroup_subsys *ss, struct cgroup *cgrp) +{ + struct task_counter *cnt = cgroup_task_counter(cgrp); + + kfree(cnt); +} + +/* Uncharge the cgroup the task was attached to */ +static void task_counter_exit(struct cgroup_subsys *ss, struct cgroup *cgrp, + struct cgroup *old_cgrp, struct task_struct *task) +{ + /* Optimize for the root cgroup case */ + if (old_cgrp->parent) + res_counter_uncharge(cgroup_task_res_counter(old_cgrp), 1); +} + +/* + * Protected amongst can_attach_task/attach_task/cancel_attach_task by + * cgroup mutex + */ +static struct res_counter *common_ancestor; + +/* + * This does more than just probing the ability to attach to the dest cgroup. + * We can not just _check_ if we can attach to the destination and do the real + * attachment later in task_counter_attach_task() because a task in the dest + * cgroup can fork before and steal the last remaining count. + * Thus we need to charge the dest cgroup right now. + */ +static int task_counter_can_attach_task(struct cgroup *cgrp, + struct cgroup *old_cgrp, + struct task_struct *tsk) +{ + struct res_counter *res = cgroup_task_res_counter(cgrp); + struct res_counter *old_res = cgroup_task_res_counter(old_cgrp); + int err; + + /* + * When moving a task from a cgroup to another, we don't want + * to charge the common ancestors, even though they will be + * uncharged later from attach_task(), because during that + * short window between charge and uncharge, a task could fork + * in the ancestor and spuriously fail due to the temporary + * charge. + */ + common_ancestor = res_counter_common_ancestor(res, old_res); + + /* + * If cgrp is the root then res is NULL, however in this case + * the common ancestor is NULL as well, making the below a NOP. + */ + err = res_counter_charge_until(res, common_ancestor, 1, NULL); + if (err) + return -EINVAL; + + return 0; +} + +/* Uncharge the dest cgroup that we charged in task_counter_can_attach_task() */ +static void task_counter_cancel_attach_task(struct cgroup *cgrp, + struct task_struct *tsk) +{ + res_counter_uncharge_until(cgroup_task_res_counter(cgrp), + common_ancestor, 1); +} + +/* + * This uncharge the old cgroup. We can do that now that we are sure the + * attachment can't cancelled anymore, because this uncharge operation + * couldn't be reverted later: a task in the old cgroup could fork after + * we uncharge and reach the task counter limit, making our return there + * not possible. + */ +static void task_counter_attach_task(struct cgroup *cgrp, + struct cgroup *old_cgrp, + struct task_struct *tsk) +{ + res_counter_uncharge_until(cgroup_task_res_counter(old_cgrp), + common_ancestor, 1); +} + +static u64 task_counter_read_u64(struct cgroup *cgrp, struct cftype *cft) +{ + int type = cft->private; + + return res_counter_read_u64(cgroup_task_res_counter(cgrp), type); +} + +static int task_counter_write_u64(struct cgroup *cgrp, struct cftype *cft, + u64 val) +{ + int type = cft->private; + + res_counter_write_u64(cgroup_task_res_counter(cgrp), type, val); + + return 0; +} + +static struct cftype files[] = { + { + .name = "limit", + .read_u64 = task_counter_read_u64, + .write_u64 = task_counter_write_u64, + .private = RES_LIMIT, + }, + + { + .name = "usage", + .read_u64 = task_counter_read_u64, + .private = RES_USAGE, + }, +}; + +static int task_counter_populate(struct cgroup_subsys *ss, struct cgroup *cgrp) +{ + if (!cgrp->parent) + return 0; + + return cgroup_add_files(cgrp, ss, files, ARRAY_SIZE(files)); +} + +/* + * Charge the task counter with the new child coming, or reject it if we + * reached the limit. + */ +int cgroup_task_counter_fork(struct task_struct *child) +{ + struct cgroup_subsys_state *css; + struct cgroup *cgrp; + int err; + + css = child->cgroups->subsys[tasks_subsys_id]; + cgrp = css->cgroup; + + /* Optimize for the root cgroup case, which doesn't have a limit */ + if (!cgrp->parent) + return 0; + + err = res_counter_charge(cgroup_task_res_counter(cgrp), 1, NULL); + if (err) + return -EAGAIN; + + return 0; +} + +struct cgroup_subsys tasks_subsys = { + .name = "tasks", + .subsys_id = tasks_subsys_id, + .create = task_counter_create, + .post_clone = task_counter_post_clone, + .destroy = task_counter_destroy, + .exit = task_counter_exit, + .can_attach_task = task_counter_can_attach_task, + .cancel_attach_task = task_counter_cancel_attach_task, + .attach_task = task_counter_attach_task, + .populate = task_counter_populate, +}; diff --git a/kernel/fork.c b/kernel/fork.c index 8e6b6f4..f716436 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1309,6 +1309,10 @@ static struct task_struct *copy_process(unsigned long clone_flags, p->group_leader = p; INIT_LIST_HEAD(&p->thread_group); + retval = cgroup_task_counter_fork(p); + if (retval) + goto bad_fork_free_pid; + /* Now that the task is set up, run cgroup callbacks if * necessary. We need to run them before the task is visible * on the tasklist. */ -- 1.7.5.4