[PATCH] cgroup/cpuset: Change nr_deadline_tasks to an atomic_t value

[PATCH] cgroup/cpuset: Change nr_deadline_tasks to an atomic_t value

[Waiman Long]

The nr_deadline_tasks field in cpuset structure was introduced by
commit 6c24849f5515 ("sched/cpuset: Keep track of SCHED_DEADLINE task
in cpusets"). Unlike nr_migrate_dl_tasks which is only modified under
cpuset_mutex, nr_deadline_tasks can be updated in various contexts
under different locks. As a result, data races may happen that cause
incorrect value to be stored in nr_deadline_tasks leading to incorrect
exit from dl_update_tasks_root_domain().  Fix that data race problem
by making nr_deadline_tasks an atomic_t value.

Fixes: 6c24849f5515 ("sched/cpuset: Keep track of SCHED_DEADLINE task in cpusets")
Reported-by: Xia Fukun <xiafukun@huawei.com>
Signed-off-by: Waiman Long <longman@redhat.com>
---
 kernel/cgroup/cpuset.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 58ec88efa4f8..3f3da468f058 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -174,7 +174,7 @@ struct cpuset {
 	 * number of SCHED_DEADLINE tasks attached to this cpuset, so that we
 	 * know when to rebuild associated root domain bandwidth information.
 	 */
-	int nr_deadline_tasks;
+	atomic_t nr_deadline_tasks;
 	int nr_migrate_dl_tasks;
 	u64 sum_migrate_dl_bw;
 
@@ -234,14 +234,14 @@ void inc_dl_tasks_cs(struct task_struct *p)
 {
 	struct cpuset *cs = task_cs(p);
 
-	cs->nr_deadline_tasks++;
+	atomic_inc(&cs->nr_deadline_tasks);
 }
 
 void dec_dl_tasks_cs(struct task_struct *p)
 {
 	struct cpuset *cs = task_cs(p);
 
-	cs->nr_deadline_tasks--;
+	atomic_dec(&cs->nr_deadline_tasks);
 }
 
 /* bits in struct cpuset flags field */
@@ -1071,7 +1071,7 @@ static void dl_update_tasks_root_domain(struct cpuset *cs)
 	struct css_task_iter it;
 	struct task_struct *task;
 
-	if (cs->nr_deadline_tasks == 0)
+	if (atomic_read(&cs->nr_deadline_tasks) == 0)
 		return;
 
 	css_task_iter_start(&cs->css, 0, &it);
@@ -2721,8 +2721,8 @@ static void cpuset_attach(struct cgroup_taskset *tset)
 	cs->old_mems_allowed = cpuset_attach_nodemask_to;
 
 	if (cs->nr_migrate_dl_tasks) {
-		cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks;
-		oldcs->nr_deadline_tasks -= cs->nr_migrate_dl_tasks;
+		atomic_add(cs->nr_migrate_dl_tasks, &cs->nr_deadline_tasks);
+		atomic_sub(cs->nr_migrate_dl_tasks, &oldcs->nr_deadline_tasks);
 		reset_migrate_dl_data(cs);
 	}
 
-- 
2.39.3

Re: [PATCH] cgroup/cpuset: Change nr_deadline_tasks to an atomic_t value

[Juri Lelli]

Hi,

On 09/10/23 15:15, Waiman Long wrote:
> The nr_deadline_tasks field in cpuset structure was introduced by
> commit 6c24849f5515 ("sched/cpuset: Keep track of SCHED_DEADLINE task
> in cpusets"). Unlike nr_migrate_dl_tasks which is only modified under
> cpuset_mutex, nr_deadline_tasks can be updated in various contexts
> under different locks. As a result, data races may happen that cause
> incorrect value to be stored in nr_deadline_tasks leading to incorrect

Could you please make an example of such data races?

Thanks!
Juri

Re: [PATCH] cgroup/cpuset: Change nr_deadline_tasks to an atomic_t value

[Waiman Long]

On 10/10/23 01:34, Juri Lelli wrote:
> Hi,
>
> On 09/10/23 15:15, Waiman Long wrote:
>> The nr_deadline_tasks field in cpuset structure was introduced by
>> commit 6c24849f5515 ("sched/cpuset: Keep track of SCHED_DEADLINE task
>> in cpusets"). Unlike nr_migrate_dl_tasks which is only modified under
>> cpuset_mutex, nr_deadline_tasks can be updated in various contexts
>> under different locks. As a result, data races may happen that cause
>> incorrect value to be stored in nr_deadline_tasks leading to incorrect
> Could you please make an example of such data races?

Since update to cs->nr_deadline_tasks is not protected by a single lock, 
it is possible that multiple CPUs may try to modify it at the same 
time.  It is possible that nr_deadline_tasks++ and nr_deadline_tasks-- 
can be done in a single instruction like in x86 and hence atomic. 
However, operation like "cs->nr_deadline_tasks += 
cs->nr_migrate_dl_tasks" is likely a RMW operation and so is subjected 
to racing. It is mostly theoretical, but probably not impossible.

Cheers,
Longman

Re: [PATCH] cgroup/cpuset: Change nr_deadline_tasks to an atomic_t value

[Waiman Long]

On 10/10/23 15:44, Waiman Long wrote:
>
> On 10/10/23 01:34, Juri Lelli wrote:
>> Hi,
>>
>> On 09/10/23 15:15, Waiman Long wrote:
>>> The nr_deadline_tasks field in cpuset structure was introduced by
>>> commit 6c24849f5515 ("sched/cpuset: Keep track of SCHED_DEADLINE task
>>> in cpusets"). Unlike nr_migrate_dl_tasks which is only modified under
>>> cpuset_mutex, nr_deadline_tasks can be updated in various contexts
>>> under different locks. As a result, data races may happen that cause
>>> incorrect value to be stored in nr_deadline_tasks leading to incorrect
>> Could you please make an example of such data races?
>
> Since update to cs->nr_deadline_tasks is not protected by a single 
> lock, it is possible that multiple CPUs may try to modify it at the 
> same time.  It is possible that nr_deadline_tasks++ and 
> nr_deadline_tasks-- can be done in a single instruction like in x86 
> and hence atomic. However, operation like "cs->nr_deadline_tasks += 
> cs->nr_migrate_dl_tasks" is likely a RMW operation and so is subjected 
> to racing. It is mostly theoretical, but probably not impossible.

Sorry, even increment and decrement operators are not atomic.

inc_dl_tasks_cs() is only called from switched_to_dl() in deadline.c 
which is protected by the rq_lock, but there are multiple rq's. 
dec_dl_tasks_cs() is called from switched_from_dl() in deadline.c and 
cgroup_exit() in cgroup.c. The later one is protected by css_set_lock. 
The other place where nr_deadline_tasks can be changed is in 
cpuset_attach() protected by cpuset_mutex.

Cheers,
Longman

Re: [PATCH] cgroup/cpuset: Change nr_deadline_tasks to an atomic_t value

[Juri Lelli]

On 10/10/23 16:03, Waiman Long wrote:
> 
> On 10/10/23 15:44, Waiman Long wrote:
> > 
> > On 10/10/23 01:34, Juri Lelli wrote:
> > > Hi,
> > > 
> > > On 09/10/23 15:15, Waiman Long wrote:
> > > > The nr_deadline_tasks field in cpuset structure was introduced by
> > > > commit 6c24849f5515 ("sched/cpuset: Keep track of SCHED_DEADLINE task
> > > > in cpusets"). Unlike nr_migrate_dl_tasks which is only modified under
> > > > cpuset_mutex, nr_deadline_tasks can be updated in various contexts
> > > > under different locks. As a result, data races may happen that cause
> > > > incorrect value to be stored in nr_deadline_tasks leading to incorrect
> > > Could you please make an example of such data races?
> > 
> > Since update to cs->nr_deadline_tasks is not protected by a single lock,
> > it is possible that multiple CPUs may try to modify it at the same
> > time.  It is possible that nr_deadline_tasks++ and nr_deadline_tasks--
> > can be done in a single instruction like in x86 and hence atomic.
> > However, operation like "cs->nr_deadline_tasks +=
> > cs->nr_migrate_dl_tasks" is likely a RMW operation and so is subjected
> > to racing. It is mostly theoretical, but probably not impossible.
> 
> Sorry, even increment and decrement operators are not atomic.
> 
> inc_dl_tasks_cs() is only called from switched_to_dl() in deadline.c which
> is protected by the rq_lock, but there are multiple rq's. dec_dl_tasks_cs()
> is called from switched_from_dl() in deadline.c and cgroup_exit() in
> cgroup.c. The later one is protected by css_set_lock. The other place where
> nr_deadline_tasks can be changed is in cpuset_attach() protected by
> cpuset_mutex.

So, let's see. :)

switched_to_dl(), switched_from_dl() and cpuset_attach() should all be
protected (for DEADLINE tasks) by cpuset_mutex, see [1] for the former
two.

What leaves me perplexed is indeed cgroup_exit(), which seems to operate
under css_set_lock as you say. I however wonder why is that not racy
already wrt, say, cpuset_attach() which AFAIU uses css information w/o
holding css_set_lock?

Thanks,
Juri

1 - https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L7688

Re: [PATCH] cgroup/cpuset: Change nr_deadline_tasks to an atomic_t value

[Waiman Long]

On 10/11/23 04:14, Juri Lelli wrote:
> On 10/10/23 16:03, Waiman Long wrote:
>> On 10/10/23 15:44, Waiman Long wrote:
>>> On 10/10/23 01:34, Juri Lelli wrote:
>>>> Hi,
>>>>
>>>> On 09/10/23 15:15, Waiman Long wrote:
>>>>> The nr_deadline_tasks field in cpuset structure was introduced by
>>>>> commit 6c24849f5515 ("sched/cpuset: Keep track of SCHED_DEADLINE task
>>>>> in cpusets"). Unlike nr_migrate_dl_tasks which is only modified under
>>>>> cpuset_mutex, nr_deadline_tasks can be updated in various contexts
>>>>> under different locks. As a result, data races may happen that cause
>>>>> incorrect value to be stored in nr_deadline_tasks leading to incorrect
>>>> Could you please make an example of such data races?
>>> Since update to cs->nr_deadline_tasks is not protected by a single lock,
>>> it is possible that multiple CPUs may try to modify it at the same
>>> time.  It is possible that nr_deadline_tasks++ and nr_deadline_tasks--
>>> can be done in a single instruction like in x86 and hence atomic.
>>> However, operation like "cs->nr_deadline_tasks +=
>>> cs->nr_migrate_dl_tasks" is likely a RMW operation and so is subjected
>>> to racing. It is mostly theoretical, but probably not impossible.
>> Sorry, even increment and decrement operators are not atomic.
>>
>> inc_dl_tasks_cs() is only called from switched_to_dl() in deadline.c which
>> is protected by the rq_lock, but there are multiple rq's. dec_dl_tasks_cs()
>> is called from switched_from_dl() in deadline.c and cgroup_exit() in
>> cgroup.c. The later one is protected by css_set_lock. The other place where
>> nr_deadline_tasks can be changed is in cpuset_attach() protected by
>> cpuset_mutex.
> So, let's see. :)
>
> switched_to_dl(), switched_from_dl() and cpuset_attach() should all be
> protected (for DEADLINE tasks) by cpuset_mutex, see [1] for the former
> two.
Yes, I missed the cpuset_lock() call.
> What leaves me perplexed is indeed cgroup_exit(), which seems to operate
> under css_set_lock as you say. I however wonder why is that not racy
> already wrt, say, cpuset_attach() which AFAIU uses css information w/o
> holding css_set_lock?

The css_set_lock protects changes made to css_set. Looking at 
cgroup_migrate_execute(), css_set_lock is taken when the tasks are 
actually moving from one css_set to another one. cpuset_attach() is 
called just to update the CPU and node affinity and cpuset_mutex is 
taken to ensure stability of the CPU and node masks. There is no change 
to css_set and so css_set_lock isn't needed.

We can argue that there can be racing between cgroup_exit() and the 
iteration of tasks in cpuset_attach() or cpuset_can_attach(). An 
rcu_read_lock() is probably needed. I am stilling investigating that.

Cheers,
Longman


>
> Thanks,
> Juri
>
> 1 - https://elixir.bootlin.com/linux/latest/source/kernel/sched/core.c#L7688
>

Re: [PATCH] cgroup/cpuset: Change nr_deadline_tasks to an atomic_t value

[Waiman Long]

On 10/11/23 08:54, Waiman Long wrote:
>
> On 10/11/23 04:14, Juri Lelli wrote:
>> On 10/10/23 16:03, Waiman Long wrote:
>>> On 10/10/23 15:44, Waiman Long wrote:
>>>> On 10/10/23 01:34, Juri Lelli wrote:
>>>>> Hi,
>>>>>
>>>>> On 09/10/23 15:15, Waiman Long wrote:
>>>>>> The nr_deadline_tasks field in cpuset structure was introduced by
>>>>>> commit 6c24849f5515 ("sched/cpuset: Keep track of SCHED_DEADLINE 
>>>>>> task
>>>>>> in cpusets"). Unlike nr_migrate_dl_tasks which is only modified 
>>>>>> under
>>>>>> cpuset_mutex, nr_deadline_tasks can be updated in various contexts
>>>>>> under different locks. As a result, data races may happen that cause
>>>>>> incorrect value to be stored in nr_deadline_tasks leading to 
>>>>>> incorrect
>>>>> Could you please make an example of such data races?
>>>> Since update to cs->nr_deadline_tasks is not protected by a single 
>>>> lock,
>>>> it is possible that multiple CPUs may try to modify it at the same
>>>> time.  It is possible that nr_deadline_tasks++ and nr_deadline_tasks--
>>>> can be done in a single instruction like in x86 and hence atomic.
>>>> However, operation like "cs->nr_deadline_tasks +=
>>>> cs->nr_migrate_dl_tasks" is likely a RMW operation and so is subjected
>>>> to racing. It is mostly theoretical, but probably not impossible.
>>> Sorry, even increment and decrement operators are not atomic.
>>>
>>> inc_dl_tasks_cs() is only called from switched_to_dl() in deadline.c 
>>> which
>>> is protected by the rq_lock, but there are multiple rq's. 
>>> dec_dl_tasks_cs()
>>> is called from switched_from_dl() in deadline.c and cgroup_exit() in
>>> cgroup.c. The later one is protected by css_set_lock. The other 
>>> place where
>>> nr_deadline_tasks can be changed is in cpuset_attach() protected by
>>> cpuset_mutex.
>> So, let's see. :)
>>
>> switched_to_dl(), switched_from_dl() and cpuset_attach() should all be
>> protected (for DEADLINE tasks) by cpuset_mutex, see [1] for the former
>> two.
> Yes, I missed the cpuset_lock() call.
>> What leaves me perplexed is indeed cgroup_exit(), which seems to operate
>> under css_set_lock as you say. I however wonder why is that not racy
>> already wrt, say, cpuset_attach() which AFAIU uses css information w/o
>> holding css_set_lock?
>
> The css_set_lock protects changes made to css_set. Looking at 
> cgroup_migrate_execute(), css_set_lock is taken when the tasks are 
> actually moving from one css_set to another one. cpuset_attach() is 
> called just to update the CPU and node affinity and cpuset_mutex is 
> taken to ensure stability of the CPU and node masks. There is no 
> change to css_set and so css_set_lock isn't needed.
>
> We can argue that there can be racing between cgroup_exit() and the 
> iteration of tasks in cpuset_attach() or cpuset_can_attach(). An 
> rcu_read_lock() is probably needed. I am stilling investigating that.

Cgroup has a rather complex task migration and iteration scheme. 
According to the following comments in include/linux/cgroup-defs.h:

         /*
          * Lists running through all tasks using this cgroup group.
          * mg_tasks lists tasks which belong to this cset but are in the
          * process of being migrated out or in.  Protected by
          * css_set_lock, but, during migration, once tasks are moved to
          * mg_tasks, it can be read safely while holding cgroup_mutex.
          */
         struct list_head tasks;
         struct list_head mg_tasks;
         struct list_head dying_tasks;

I haven't fully figured out how that protection works yet. Assuming that 
is the case, task iteration in cpuset_attach() should be fine since 
cgroup_mutex is indeed held when it is invoked. That protection, 
however, does not applied to nr_deadline_tasks. It may be too costly to 
acquire cpuset_mutex before updating nr_deadline_tasks in cgroup_exit(). 
So changing it to an atomic_t should be the easy way out of the 
potential racing problem.

I can update the commit log with these new analysis if you have no 
further objection to this change.

Cheers,
Longman

Re: [PATCH] cgroup/cpuset: Change nr_deadline_tasks to an atomic_t value

[Juri Lelli]

On 12/10/23 12:35, Waiman Long wrote:
> On 10/11/23 08:54, Waiman Long wrote:

...

> > We can argue that there can be racing between cgroup_exit() and the
> > iteration of tasks in cpuset_attach() or cpuset_can_attach(). An
> > rcu_read_lock() is probably needed. I am stilling investigating that.
> 
> Cgroup has a rather complex task migration and iteration scheme. According
> to the following comments in include/linux/cgroup-defs.h:
> 
>         /*
>          * Lists running through all tasks using this cgroup group.
>          * mg_tasks lists tasks which belong to this cset but are in the
>          * process of being migrated out or in.  Protected by
>          * css_set_lock, but, during migration, once tasks are moved to
>          * mg_tasks, it can be read safely while holding cgroup_mutex.
>          */
>         struct list_head tasks;
>         struct list_head mg_tasks;
>         struct list_head dying_tasks;
> 
> I haven't fully figured out how that protection works yet. Assuming that is
> the case, task iteration in cpuset_attach() should be fine since
> cgroup_mutex is indeed held when it is invoked. That protection, however,
> does not applied to nr_deadline_tasks. It may be too costly to acquire
> cpuset_mutex before updating nr_deadline_tasks in cgroup_exit(). So changing
> it to an atomic_t should be the easy way out of the potential racing
> problem.

My biggest perplexity is/was still about dl_rebuild_rd_accounting() and
cgroup_exit(); I wonder if the latter, operating outside cpuset_mutex
guard, might still be racy wrt the former (even if we change to
atomic_t).

However, looking again at it, dl_rebuild_rd_accounting() operates on
css(es) via css_task_iter_start(), which grabs css_set_lock. So maybe we
are OK already also for this case?

Apologies for being pedantic, but we fought already several times with
races around these bits and now I'm probably over-suspicious. :)

Thanks,
Juri