[PATCH] rlimits: do not grab tasklist_lock for do_prlimit on current

[PATCH] rlimits: do not grab tasklist_lock for do_prlimit on current

[Barret Rhoden]

The tasklist_lock can be a scalability bottleneck.  For current tasks,
we don't need the tasklist_lock to protect tsk->sighand or tsk->signal.
If non-current callers become a bottleneck, we could use
lock_task_sighand().

Signed-off-by: Barret Rhoden <brho@google.com>
---
 kernel/sys.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/kernel/sys.c b/kernel/sys.c
index 8fdac0d90504..e56d1ae910af 100644
--- a/kernel/sys.c
+++ b/kernel/sys.c
@@ -1576,7 +1576,8 @@ int do_prlimit(struct task_struct *tsk, unsigned int resource,
 	}
 
 	/* protect tsk->signal and tsk->sighand from disappearing */
-	read_lock(&tasklist_lock);
+	if (tsk != current)
+		read_lock(&tasklist_lock);
 	if (!tsk->sighand) {
 		retval = -ESRCH;
 		goto out;
@@ -1611,7 +1612,8 @@ int do_prlimit(struct task_struct *tsk, unsigned int resource,
 	     IS_ENABLED(CONFIG_POSIX_TIMERS))
 		update_rlimit_cpu(tsk, new_rlim->rlim_cur);
 out:
-	read_unlock(&tasklist_lock);
+	if (tsk != current)
+		read_unlock(&tasklist_lock);
 	return retval;
 }
 
-- 
2.34.1.173.g76aa8bc2d0-goog

Re: [PATCH] rlimits: do not grab tasklist_lock for do_prlimit on current

[Eric W. Biederman]

Barret Rhoden <brho@google.com> writes:

> The tasklist_lock can be a scalability bottleneck.  For current tasks,
> we don't need the tasklist_lock to protect tsk->sighand or tsk->signal.
> If non-current callers become a bottleneck, we could use
> lock_task_sighand().

Do you have any numbers?  As the entire point of this change is
performance it would be good to see how the performance changes.

Especially as a read_lock should not be too bad as it allows sharing,
nor do I expect reading or writing the rlimit values to be particularly
frequent.  So some insight into what kinds of userspace patterns make
this a problem would be nice.

This change is a bit scary as it makes taking a lock conditional and
increases the probability of causing a locking mistake.

If you are going to make this change I would say that do_prlimit should
become static and taking the tasklist_lock should move into prlimit64.


Looking a little closer it looks like that update_rlimit_cpu should use
lock_task_sighand, and once lock_task_sighand is used there is actually
no need for the tasklist_lock at all.  As holding the reference to tsk
guarantees that tsk->signal remains valid.

So I completely agree there are cleanups that can happen in this area.
Please make those and show numbers in how they improve things, instead
of making the code worse with a conditional lock.

Eric


> Signed-off-by: Barret Rhoden <brho@google.com>
> ---
>  kernel/sys.c | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/kernel/sys.c b/kernel/sys.c
> index 8fdac0d90504..e56d1ae910af 100644
> --- a/kernel/sys.c
> +++ b/kernel/sys.c
> @@ -1576,7 +1576,8 @@ int do_prlimit(struct task_struct *tsk, unsigned int resource,
>  	}
>  
>  	/* protect tsk->signal and tsk->sighand from disappearing */
> -	read_lock(&tasklist_lock);
> +	if (tsk != current)
> +		read_lock(&tasklist_lock);
>  	if (!tsk->sighand) {
>  		retval = -ESRCH;
>  		goto out;
> @@ -1611,7 +1612,8 @@ int do_prlimit(struct task_struct *tsk, unsigned int resource,
>  	     IS_ENABLED(CONFIG_POSIX_TIMERS))
>  		update_rlimit_cpu(tsk, new_rlim->rlim_cur);
>  out:
> -	read_unlock(&tasklist_lock);
> +	if (tsk != current)
> +		read_unlock(&tasklist_lock);
>  	return retval;
>  }

Re: [PATCH] rlimits: do not grab tasklist_lock for do_prlimit on current

[Barret Rhoden]

Hi -

On 12/13/21 5:34 PM, Eric W. Biederman wrote:
> Do you have any numbers?  As the entire point of this change is
> performance it would be good to see how the performance changes.
> 
> Especially as a read_lock should not be too bad as it allows sharing,
> nor do I expect reading or writing the rlimit values to be particularly
> frequent.  So some insight into what kinds of userspace patterns make
> this a problem would be nice.

This was motivated by slowdowns we observed on a few machines running 
tests in a cluster.  AFAIK, there were a lot of small tests, many of 
which mucked with process management syscalls while fork/joining other 
tasks.

Based on a cycles profile, it looked like ~87% of the time was spent in 
the kernel, ~42% of which was just trying to get *some* spinlock 
(queued_spin_lock_slowpath, not necessarily the tasklist_lock).

The big offenders (with rough percentages in cycles of the overall trace):

- do_wait 11%
- setpriority 8% (potential future patch)
- kill 8%
- do_exit 5%
- clone 3%
- prlimit64 2%   (this patch)
- getrlimit 1%   (this patch)

Even though do_prlimit was using a read_lock, it was still contending on 
the internal queued_spin_lock.

The prlimit was only 3% of the total.  This patch was more of a "oh, 
this doesn't *need* the tasklist_lock for p == current" - can we fix 
that?  I actually don't even know often those prlimit64 calls had p == 
current.

setpriority was a bigger one too - is the tasklist lock only needed for 
the PGRP ops?  (I thought so based on where the tasklist_lock is write 
locked and the comment on task_pgrp()).  If so, I could do that in 
another patch.

> This change is a bit scary as it makes taking a lock conditional and
> increases the probability of causing a locking mistake.

I definitely see how making the code more brittle might not be worth the 
small win.  If this is more "damage" than "cleanup", then I can drop it.

> If you are going to make this change I would say that do_prlimit should
> become static and taking the tasklist_lock should move into prlimit64.
> 
> 
> Looking a little closer it looks like that update_rlimit_cpu should use
> lock_task_sighand, and once lock_task_sighand is used there is actually
> no need for the tasklist_lock at all.  As holding the reference to tsk
> guarantees that tsk->signal remains valid.

Maybe do both?  unconditionally grab lock_task_sighand (instead of 
tasklist_lock) in prlimit64.

> So I completely agree there are cleanups that can happen in this area.
> Please make those and show numbers in how they improve things, instead
> of making the code worse with a conditional lock.

Unfortunately, I can't easily get a "before and after" on this change. 
The motivating issue popped up sporadically, but getting it to happen in 
a setup under *my* control is organizationally a pain.  So I understand 
if you wouldn't want the patch for that reason.  Ideally, the changes 
would make the code easier to follow and clearer about why we're locking.

If you're OK with two patches that 1) grab lock_task_sighand in 
prlimit64 and 2) moving the read_lock in {set,get}priority into the PGRP 
cases (assuming I was correct on that), I can send them out.

If it's too much of a risk/ugliness for not clear enough gain (in code 
quality or performance), I'm fine with dropping it.

Thanks for looking,

Barret

Re: [PATCH] rlimits: do not grab tasklist_lock for do_prlimit on current

[Eric W. Biederman]

Barret Rhoden <brho@google.com> writes:

> Hi -
>
> On 12/13/21 5:34 PM, Eric W. Biederman wrote:
>> Do you have any numbers?  As the entire point of this change is
>> performance it would be good to see how the performance changes.
>>
>> Especially as a read_lock should not be too bad as it allows sharing,
>> nor do I expect reading or writing the rlimit values to be particularly
>> frequent.  So some insight into what kinds of userspace patterns make
>> this a problem would be nice.
>
> This was motivated by slowdowns we observed on a few machines running
> tests in a cluster.  AFAIK, there were a lot of small tests, many of
> which mucked with process management syscalls while fork/joining other
> tasks.
>
> Based on a cycles profile, it looked like ~87% of the time was spent
> in the kernel, ~42% of which was just trying to get *some* spinlock
> (queued_spin_lock_slowpath, not necessarily the tasklist_lock).
>
> The big offenders (with rough percentages in cycles of the overall trace):
>
> - do_wait 11%
> - setpriority 8% (potential future patch)
> - kill 8%
> - do_exit 5%
> - clone 3%
> - prlimit64 2%   (this patch)
> - getrlimit 1%   (this patch)
>
> Even though do_prlimit was using a read_lock, it was still contending
> on the internal queued_spin_lock.
>
> The prlimit was only 3% of the total.  This patch was more of a "oh,
> this doesn't *need* the tasklist_lock for p == current" - can we fix
> that?  I actually don't even know often those prlimit64 calls had p ==
> current.
>
> setpriority was a bigger one too - is the tasklist lock only needed
> for the PGRP ops?  (I thought so based on where the tasklist_lock is
> write locked and the comment on task_pgrp()).  If so, I could do that
> in another patch.

That is my understanding.  For setpriority to change everything
atomically it must hold the tasklist lock when dealing with more than
one process at a time.

>> This change is a bit scary as it makes taking a lock conditional and
>> increases the probability of causing a locking mistake.
>
> I definitely see how making the code more brittle might not be worth
> the small win.  If this is more "damage" than "cleanup", then I can
> drop it.
>
>> If you are going to make this change I would say that do_prlimit should
>> become static and taking the tasklist_lock should move into prlimit64.
>>
>>
>> Looking a little closer it looks like that update_rlimit_cpu should use
>> lock_task_sighand, and once lock_task_sighand is used there is actually
>> no need for the tasklist_lock at all.  As holding the reference to tsk
>> guarantees that tsk->signal remains valid.
>
> Maybe do both?  unconditionally grab lock_task_sighand (instead of
> tasklist_lock) in prlimit64.

In update_rlimit_cpu use lock_task_sighand instead of unconditionally
grabbing sighand->siglock (because without tasklist_lock sighand might
be NULL).

Then do_prlimit64 can drop the tasklist lock and the test for
sighand == NULL.

This will address every prlimit64 case instead of just when updating
current.  With prlimit64 in your profile I expect some of those are
non-current.


>> So I completely agree there are cleanups that can happen in this area.
>> Please make those and show numbers in how they improve things, instead
>> of making the code worse with a conditional lock.
>
> Unfortunately, I can't easily get a "before and after" on this
> change. The motivating issue popped up sporadically, but getting it to
> happen in a setup under *my* control is organizationally a pain.  So I
> understand if you wouldn't want the patch for that reason.  Ideally,
> the changes would make the code easier to follow and clearer about why
> we're locking.

Even a microbenchmark that stresses the lock and can show the
performance impact of the change you are making is useful.

Simply reorganizing and removing an unnecessary lock lowers the bar
because the code is then both simpler and pretty much by definition
faster.  Although sometimes another lock is then hit and the contention
moves.

> If you're OK with two patches that 1) grab lock_task_sighand in
> prlimit64 and 2) moving the read_lock in {set,get}priority into the
> PGRP cases (assuming I was correct on that), I can send them out.

I think getpriority can only use the rcu_read_lock.  I don't think it
has any atomicity guarantees.

For setpriority the single process case should be safe just use rcu.
I haven't read through all of what set_one_prio is doing to confirm it
is safe, but in principle it should be safe.

> If it's too much of a risk/ugliness for not clear enough gain (in code
> quality or performance), I'm fine with dropping it.

Removing the tasklist_lock where we can is definitely a clear gain.

Simply shoving tasklist_lock aside and making the code more complicated
is much less clear.

Plus anything you can benchmark (even microbenchmark) and show the
benefit of is welcome.  Especially when you have indications that it
makes a difference in a larger context.

Eric

Re: [PATCH] rlimits: do not grab tasklist_lock for do_prlimit on current

[Cyrill Gorcunov]

On Wed, Dec 15, 2021 at 01:42:32PM -0600, Eric W. Biederman wrote:
...
> 
> > If it's too much of a risk/ugliness for not clear enough gain (in code
> > quality or performance), I'm fine with dropping it.
> 
> Removing the tasklist_lock where we can is definitely a clear gain.
> 
> Simply shoving tasklist_lock aside and making the code more complicated
> is much less clear.
> 
> Plus anything you can benchmark (even microbenchmark) and show the
> benefit of is welcome.  Especially when you have indications that it
> makes a difference in a larger context.

Thanks for looking into this, Eric! I must confess I've a vague memory
about this code. Still while you're talking about cleanup I wonder if
we should make do_prlimit() being a static function, not global as it
now.

Re: [PATCH] rlimits: do not grab tasklist_lock for do_prlimit on current

[Barret Rhoden]

On 12/15/21 14:42, Eric W. Biederman wrote:
> In update_rlimit_cpu use lock_task_sighand instead of unconditionally
> grabbing sighand->siglock (because without tasklist_lock sighand might
> be NULL).

this ended up being a minor complication, since update_rlimit_cpu() 
could fail if the task was exiting, but i think i sorted it out.

i'll send out revised patchset shortly with this change, including 
making do_prlimit() static.

thanks,

barret

Re: [PATCH] rlimits: do not grab tasklist_lock for do_prlimit on current

[kernel test robot]

[-- Attachment #1: Type: text/plain, Size: 6572 bytes --]

CC: kbuild-all(a)lists.01.org
In-Reply-To: <20211213220401.1039578-1-brho@google.com>
References: <20211213220401.1039578-1-brho@google.com>
TO: Barret Rhoden <brho@google.com>
TO: "Eric W. Biederman" <ebiederm@xmission.com>
TO: Christian Brauner <christian.brauner@ubuntu.com>
TO: Andrew Morton <akpm@linux-foundation.org>
CC: Linux Memory Management List <linux-mm@kvack.org>
TO: Alexey Gladkov <legion@kernel.org>
TO: William Cohen <wcohen@redhat.com>
TO: Viresh Kumar <viresh.kumar@linaro.org>
TO: Alexey Dobriyan <adobriyan@gmail.com>
TO: Chris Hyser <chris.hyser@oracle.com>
TO: Peter Collingbourne <pcc@google.com>

Hi Barret,

Thank you for the patch! Perhaps something to improve:

[auto build test WARNING on linus/master]
[also build test WARNING on v5.16-rc5 next-20211215]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Barret-Rhoden/rlimits-do-not-grab-tasklist_lock-for-do_prlimit-on-current/20211214-060452
base:   https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git aa50faff4416c869b52dff68a937c84d29e12f4b
:::::: branch date: 3 days ago
:::::: commit date: 3 days ago
config: x86_64-randconfig-c002-20211216 (https://download.01.org/0day-ci/archive/20211217/202112170456.6wFecpJp-lkp(a)intel.com/config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
Reported-by: Julia Lawall <julia.lawall@lip6.fr>


cocci warnings: (new ones prefixed by >>)
>> kernel/sys.c:1617:1-7: preceding lock on line 1580

vim +1617 kernel/sys.c

c022a0acad534f Jiri Slaby        2010-05-04  1560  
1c1e618ddd15f6 Jiri Slaby        2009-08-28  1561  /* make sure you are allowed to change @tsk limits before calling this */
5b41535aac0c07 Jiri Slaby        2010-03-24  1562  int do_prlimit(struct task_struct *tsk, unsigned int resource,
5b41535aac0c07 Jiri Slaby        2010-03-24  1563  		struct rlimit *new_rlim, struct rlimit *old_rlim)
^1da177e4c3f41 Linus Torvalds    2005-04-16  1564  {
5b41535aac0c07 Jiri Slaby        2010-03-24  1565  	struct rlimit *rlim;
86f162f4c75ceb Jiri Slaby        2009-11-14  1566  	int retval = 0;
^1da177e4c3f41 Linus Torvalds    2005-04-16  1567  
^1da177e4c3f41 Linus Torvalds    2005-04-16  1568  	if (resource >= RLIM_NLIMITS)
^1da177e4c3f41 Linus Torvalds    2005-04-16  1569  		return -EINVAL;
5b41535aac0c07 Jiri Slaby        2010-03-24  1570  	if (new_rlim) {
7855c35da7ba16 Jiri Slaby        2009-08-26  1571  		if (new_rlim->rlim_cur > new_rlim->rlim_max)
60fd760fb9ff70 Andrew Morton     2009-02-04  1572  			return -EINVAL;
5b41535aac0c07 Jiri Slaby        2010-03-24  1573  		if (resource == RLIMIT_NOFILE &&
5b41535aac0c07 Jiri Slaby        2010-03-24  1574  				new_rlim->rlim_max > sysctl_nr_open)
^1da177e4c3f41 Linus Torvalds    2005-04-16  1575  			return -EPERM;
5b41535aac0c07 Jiri Slaby        2010-03-24  1576  	}
^1da177e4c3f41 Linus Torvalds    2005-04-16  1577  
1c1e618ddd15f6 Jiri Slaby        2009-08-28  1578  	/* protect tsk->signal and tsk->sighand from disappearing */
2a9a9e76b0164e Barret Rhoden     2021-12-13  1579  	if (tsk != current)
1c1e618ddd15f6 Jiri Slaby        2009-08-28 @1580  		read_lock(&tasklist_lock);
1c1e618ddd15f6 Jiri Slaby        2009-08-28  1581  	if (!tsk->sighand) {
1c1e618ddd15f6 Jiri Slaby        2009-08-28  1582  		retval = -ESRCH;
1c1e618ddd15f6 Jiri Slaby        2009-08-28  1583  		goto out;
1c1e618ddd15f6 Jiri Slaby        2009-08-28  1584  	}
1c1e618ddd15f6 Jiri Slaby        2009-08-28  1585  
5b41535aac0c07 Jiri Slaby        2010-03-24  1586  	rlim = tsk->signal->rlim + resource;
86f162f4c75ceb Jiri Slaby        2009-11-14  1587  	task_lock(tsk->group_leader);
5b41535aac0c07 Jiri Slaby        2010-03-24  1588  	if (new_rlim) {
fc832ad3645f05 Serge E. Hallyn   2011-03-23  1589  		/* Keep the capable check against init_user_ns until
fc832ad3645f05 Serge E. Hallyn   2011-03-23  1590  		   cgroups can contain all limits */
5b41535aac0c07 Jiri Slaby        2010-03-24  1591  		if (new_rlim->rlim_max > rlim->rlim_max &&
86f162f4c75ceb Jiri Slaby        2009-11-14  1592  				!capable(CAP_SYS_RESOURCE))
86f162f4c75ceb Jiri Slaby        2009-11-14  1593  			retval = -EPERM;
86f162f4c75ceb Jiri Slaby        2009-11-14  1594  		if (!retval)
cad4ea546b1a8a Eric W. Biederman 2017-04-12  1595  			retval = security_task_setrlimit(tsk, resource, new_rlim);
5b41535aac0c07 Jiri Slaby        2010-03-24  1596  	}
5b41535aac0c07 Jiri Slaby        2010-03-24  1597  	if (!retval) {
5b41535aac0c07 Jiri Slaby        2010-03-24  1598  		if (old_rlim)
5b41535aac0c07 Jiri Slaby        2010-03-24  1599  			*old_rlim = *rlim;
5b41535aac0c07 Jiri Slaby        2010-03-24  1600  		if (new_rlim)
5b41535aac0c07 Jiri Slaby        2010-03-24  1601  			*rlim = *new_rlim;
5b41535aac0c07 Jiri Slaby        2010-03-24  1602  	}
7855c35da7ba16 Jiri Slaby        2009-08-26  1603  	task_unlock(tsk->group_leader);
^1da177e4c3f41 Linus Torvalds    2005-04-16  1604  
d3561f78fd379a Andrew Morton     2006-03-24  1605  	/*
24db4dd90dd53a Thomas Gleixner   2019-08-21  1606  	 * RLIMIT_CPU handling. Arm the posix CPU timer if the limit is not
5afe69c2ccd069 Xiaofeng Cao      2021-05-06  1607  	 * infinite. In case of RLIM_INFINITY the posix CPU timer code
24db4dd90dd53a Thomas Gleixner   2019-08-21  1608  	 * ignores the rlimit.
d3561f78fd379a Andrew Morton     2006-03-24  1609  	 */
5b41535aac0c07 Jiri Slaby        2010-03-24  1610  	 if (!retval && new_rlim && resource == RLIMIT_CPU &&
baa73d9e478ff3 Nicolas Pitre     2016-11-11  1611  	     new_rlim->rlim_cur != RLIM_INFINITY &&
baa73d9e478ff3 Nicolas Pitre     2016-11-11  1612  	     IS_ENABLED(CONFIG_POSIX_TIMERS))
7855c35da7ba16 Jiri Slaby        2009-08-26  1613  		update_rlimit_cpu(tsk, new_rlim->rlim_cur);
ec9e16bacdba1d Andrew Morton     2006-03-24  1614  out:
2a9a9e76b0164e Barret Rhoden     2021-12-13  1615  	if (tsk != current)
1c1e618ddd15f6 Jiri Slaby        2009-08-28  1616  		read_unlock(&tasklist_lock);
2fb9d2689a0041 Oleg Nesterov     2009-09-03 @1617  	return retval;
^1da177e4c3f41 Linus Torvalds    2005-04-16  1618  }
^1da177e4c3f41 Linus Torvalds    2005-04-16  1619  

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org