* [PATCH] -mm: Small schedule() optimization
@ 2006-03-08 17:54 Andreas Mohr
2006-03-11 1:00 ` Con Kolivas
0 siblings, 1 reply; 4+ messages in thread
From: Andreas Mohr @ 2006-03-08 17:54 UTC (permalink / raw)
To: Andrew Morton; +Cc: linux-kernel, ck
Hello all,
I found that there's a possible small optimization right at the very
beginning of schedule():
if (likely(!current->exit_state)) {
if (unlikely(in_atomic())) {
can be reversed into
if (unlikely(in_atomic())) {
if (likely(!current->exit_state)) {
This is a Good Thing since it avoids having to evaluate both checks,
and both use current_thread_info() which has an inherent AGI stall risk on
x86 CPUs if it cannot be inter-mingled with other unrelated opcodes.
I'm a bit puzzled that this has not been done like that before.
Probably since the exit_state check got added as an after-thought...
Or did I miss some important reason here? (branch prediction??)
Patch against 2.6.16-rc5-mm3.
Thanks!
Signed-off-by: Andreas Mohr <andi@lisas.de>
--- linux-2.6.16-rc5-mm3/kernel/sched.c.orig 2006-03-08 18:36:58.000000000 +0100
+++ linux-2.6.16-rc5-mm3/kernel/sched.c 2006-03-08 18:39:55.000000000 +0100
@@ -3022,8 +3022,8 @@
* schedule() atomically, we ignore that path for now.
* Otherwise, whine if we are scheduling when we should not be.
*/
- if (likely(!current->exit_state)) {
- if (unlikely(in_atomic())) {
+ if (unlikely(in_atomic())) {
+ if (likely(!current->exit_state)) {
printk(KERN_ERR "BUG: scheduling while atomic: "
"%s/0x%08x/%d\n",
current->comm, preempt_count(), current->pid);
--
No programming skills!? Why not help translate many Linux applications!
https://launchpad.ubuntu.com/rosetta
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH] -mm: Small schedule() optimization
2006-03-08 17:54 [PATCH] -mm: Small schedule() optimization Andreas Mohr
@ 2006-03-11 1:00 ` Con Kolivas
2006-03-17 9:13 ` Ingo Molnar
0 siblings, 1 reply; 4+ messages in thread
From: Con Kolivas @ 2006-03-11 1:00 UTC (permalink / raw)
To: Andreas Mohr; +Cc: ck, Andrew Morton, linux-kernel, Ingo Molnar
cc'ed Ingo since he's maintainer.
On Thursday 09 March 2006 04:54, Andreas Mohr wrote:
> Hello all,
>
> I found that there's a possible small optimization right at the very
> beginning of schedule():
>
> if (likely(!current->exit_state)) {
> if (unlikely(in_atomic())) {
>
> can be reversed into
>
> if (unlikely(in_atomic())) {
> if (likely(!current->exit_state)) {
>
> This is a Good Thing since it avoids having to evaluate both checks,
> and both use current_thread_info() which has an inherent AGI stall risk on
> x86 CPUs if it cannot be inter-mingled with other unrelated opcodes.
>
> I'm a bit puzzled that this has not been done like that before.
> Probably since the exit_state check got added as an after-thought...
> Or did I miss some important reason here? (branch prediction??)
This looks good. See below.
> Patch against 2.6.16-rc5-mm3.
>
> Thanks!
>
> Signed-off-by: Andreas Mohr <andi@lisas.de>
>
>
> --- linux-2.6.16-rc5-mm3/kernel/sched.c.orig 2006-03-08 18:36:58.000000000
> +0100 +++ linux-2.6.16-rc5-mm3/kernel/sched.c 2006-03-08 18:39:55.000000000
> +0100 @@ -3022,8 +3022,8 @@
> * schedule() atomically, we ignore that path for now.
> * Otherwise, whine if we are scheduling when we should not be.
> */
> - if (likely(!current->exit_state)) {
> - if (unlikely(in_atomic())) {
> + if (unlikely(in_atomic())) {
> + if (likely(!current->exit_state)) {
I suspect that once we're in_atomic() then we're no longer likely to
be !current->exit_state
Probably better to just
if (unlikely(in_atomic())) {
if (!current->exit_state) {
Ingo?
Cheers,
Con
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [PATCH] -mm: Small schedule() optimization
2006-03-11 1:00 ` Con Kolivas
@ 2006-03-17 9:13 ` Ingo Molnar
2006-03-17 9:52 ` [ck] " Andreas Mohr
0 siblings, 1 reply; 4+ messages in thread
From: Ingo Molnar @ 2006-03-17 9:13 UTC (permalink / raw)
To: Con Kolivas; +Cc: Andreas Mohr, ck, Andrew Morton, linux-kernel
* Con Kolivas <kernel@kolivas.org> wrote:
> > - if (likely(!current->exit_state)) {
> > - if (unlikely(in_atomic())) {
> > + if (unlikely(in_atomic())) {
> > + if (likely(!current->exit_state)) {
>
> I suspect that once we're in_atomic() then we're no longer likely to
> be !current->exit_state
>
> Probably better to just
> if (unlikely(in_atomic())) {
> if (!current->exit_state) {
>
> Ingo?
yeah. There's not much point in nesting likely/unlikely. In fact we can
just merge the two conditions, as per updated patch below.
Ingo
---
From: Andreas Mohr <andi@lisas.de>
small schedule() microoptimization.
Signed-off-by: Andreas Mohr <andi@lisas.de>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
--- linux/kernel/sched.c.orig
+++ linux/kernel/sched.c
@@ -2873,13 +2873,11 @@ asmlinkage void __sched schedule(void)
* schedule() atomically, we ignore that path for now.
* Otherwise, whine if we are scheduling when we should not be.
*/
- if (likely(!current->exit_state)) {
- if (unlikely(in_atomic())) {
- printk(KERN_ERR "scheduling while atomic: "
- "%s/0x%08x/%d\n",
- current->comm, preempt_count(), current->pid);
- dump_stack();
- }
+ if (unlikely(in_atomic() && !current->exit_state)) {
+ printk(KERN_ERR "scheduling while atomic: "
+ "%s/0x%08x/%d\n",
+ current->comm, preempt_count(), current->pid);
+ dump_stack();
}
profile_hit(SCHED_PROFILING, __builtin_return_address(0));
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [ck] Re: [PATCH] -mm: Small schedule() optimization
2006-03-17 9:13 ` Ingo Molnar
@ 2006-03-17 9:52 ` Andreas Mohr
0 siblings, 0 replies; 4+ messages in thread
From: Andreas Mohr @ 2006-03-17 9:52 UTC (permalink / raw)
To: Ingo Molnar; +Cc: Con Kolivas, ck, linux-kernel, Andrew Morton
Hi,
On Fri, Mar 17, 2006 at 10:13:47AM +0100, Ingo Molnar wrote:
>
> * Con Kolivas <kernel@kolivas.org> wrote:
> > Probably better to just
> > if (unlikely(in_atomic())) {
> > if (!current->exit_state) {
> >
> > Ingo?
>
> yeah. There's not much point in nesting likely/unlikely. In fact we can
> just merge the two conditions, as per updated patch below.
ACK, thanks!
> Ingo
Andreas Mohr
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2006-03-17 9:52 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2006-03-08 17:54 [PATCH] -mm: Small schedule() optimization Andreas Mohr
2006-03-11 1:00 ` Con Kolivas
2006-03-17 9:13 ` Ingo Molnar
2006-03-17 9:52 ` [ck] " Andreas Mohr
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox