[ia64] Question on __ARCH_WANT_UNLOCKED_CTXSW

[ia64] Question on __ARCH_WANT_UNLOCKED_CTXSW

[Peter Zijlstra]

Ken, Tony,

happen to remember what the perceived benefit of using
__ARCH_WANT_UNLOCKED_CTXSW was about?

---
commit f8efa27662532ad5adb2790bfc3f4c78e019cfad
Author: Chen, Kenneth W <kenneth.w.chen@intel.com>
Date:   Thu Jan 26 18:24:59 2006 -0800

    [IA64] remove staled comments in asm/system.h
    
    With the recent optimization made to wrap_mmu_context function,
    we don't hold tasklist_lock anymore when wrapping context id.
    The comments in asm/system.h must fall through the crack earlier.
    Remove staled comments.
    
    I believe it is still beneficial to unlock the runqueue lock
    across context switch. So leave __ARCH_WANT_UNLOCKED_CTXSW on.
    
    Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
    Signed-off-by: Tony Luck <tony.luck@intel.com>

diff --git a/include/asm-ia64/system.h b/include/asm-ia64/system.h
index 80c5a23..0625387 100644
--- a/include/asm-ia64/system.h
+++ b/include/asm-ia64/system.h
@@ -249,32 +249,7 @@ extern void ia64_load_extra (struct task_struct *task);
 # define switch_to(prev,next,last)	__switch_to(prev, next, last)
 #endif
 
-/*
- * On IA-64, we don't want to hold the runqueue's lock during the low-level context-switch,
- * because that could cause a deadlock.  Here is an example by Erich Focht:
- *
- * Example:
- * CPU#0:
- * schedule()
- *    -> spin_lock_irq(&rq->lock)
- *    -> context_switch()
- *       -> wrap_mmu_context()
- *          -> read_lock(&tasklist_lock)
- *
- * CPU#1:
- * sys_wait4() or release_task() or forget_original_parent()
- *    -> write_lock(&tasklist_lock)
- *    -> do_notify_parent()
- *       -> wake_up_parent()
- *          -> try_to_wake_up()
- *             -> spin_lock_irq(&parent_rq->lock)
- *
- * If the parent's rq happens to be on CPU#0, we'll wait for the rq->lock
- * of that CPU which will not be released, because there we wait for the
- * tasklist_lock to become available.
- */
 #define __ARCH_WANT_UNLOCKED_CTXSW
-
 #define ARCH_HAS_PREFETCH_SWITCH_STACK
 #define ia64_platform_is(x) (strcmp(x, platform_name) == 0)
 

Re: [ia64] Question on __ARCH_WANT_UNLOCKED_CTXSW

[Ken Chen]

On Wed, Aug 24, 2011 at 5:27 AM, Peter Zijlstra wrote:
> Ken, Tony,
>
> happen to remember what the perceived benefit of using
> __ARCH_WANT_UNLOCKED_CTXSW was about?

Boy, this was so long ago and my memory is fading.  Low level ia64 arch
has lengthy context switch path in assembly code due to large CPU register
file, IIRC __ARCH_WANT_UNLOCKED_CTXSW gave lower runqueue lock contention
for workloads that was heavy in wakeup and ctx switch events (I remember it
was particularly useful on large smp machine that has hundreds of CPUs).
I don't know whether the situation still applies today when scheduler code
evolved quite a bit over past 5 years.  I think the needs for ia64 require
a fresh re-evaluation.

- Ken

RE: [ia64] Question on __ARCH_WANT_UNLOCKED_CTXSW

[Luck, Tony]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 699 bytes --]

> happen to remember what the perceived benefit of using
> __ARCH_WANT_UNLOCKED_CTXSW was about?

No - digging around the code hasn't rung any bells for me either.

Perhaps just general goodness for not holding a lock for
longer than we need to?  But that would imply some case where
someone else could do something useful by being able to grab
the lock when we drop it. About the only thing I can think
of is that it would allow tasks to be re-balanced just a
teeny bit earlier --- but re-balancing ought to be somewhat
rare, yes?

-Tony
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

RE: [ia64] Question on __ARCH_WANT_UNLOCKED_CTXSW

[Peter Zijlstra]

On Wed, 2011-08-24 at 13:46 -0700, Luck, Tony wrote:
> > happen to remember what the perceived benefit of using
> > __ARCH_WANT_UNLOCKED_CTXSW was about?
> 
> No - digging around the code hasn't rung any bells for me either.
> 
> Perhaps just general goodness for not holding a lock for
> longer than we need to?  But that would imply some case where
> someone else could do something useful by being able to grab
> the lock when we drop it. About the only thing I can think
> of is that it would allow tasks to be re-balanced just a
> teeny bit earlier --- but re-balancing ought to be somewhat
> rare, yes?

Mostly yes, except remote wakeups, however that got a complete overhaul
in 3.0. Instead of taking the remote rq->lock we now enqueue the task on
a list and IPI the thing, then let the IPI do the remote enqueue and
trigger the reschedule.

So it might make sense to re-evaluate this on ia64 like Ken suggested..
then again, who has a large ia64 box and is still willing to put time
in? 

Re: [ia64] Question on __ARCH_WANT_UNLOCKED_CTXSW

[Paul Turner]

On 09/13/11 11:59, Peter Zijlstra wrote:
> On Wed, 2011-08-24 at 13:46 -0700, Luck, Tony wrote:
>>> happen to remember what the perceived benefit of using
>>> __ARCH_WANT_UNLOCKED_CTXSW was about?
>>
>> No - digging around the code hasn't rung any bells for me either.
>>
>> Perhaps just general goodness for not holding a lock for
>> longer than we need to?  But that would imply some case where
>> someone else could do something useful by being able to grab
>> the lock when we drop it. About the only thing I can think
>> of is that it would allow tasks to be re-balanced just a
>> teeny bit earlier --- but re-balancing ought to be somewhat
>> rare, yes?
>
> Mostly yes, except remote wakeups, however that got a complete overhaul
> in 3.0. Instead of taking the remote rq->lock we now enqueue the task on
> a list and IPI the thing, then let the IPI do the remote enqueue and
> trigger the reschedule.
>
> So it might make sense to re-evaluate this on ia64 like Ken suggested..
> then again, who has a large ia64 box and is still willing to put time
> in?

Hum -- perhaps they'll come out of the woodwork if we just rip it out 
(if they exist)

- Paul