Scheduler on C2D CPU and latest 2.6.27 kernel

Scheduler on C2D CPU and latest 2.6.27 kernel

[Zdenek Kabelac]

Hi

Recently I'm noticing bad behavior of the CPU scheduler on my T61 (2GB, C2D)

It looks like Linux concentrates all running tasks on one CPU and the
second cpu is sleeping.

With recent changes to DRI - glxgears went up to 840FPS but also takes
100% (with Xorg) and when I run 'while :; do true; done' loop in
parallel frame rate drops to 300FPS.

But as I have C2D CPU I would expect that there should be no such
dramatic slowdown.

Xosview shows that only one CPU is fully loaded.

Here are my .config scheduler options:

CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
# CONFIG_GROUP_SCHED is not set
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_SCHED_SMT=y
# CONFIG_SCHED_MC is not set
CONFIG_SCHED_HRTICK=y
# CONFIG_NET_SCHED is not set
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_SCHED_TRACER=y


Am I missing something?


Zdenek

Re: Scheduler on C2D CPU and latest 2.6.27 kernel

[Chris Snook]

Zdenek Kabelac wrote:
> Hi
> 
> Recently I'm noticing bad behavior of the CPU scheduler on my T61 (2GB, C2D)
> 
> It looks like Linux concentrates all running tasks on one CPU and the
> second cpu is sleeping.
> 
> With recent changes to DRI - glxgears went up to 840FPS but also takes
> 100% (with Xorg) and when I run 'while :; do true; done' loop in
> parallel frame rate drops to 300FPS.
> 
> But as I have C2D CPU I would expect that there should be no such
> dramatic slowdown.
> 
> Xosview shows that only one CPU is fully loaded.
> 
> Here are my .config scheduler options:
> 
> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> # CONFIG_GROUP_SCHED is not set
> CONFIG_IOSCHED_NOOP=y
> CONFIG_IOSCHED_AS=y
> CONFIG_IOSCHED_DEADLINE=y
> CONFIG_IOSCHED_CFQ=y
> CONFIG_DEFAULT_IOSCHED="cfq"
> CONFIG_SCHED_SMT=y
> # CONFIG_SCHED_MC is not set
> CONFIG_SCHED_HRTICK=y
> # CONFIG_NET_SCHED is not set
> CONFIG_USB_EHCI_TT_NEWSCHED=y
> CONFIG_SCHED_DEBUG=y
> CONFIG_SCHEDSTATS=y
> CONFIG_SCHED_TRACER=y
> 
> 
> Am I missing something?

You're running a loop that does nothing except create new tasks that have no 
scheduling history, and then disappear before the scheduler can migrate them.

Try running 'openssl speed' to chew up CPU.  I promise you the scheduler will 
behave very nicely.

-- Chris

Re: Scheduler on C2D CPU and latest 2.6.27 kernel

[Zdenek Kabelac]

2008/10/21 Chris Snook <csnook@redhat.com>:
> Zdenek Kabelac wrote:
>>
>> Hi
>>
>> Recently I'm noticing bad behavior of the CPU scheduler on my T61 (2GB,
>> C2D)
>>
>> It looks like Linux concentrates all running tasks on one CPU and the
>> second cpu is sleeping.
>>
>> With recent changes to DRI - glxgears went up to 840FPS but also takes
>> 100% (with Xorg) and when I run 'while :; do true; done' loop in
>> parallel frame rate drops to 300FPS.
>>
>> But as I have C2D CPU I would expect that there should be no such
>> dramatic slowdown.
>>
>> Xosview shows that only one CPU is fully loaded.
>>
>> Here are my .config scheduler options:
>>
>> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
>> # CONFIG_GROUP_SCHED is not set
>> CONFIG_IOSCHED_NOOP=y
>> CONFIG_IOSCHED_AS=y
>> CONFIG_IOSCHED_DEADLINE=y
>> CONFIG_IOSCHED_CFQ=y
>> CONFIG_DEFAULT_IOSCHED="cfq"
>> CONFIG_SCHED_SMT=y
>> # CONFIG_SCHED_MC is not set

I've also tested   SMT=n  MC=y - with same result

>> CONFIG_SCHED_HRTICK=y
>> # CONFIG_NET_SCHED is not set
>> CONFIG_USB_EHCI_TT_NEWSCHED=y
>> CONFIG_SCHED_DEBUG=y
>> CONFIG_SCHEDSTATS=y
>> CONFIG_SCHED_TRACER=y
>>
>>
>> Am I missing something?
>
> You're running a loop that does nothing except create new tasks that have no
> scheduling history, and then disappear before the scheduler can migrate
> them.
>
> Try running 'openssl speed' to chew up CPU.  I promise you the scheduler
> will behave very nicely.

Well -  sorry you really shouldn't promise things you cannot guarantee.

It's obviously showing exactly same problem on my box.

And I should add that with  2.6.27-rc8 I do not experience this behavior.
Gears are showning 80% speed - but consumes only 25%
and bash loop or openssl speed task do not influence its speed.
(as expected with 2CPU machine)

Zdenek.

Re: Scheduler on C2D CPU and latest 2.6.27 kernel

[Chris Snook]

Zdenek Kabelac wrote:
> 2008/10/21 Chris Snook <csnook@redhat.com>:
>> Zdenek Kabelac wrote:
>>> Hi
>>>
>>> Recently I'm noticing bad behavior of the CPU scheduler on my T61 (2GB,
>>> C2D)
>>>
>>> It looks like Linux concentrates all running tasks on one CPU and the
>>> second cpu is sleeping.
>>>
>>> With recent changes to DRI - glxgears went up to 840FPS but also takes
>>> 100% (with Xorg) and when I run 'while :; do true; done' loop in
>>> parallel frame rate drops to 300FPS.
>>>
>>> But as I have C2D CPU I would expect that there should be no such
>>> dramatic slowdown.
>>>
>>> Xosview shows that only one CPU is fully loaded.
>>>
>>> Here are my .config scheduler options:
>>>
>>> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
>>> # CONFIG_GROUP_SCHED is not set
>>> CONFIG_IOSCHED_NOOP=y
>>> CONFIG_IOSCHED_AS=y
>>> CONFIG_IOSCHED_DEADLINE=y
>>> CONFIG_IOSCHED_CFQ=y
>>> CONFIG_DEFAULT_IOSCHED="cfq"
>>> CONFIG_SCHED_SMT=y
>>> # CONFIG_SCHED_MC is not set
> 
> I've also tested   SMT=n  MC=y - with same result

Expected.

>>> CONFIG_SCHED_HRTICK=y
>>> # CONFIG_NET_SCHED is not set
>>> CONFIG_USB_EHCI_TT_NEWSCHED=y
>>> CONFIG_SCHED_DEBUG=y
>>> CONFIG_SCHEDSTATS=y
>>> CONFIG_SCHED_TRACER=y
>>>
>>>
>>> Am I missing something?
>> You're running a loop that does nothing except create new tasks that have no
>> scheduling history, and then disappear before the scheduler can migrate
>> them.
>>
>> Try running 'openssl speed' to chew up CPU.  I promise you the scheduler
>> will behave very nicely.
> 
> Well -  sorry you really shouldn't promise things you cannot guarantee.

You must be new here.

> It's obviously showing exactly same problem on my box.

If you start 2 'openssl speed' tasks while glxgears is running, what happens?

> And I should add that with  2.6.27-rc8 I do not experience this behavior.

Thanks.  Can you bisect?

> Gears are showning 80% speed - but consumes only 25%
> and bash loop or openssl speed task do not influence its speed.
> (as expected with 2CPU machine)

That's definitely what it should be doing with openssl speed.  The bash loop 
isn't something we should go out of our way to optimize for, since real world 
apps don't behave that way, but if there's a free fix, we might as well do it.

-- Chris

Re: Scheduler on C2D CPU and latest 2.6.27 kernel

[Zdenek Kabelac]

2008/10/21 Chris Snook <csnook@redhat.com>:
> Zdenek Kabelac wrote:
>>
>> 2008/10/21 Chris Snook <csnook@redhat.com>:
>>>
>>> Zdenek Kabelac wrote:
>>>>
>>>> Hi
>>>>
>>>> Recently I'm noticing bad behavior of the CPU scheduler on my T61 (2GB,
>>>> C2D)
>>>>
>>>> It looks like Linux concentrates all running tasks on one CPU and the
>>>> second cpu is sleeping.
>>>>
>>>> With recent changes to DRI - glxgears went up to 840FPS but also takes
>>>> 100% (with Xorg) and when I run 'while :; do true; done' loop in
>>>> parallel frame rate drops to 300FPS.
>>>>
>>>> But as I have C2D CPU I would expect that there should be no such
>>>> dramatic slowdown.
>>>>
>>>> Xosview shows that only one CPU is fully loaded.
>>>>
>>>> Here are my .config scheduler options:
>>>>
>>>> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
>>>> # CONFIG_GROUP_SCHED is not set
>>>> CONFIG_IOSCHED_NOOP=y
>>>> CONFIG_IOSCHED_AS=y
>>>> CONFIG_IOSCHED_DEADLINE=y
>>>> CONFIG_IOSCHED_CFQ=y
>>>> CONFIG_DEFAULT_IOSCHED="cfq"
>>>> CONFIG_SCHED_SMT=y
>>>> # CONFIG_SCHED_MC is not set
>>
>> I've also tested   SMT=n  MC=y - with same result
>
> Expected.
>
>>>> CONFIG_SCHED_HRTICK=y
>>>> # CONFIG_NET_SCHED is not set
>>>> CONFIG_USB_EHCI_TT_NEWSCHED=y
>>>> CONFIG_SCHED_DEBUG=y
>>>> CONFIG_SCHEDSTATS=y
>>>> CONFIG_SCHED_TRACER=y
>>>>
>>>>
>>>> Am I missing something?
>>>
>>> You're running a loop that does nothing except create new tasks that have
>>> no
>>> scheduling history, and then disappear before the scheduler can migrate
>>> them.
>>>
>>> Try running 'openssl speed' to chew up CPU.  I promise you the scheduler
>>> will behave very nicely.
>>
>> Well -  sorry you really shouldn't promise things you cannot guarantee.
>
> You must be new here.

hmm, I don't think so :)

>
>> It's obviously showing exactly same problem on my box.
>
> If you start 2 'openssl speed' tasks while glxgears is running, what
> happens?
>
>> And I should add that with  2.6.27-rc8 I do not experience this behavior.
>
> Thanks.  Can you bisect?

Maybe the author of this regression will 'confess' himself and will
safe my time?

I can see some major merge  on Oct 20 for kernel/sched.c

>
>> Gears are showning 80% speed - but consumes only 25%
>> and bash loop or openssl speed task do not influence its speed.
>> (as expected with 2CPU machine)
>
> That's definitely what it should be doing with openssl speed.  The bash loop
> isn't something we should go out of our way to optimize for, since real
> world apps don't behave that way, but if there's a free fix, we might as
> well do it.

You really should check first what the loop I've posted actually does :)
(hint -  /bin/true  !=  true)

Zdenek