CFS not suitable for desktop computers

CFS not suitable for desktop computers

[Nico Schümann]

Dear Linux developers,

I have been using Linux for some years now and for me, the best thing 
about 2.6 was that Linux ran desktop applications just smoothly. I was 
able to compile in the background, while all applications under X11 were 
just usable as if the machine was in idle mode. This was due to the 
priority of gcc being set to 30, for instance.

Then, somewhere around 2.6.19 or 2.6.21, I do not remember exactly, the 
CFS was introduced, which removed all those "latency-based" scheduling 
policies. Now that I use 2.6.29 (I did not write earlier because I 
though it was a regression issue) I have to say: Linux is not as 
perfectly usable as before. End users do not want to experiment with 
nice levels and stuff, they just want that the system stays responsible 
even if there is a cpu-consuming process in the _background_. For me, 
this had been the greatest benefit from using Linux .

Now what can we do, so that foreground applications are smoothly usable 
during hard cpu load? Is there any way to restore the old behaviour that 
cpu-consuming processes get a lower priority? It had always worked until 
this new scheduler was introduced.

Regards,
Nico Schümann

Re: CFS not suitable for desktop computers

[Ray Lee]

On Sun, May 3, 2009 at 12:26 PM, Nico Schümann <spam@nico22.de> wrote:
> Dear Linux developers,
>
> I have been using Linux for some years now and for me, the best thing about
> 2.6 was that Linux ran desktop applications just smoothly. I was able to
> compile in the background, while all applications under X11 were just usable
> as if the machine was in idle mode. This was due to the priority of gcc
> being set to 30, for instance.
>
> Then, somewhere around 2.6.19 or 2.6.21, I do not remember exactly, the CFS
> was introduced, which removed all those "latency-based" scheduling policies.
> Now that I use 2.6.29 (I did not write earlier because I though it was a
> regression issue) I have to say: Linux is not as perfectly usable as before.
> End users do not want to experiment with nice levels and stuff, they just
> want that the system stays responsible even if there is a cpu-consuming
> process in the _background_. For me, this had been the greatest benefit from
> using Linux .
>
> Now what can we do, so that foreground applications are smoothly usable
> during hard cpu load? Is there any way to restore the old behaviour that
> cpu-consuming processes get a lower priority? It had always worked until
> this new scheduler was introduced.

My experience is exactly the opposite. Before CFS, my system was
completely unusable unless I manually controlled all nice levels in
the system for the big jobs. With CFS, everything just works, no
twiddling of nice levels necessary.

To get a useful response from the developers (who really do care about
latency issues), you should run a workload that causes unacceptable
behavior, and follow the suggestions below (taken from an old email
from Ingo Molnar):

----------
could you run this script while such a slowdown is really prominent:

 http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh

and send me the output it generates? The output is the most useful if
you do this on a kernel that has CONFIG_SCHED_DEBUG=y and
CONFIG_SCHEDSTATS=y enabled.
----------

The problem may not be related to the scheduler at all, you realize,
but rather some other regression in another subsystem, such as the IO
scheduler or X or a driver, but hopefully the above script will be a
good starting point for those with the big brains to figure out what's
wrong.

Re: CFS not suitable for desktop computers

[Nico Schümann]

[-- Attachment #1: Type: text/plain, Size: 943 bytes --]

Thank you Ray Lee and Mike Galbraith for your responses, I ran the 
script and attached its gathered information.

Mike Galbraith wrote:
> How hard is hard?  Can you describe the loads you're having trouble
> with, and the hardware you're running them on?
>
>   
I could reproduce "hard" load by just compiling the linux kernel, make 
-j3 while reading mails with Thunderbird, which is not that hard 
foreground load. Thunderbird starts reacting really slowly while compiling.

My system has a 1,3 GHz AMD Athlon CPU (32 bits) and 1 GB of RAM. Now 
you will say: That is not very much. Of course it is not, but with the 
old scheduler, the system felt way faster, so it seemed to be enough for 
compiling and reading mails.
I hope you can find useful information in the attached log, I enabled 
SCHED_DEBUG and SCHEDSTATS, if you need any more information, just ask 
me, I will try to answer.

And thank you all for your help,

Nico Schümann

[-- Attachment #2: cfs-debug-info-2009.05.04-17.04.35.gz --]
[-- Type: application/x-gzip, Size: 63145 bytes --]

Re: CFS not suitable for desktop computers

[Mike Galbraith]

On Mon, 2009-05-04 at 17:16 +0200, Nico Schümann wrote:
> Thank you Ray Lee and Mike Galbraith for your responses, I ran the 
> script and attached its gathered information.
> 
> Mike Galbraith wrote:
> > How hard is hard?  Can you describe the loads you're having trouble
> > with, and the hardware you're running them on?
> >
> >   
> I could reproduce "hard" load by just compiling the linux kernel, make 
> -j3 while reading mails with Thunderbird, which is not that hard 
> foreground load. Thunderbird starts reacting really slowly while compiling.
> 
> My system has a 1,3 GHz AMD Athlon CPU (32 bits) and 1 GB of RAM. Now 
> you will say: That is not very much. Of course it is not, but with the 
> old scheduler, the system felt way faster, so it seemed to be enough for 
> compiling and reading mails.

Hm.  The load isn't extreme, but it appears to me that between X and
Thunderbird, CPU demand is high enough that you WILL feel the slowdown
when you toss in three competing CPU hogs plus other system activity all
on one core.

> I hope you can find useful information in the attached log, I enabled 
> SCHED_DEBUG and SCHEDSTATS, if you need any more information, just ask 
> me, I will try to answer.

I'll look closer tomorrow (ill).  For now, how much CPU does
X/Thunderbird consume without the kbuild?

(I know this isn't what you want to hear, but SCHED_IDLE is a major case
of happiness for heavy lifting background loads, especially so on UP.)

	-Mike

Re: CFS not suitable for desktop computers

[Chris Friesen]

Nico Schümann wrote:
> Thank you Ray Lee and Mike Galbraith for your responses, I ran the 
> script and attached its gathered information.
> 
> Mike Galbraith wrote:
>> How hard is hard?  Can you describe the loads you're having trouble
>> with, and the hardware you're running them on?
>>
>>   
> I could reproduce "hard" load by just compiling the linux kernel, make 
> -j3 while reading mails with Thunderbird, which is not that hard 
> foreground load. Thunderbird starts reacting really slowly while compiling.

I wonder if this could be related to I/O rather than the task scheduler 
proper.

Chris

Re: CFS not suitable for desktop computers

[Mike Galbraith]

On Mon, 2009-05-04 at 15:01 -0600, Chris Friesen wrote:
> Nico Schümann wrote:
> > Thank you Ray Lee and Mike Galbraith for your responses, I ran the 
> > script and attached its gathered information.
> > 
> > Mike Galbraith wrote:
> >> How hard is hard?  Can you describe the loads you're having trouble
> >> with, and the hardware you're running them on?
> >>
> >>   
> > I could reproduce "hard" load by just compiling the linux kernel, make 
> > -j3 while reading mails with Thunderbird, which is not that hard 
> > foreground load. Thunderbird starts reacting really slowly while compiling.
> 
> I wonder if this could be related to I/O rather than the task scheduler 
> proper.

That's something I was wondering as well, particularly when I noticed
he's using data=journal and anticipatory io-scheduler.

Doing some testing with different journal modes and io-schedulers
recently, I found loads where anticipatory didn't do very well compared
to cfq, and journal mode can make a tremendous difference ala..

time ./postmark < config

          ordered     writeback      guarded      journal
real    11m18.752s    3m5.160s     3m1.367s    15m18.610s
user    0m5.672s      0m6.220s     0m5.976s    0m6.856s
sys     0m26.094s     0m20.965s    0m21.673s   0m20.969s

real    9m33.922s     2m53.984s    3m6.284s    15m12.969s
user    0m5.700s      0m5.976s     0m6.356s    0m6.980s
sys     0m26.682s     0m21.413s    0m20.717s   0m19.641s

real    10m8.766s     3m0.382s     2m58.714s   15m33.400s
user    0m5.608s      0m6.080s     0m6.232s    0m6.776s
sys     0m27.130s     0m21.041s    0m21.709s   0m19.489s

	-Mike

Re: CFS not suitable for desktop computers

[Mike Galbraith]

On Tue, 2009-05-05 at 07:42 +0200, Mike Galbraith wrote:
> On Mon, 2009-05-04 at 15:01 -0600, Chris Friesen wrote:
> > Nico Schümann wrote:
> > > Thank you Ray Lee and Mike Galbraith for your responses, I ran the 
> > > script and attached its gathered information.
> > > 
> > > Mike Galbraith wrote:
> > >> How hard is hard?  Can you describe the loads you're having trouble
> > >> with, and the hardware you're running them on?
> > >>
> > >>   
> > > I could reproduce "hard" load by just compiling the linux kernel, make 
> > > -j3 while reading mails with Thunderbird, which is not that hard 
> > > foreground load. Thunderbird starts reacting really slowly while compiling.
> > 
> > I wonder if this could be related to I/O rather than the task scheduler 
> > proper.
> 
> That's something I was wondering as well, particularly when I noticed
> he's using data=journal and anticipatory io-scheduler.

Well, I offlined 3 of 4 cores, set data=journal, and throttled down to
1.6GHz to load test a bit, and it didn't make any difference with
_Evolution_ and a make -j3 running.  Evolution remained responsive.

I can feel the load though. X here is unaccelerated (damn nvidia), so
despite heftier core (Q6600), CPU usage while browsing mail is pretty
similar to Nico's data, often heavier.  Behavior I see here isn't bad at
all.. unless I move a window that is, X then becomes a major oinker, so
I feel every bit of load.  That's unpleasant, but expected.

I have an old P4 lying about (has X candy).  I'll update it and try some
UP testing.  Can't test Thunderbird though.

	-Mike

Re: CFS not suitable for desktop computers

[Peter Zijlstra]

On Mon, 2009-05-04 at 17:16 +0200, Nico Schümann wrote: 
> Thank you Ray Lee and Mike Galbraith for your responses, I ran the 
> script and attached its gathered information.
> 
> Mike Galbraith wrote:
> > How hard is hard?  Can you describe the loads you're having trouble
> > with, and the hardware you're running them on?
> >
> >   
> I could reproduce "hard" load by just compiling the linux kernel, make 
> -j3 while reading mails with Thunderbird, which is not that hard 
> foreground load. Thunderbird starts reacting really slowly while compiling.

Right, such a load will leave your thunderbird about 25% of the cpu. I
think its fair to expect it to be slower, but it should still be
workable.

I used to test CFS on my 1.2GHz laptop with 512M and a make -j5. That
would result in a slow but steady system. With steady I mean the latency
was pretty constant.

The O(1) scheduler would utterly mess this up, it would be fast, until
the desktop bloat took enough cpu time and then it would starve a while,
etc..

The thing is, we really cannot go about guessing, its too easy to guess
wrong. If you know its a background task, nice it.

alias make='nice make'

and you're done.