Question about fair schedulers

Question about fair schedulers

[Alberto Gonzalez]

Hi,

First I'd like to say I'm not a programmer or even a geek, just a normal user, 
so my question might be very basic or even stupid. If so, please excuse me.

I've been reading about CFS and SD schedulers here on the list and my basic 
understanding is that they try to improve interactivity by being completely 
fair, i.e., giving the same amount of CPU time to each task running at a 
given time. There is something I don't get about this, so today I tried a 
little test with the -ck kernel comparing it to mainline.

Let's say I have a HD video that uses ~70% CPU. Let's say I want to watch it 
while I encode my music to vorbis (or rip a DVD). This is the only reasonable 
scenario I can imagine on a normal desktop, since most desktops have the CPU 
idle or under 10% usage during 95% of the time and a CPU scheduler makes no 
difference (maybe people on this list compile a few kernels every day, but 
that's not what most normal users like me do).

Ok, so what will a fair scheduler do in this case? It is my understanding that 
it would give 50% CPU to each task, resulting in the video dropping frames. 
Is this correct?

Now, the _ideal_ solution for this situation would be to give ~70% to the 
video and the rest (~30%) to the encoder. But this goes against fairness, 
doesn't it? Yet most reports I've read about these two fair schedulers say 
that videos play smoother under load. What am I missing?

If I ask this question is because today I tested it and found that using 
the -ck kernel (with SD scheduler) the video would drop frames, while with 
the mainline kernel it played fine.

My conclusion is that SD behaves as expected: it's more fair. But for a 
desktop, shouldn't an "intelligently unfair" scheduler be better?

Thanks for any insights.

P.S: As a second thought, a fair scheduler could behave really good in other 
scenarios, like a server running a busy forum on apache+mysql+php. Besides, 
this is a more real world scenario (and easier to benchmark). Why aren't 
people testing these schedulers under this kind of load?

Re: Question about fair schedulers

[Kyle Moffett]

On Jun 22, 2007, at 18:07:15, Alberto Gonzalez wrote:
> Let's say I have a HD video that uses ~70% CPU. Let's say I want to  
> watch it while I encode my music to vorbis (or rip a DVD). This is  
> the only reasonable scenario I can imagine on a normal desktop,  
> since most desktops have the CPU idle or under 10% usage during 95%  
> of the time and a CPU scheduler makes no difference (maybe people  
> on this list compile a few kernels every day, but that's not what  
> most normal users like me do).

> Ok, so what will a fair scheduler do in this case? It is my  
> understanding that it would give 50% CPU to each task, resulting in  
> the video dropping frames. Is this correct?

Yes, that's correct.

What this *actually* means is that you want the media player to have  
higher priority than the DVD ripping program.  Ergo you should run  
"nice +20 my_dvd_burner" or "nice +20 my_vorbis_encoder" under CFS or  
other fair schedulers.  (Although you might not need to nice the dvd  
burner) The key difference is this:

With the older scheduler your DVD ripping process would have had  
buffer underrun problems; even though it doesn't require much CPU at  
all (and can slow down burning if it doesn't get enough) it wouldn't  
get a chance to run at appropriate times to keep the buffer full.  If  
you were encoding vorbis files you would *definitely* want to renice  
it as it will consume all available CPU.

With CFS, your HD video player would get the 70% of the CPU that it  
needs (exactly), and the rest would be allocated to the DVD ripper  
and other processes.  In addition, the DVD ripper gets certain  
latency guarantees even though it's *LOWER* priority than the media  
player (which also gets latency guarantees).

> Now, the _ideal_ solution for this situation would be to give ~70%  
> to the video and the rest (~30%) to the encoder. But this goes  
> against fairness, doesn't it? Yet most reports I've read about  
> these two fair schedulers say that videos play smoother under load.  
> What am I missing?

Well, typically your video player *doesn't* chew up all of the CPU,  
so the fairness helps.  Here's another scenario for you:  You are  
running un-niced HD video player and recoding processes side-by-side,  
so that the HD video player only gets 50% of the CPU.  Under older  
schedulers the video player would not get uniformly allocated CPU, it  
would get it in fits and spurts, causing more _visibly_dropped_  
frames (IE: render 30 frames, drop 8, render 15, drop 2, etc).  Under  
the new CFS scheduler it will get evenly allocated CPU and so while  
you *will* get dropped frames, they will be less visible (IE: Render  
7 frames, drop 1, render 7 frames, drop 1, render 7 frames, drop 1).

> If I ask this question is because today I tested it and found that  
> using the -ck kernel (with SD scheduler) the video would drop  
> frames, while with the mainline kernel it played fine.

Basically you are telling the kernel that your video player is no  
more important than your re-encoder process which, judging by your  
email, is completely untrue.  If you really want it to treat the  
video player as specially important (or alternatively treat the  
reencoder process and specially unimportant) then you must tell it so  
with "nice" or "renice".

> My conclusion is that SD behaves as expected: it's more fair. But  
> for a desktop, shouldn't an "intelligently unfair" scheduler be  
> better?

Well, it is "intelligently unfair", but you the user have to tell it  
what is more important for *YOU*.  If I _really_ have to get some  
music recoded quickly then I may be willing to deal with some dropped  
frames in order to let that happen appropriately.

> P.S: As a second thought, a fair scheduler could behave really good  
> in other scenarios, like a server running a busy forum on apache 
> +mysql+php. Besides, this is a more real world scenario (and easier  
> to benchmark). Why aren't people testing these schedulers under  
> this kind of load?

That kind of load is boring precisely because it doesn't care about  
interactivity.  CFS/SD aren't a whole lot different from mainline- 
without-interactivity in that respect, precisely because the latency  
of the network is between ten and a hundred times more than the  
latency of the scheduler.  The only time it really matters is with  
desktops where users care about smoothness.

Cheers,
Kyle Moffett

Re: Question about fair schedulers

[Alberto Gonzalez]

On Saturday 23 June 2007, Kyle Moffett wrote:
> On Jun 22, 2007, at 18:07:15, Alberto Gonzalez wrote:
> > Ok, so what will a fair scheduler do in this case? It is my
> > understanding that it would give 50% CPU to each task, resulting in
> > the video dropping frames. Is this correct?
>
> Yes, that's correct.
>
> What this *actually* means is that you want the media player to have
> higher priority than the DVD ripping program.  Ergo you should run
> "nice +20 my_dvd_burner" or "nice +20 my_vorbis_encoder" under CFS or
> other fair schedulers.

Ok, that makes sense. The problem is that desktop users don't know about such 
things, so the ideal situation would be that the scheduler knows about it and 
does it for you.

> Well, typically your video player *doesn't* chew up all of the CPU,
> so the fairness helps.  Here's another scenario for you:  You are
> running un-niced HD video player and recoding processes side-by-side,
> so that the HD video player only gets 50% of the CPU.  Under older
> schedulers the video player would not get uniformly allocated CPU, it
> would get it in fits and spurts, causing more _visibly_dropped_
> frames (IE: render 30 frames, drop 8, render 15, drop 2, etc).  Under
> the new CFS scheduler it will get evenly allocated CPU and so while
> you *will* get dropped frames, they will be less visible (IE: Render
> 7 frames, drop 1, render 7 frames, drop 1, render 7 frames, drop 1).

Yes, I see your point. In a scenario of dropping frames it seems that CFS does 
a better job. It's just that an ideal scheduler shouldn't drop frames in this 
case (it should give 70% to the video even without nicing the encoder).

> > If I ask this question is because today I tested it and found that
> > using the -ck kernel (with SD scheduler) the video would drop
> > frames, while with the mainline kernel it played fine.
>
> Basically you are telling the kernel that your video player is no
> more important than your re-encoder process which, judging by your
> email, is completely untrue.  If you really want it to treat the
> video player as specially important (or alternatively treat the
> reencoder process and specially unimportant) then you must tell it so
> with "nice" or "renice".

That's my whole point. On a desktop, some tasks have by nature a higher 
priority than others. A fair scheduler treats them all the same (hence, fair) 
way. So a user must nice/renice processes for them to get the correct CPU 
time (something you can't expect normal desktop users to do).

> > My conclusion is that SD behaves as expected: it's more fair. But
> > for a desktop, shouldn't an "intelligently unfair" scheduler be
> > better?
>
> Well, it is "intelligently unfair", but you the user have to tell it
> what is more important for *YOU*.  If I _really_ have to get some
> music recoded quickly then I may be willing to deal with some dropped
> frames in order to let that happen appropriately.

Well, my point is that encoding music or video can use 100% CPU if available, 
but if you give it just 10% the job will also be completed successfully 
(though slower). OTOH, playing audio or video won't use 100% CPU (in most 
cases), but whatever CPU they need, they _really_ need it. It makes no sense 
to watch a video or listen to music with constant skips. So in 99.9% of the 
cases a user would want to watch a video smoothly, not dropping frames so 
that the encoding finishes faster (if he wants the encoding to be as fast as 
possible he shouldn't watch a video at the same time, I guess).

> Cheers,
> Kyle Moffett

Thanks for your answers,
Alberto.

Re: Question about fair schedulers

[Kyle Moffett]

On Jun 23, 2007, at 03:46:43, Alberto Gonzalez wrote:
> On Saturday 23 June 2007, Kyle Moffett wrote:
>> On Jun 22, 2007, at 18:07:15, Alberto Gonzalez wrote:
>>> Ok, so what will a fair scheduler do in this case? It is my  
>>> understanding that it would give 50% CPU to each task, resulting  
>>> in the video dropping frames. Is this correct?
>>
>> Yes, that's correct.
>>
>> What this *actually* means is that you want the media player to  
>> have higher priority than the DVD ripping program.  Ergo you  
>> should run "nice +20 my_dvd_burner" or "nice +20  
>> my_vorbis_encoder" under CFS or other fair schedulers.
>
> Ok, that makes sense. The problem is that desktop users don't know  
> about such things, so the ideal situation would be that the  
> scheduler knows about it and does it for you.

No, it is *categorically* *impossible* for the scheduler to get that  
decision right.  For example, if my job is as a programmer and I'm  
waiting for a build during lunchbreak then I may not care that the  
process in the background makes my DVD-player drop a few frames, as  
long as it gets done 10% quicker.  That kind of decision is *POLICY*,  
and it does not belong in the kernel.  If you want the kernel to  
treat one job or the other as more important then you must *TELL* it  
that, end of story.

>> Under the new CFS scheduler it will get evenly allocated CPU and  
>> so while you *will* get dropped frames, they will be less visible  
>> (IE: Render 7 frames, drop 1, render 7 frames, drop 1, render 7  
>> frames, drop 1).
>
> Yes, I see your point. In a scenario of dropping frames it seems  
> that CFS does a better job. It's just that an ideal scheduler  
> shouldn't drop frames in this case (it should give 70% to the video  
> even without nicing the encoder).

It can't do that.  The with-CFS kernel just sees 2 CPU-heavy  
processes and guesses that it should give them equal CPU.  "stock"  
kernels have an algorithm designed to promote some tasks for  
"interactivity", but in practice it also tended to cause other  
processes to be denied CPU for arbitrarily long periods of time,  
hence why CFS is an improvement.  Under the old scheduler even if you  
had 2 DVD player processes each chewing 45% CPU, you could still have  
dropped frames because for a second or two one would be more  
"interactive" than the other, and vice versa.  Under CFS/SD, they are  
both classified equally and so get equal CPU allocation *AND* latency.

>> Basically you are telling the kernel that your video player is no  
>> more important than your re-encoder process which, judging by your  
>> email, is completely untrue.  If you really want it to treat the  
>> video player as specially important (or alternatively treat the  
>> reencoder process and specially unimportant) then you must tell it  
>> so with "nice" or "renice".
>
> That's my whole point. On a desktop, some tasks have by nature a  
> higher priority than others. A fair scheduler treats them all the  
> same (hence, fair) way. So a user must nice/renice processes for  
> them to get the correct CPU time (something you can't expect normal  
> desktop users to do).

I don't see any reason someone couldn't write a simple little GUI  
program to enumerate the user-owned X processes (somewhat like the  
Windows Task-Manager but less complicated) and allow them to change  
priorities.  Alternatively your desktop environment could set up a  
little privileged wrapper which appropriately executes the HD video  
player.  One of the primary rules of kernel development is that you  
cannot put policy in the kernel, and a statement of the form  
"PROCESS1 is more important than PROCESS2" is pure policy and must be  
done from userspace.  We even give appropriate enforcement mechanisms  
to userspace to take such action (nice levels).

Cheers,
Kyle Moffett

Re: Question about fair schedulers

[Alberto Gonzalez]

El Saturday 23 June 2007 18:35:18 Kyle Moffett escribió:
> If you want the kernel to
> treat one job or the other as more important then you must *TELL* it
> that, end of story.

Yes, that makes sense now that it's been explained to me conveniently. As long 
as a normal user is not left alone with such task (as it won't happen), I'm 
more than happy with the solution.

> > Yes, I see your point. In a scenario of dropping frames it seems
> > that CFS does a better job. It's just that an ideal scheduler
> > shouldn't drop frames in this case (it should give 70% to the video
> > even without nicing the encoder).
>
> It can't do that.  The with-CFS kernel just sees 2 CPU-heavy
> processes and guesses that it should give them equal CPU.  "stock"
> kernels have an algorithm designed to promote some tasks for
> "interactivity", but in practice it also tended to cause other
> processes to be denied CPU for arbitrarily long periods of time,
> hence why CFS is an improvement.  Under the old scheduler even if you
> had 2 DVD player processes each chewing 45% CPU, you could still have
> dropped frames because for a second or two one would be more
> "interactive" than the other, and vice versa.  Under CFS/SD, they are
> both classified equally and so get equal CPU allocation *AND* latency.

This adds even more sense to all the explanations I've heard up to now. 
Thanks :)

> I don't see any reason someone couldn't write a simple little GUI
> program to enumerate the user-owned X processes (somewhat like the
> Windows Task-Manager but less complicated) and allow them to change
> priorities.  Alternatively your desktop environment could set up a
> little privileged wrapper which appropriately executes the HD video
> player.  One of the primary rules of kernel development is that you
> cannot put policy in the kernel, and a statement of the form
> "PROCESS1 is more important than PROCESS2" is pure policy and must be
> done from userspace.  We even give appropriate enforcement mechanisms
> to userspace to take such action (nice levels).

Yes, an app to change priorities would be very nice, though I'd be happy with 
just sensible defaults. Probably once CFS or SD go mainline more application 
developers will make their apps run at the appropriate nice level (or else 
distributions will provide this when packaging the apps), so end users won't 
have any problems.

> Cheers,
> Kyle Moffett

Thank you,
Alberto.

Re: Question about fair schedulers

[Jesper Juhl]

On 23/06/07, Alberto Gonzalez <info@gnebu.es> wrote:
> El Saturday 23 June 2007 18:35:18 Kyle Moffett escribió:
[snip]
> > "PROCESS1 is more important than PROCESS2" is pure policy and must be
> > done from userspace.  We even give appropriate enforcement mechanisms
> > to userspace to take such action (nice levels).
>
> Yes, an app to change priorities would be very nice,

You already have such an app. It is called 'renice' (and 'nice'). Most
people can work out how to use those. :-)

You also have graphical apps to do the job. For example, in KDE, try
starting the "KDE System Guard" application (usually bound to the
CTRL+ESC hotkey) and you'll see a nice list of all running processes.
If you right-click a process you'll get a drop-down box with the
bottom entry being "Renice Process...", simply click that and you can
adjust the priority of that process.  You can also easily make it so
that whenever you start a given application it gets niced to a
specific priority; simply (again assuming KDE) right-click the icon
you use to launch the application and edit it, select the
"Application" tab and in the "Command: " field prefix the command used
to launch the application with 'nice' and the priority you want, save
your changes and the next time you launch that app it'll get the
priority you wish.
Gnome and most other environments also have similar capabilities.

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

RE: Question about fair schedulers

[David Schwartz]

Alberto Gonzalez wrote:

> On Saturday 23 June 2007, Kyle Moffett wrote:
> > On Jun 22, 2007, at 18:07:15, Alberto Gonzalez wrote:

> > What this *actually* means is that you want the media player to have
> > higher priority than the DVD ripping program.  Ergo you should run
> > "nice +20 my_dvd_burner" or "nice +20 my_vorbis_encoder" under CFS or
> > other fair schedulers.
>
> Ok, that makes sense. The problem is that desktop users don't
> know about such
> things, so the ideal situation would be that the scheduler knows
> about it and
> does it for you.

I disagree. How could the scheduler know that you care more about the media
player than the DVD ripping program? Maybe you're just watching unimportant
stuff to waste time while anxiously waiting for the DVD rip to finish.

Why should the scheduler make assumptions about what's important to the
user? If the user has priorities, they should tell the scheduler.

DS

Re: Question about fair schedulers

[Helge Hafting]

Alberto Gonzalez wrote:
> On Saturday 23 June 2007, Kyle Moffett wrote:
>   
>> On Jun 22, 2007, at 18:07:15, Alberto Gonzalez wrote:
>>     
>>> Ok, so what will a fair scheduler do in this case? It is my
>>> understanding that it would give 50% CPU to each task, resulting in
>>> the video dropping frames. Is this correct?
>>>       
>> Yes, that's correct.
>>
>> What this *actually* means is that you want the media player to have
>> higher priority than the DVD ripping program.  Ergo you should run
>> "nice +20 my_dvd_burner" or "nice +20 my_vorbis_encoder" under CFS or
>> other fair schedulers.
>>     
>
> Ok, that makes sense. The problem is that desktop users don't know about such 
> things, so the ideal situation would be that the scheduler knows about it and 
> does it for you.
>   
Well, the scheduler can't really know that.  But the people who
write the dvd burning/ripping/encoding/viewing software
know what is needed, they can write their programs so they set useful 
priorities.

Helge Hafting

Re: Question about fair schedulers

[Alberto Gonzalez]

On Saturday 23 June 2007, Kyle Moffett wrote:
> On Jun 22, 2007, at 18:07:15, Alberto Gonzalez wrote:
> > P.S: As a second thought, a fair scheduler could behave really good
> > in other scenarios, like a server running a busy forum on apache
> > +mysql+php. Besides, this is a more real world scenario (and easier
> > to benchmark). Why aren't people testing these schedulers under
> > this kind of load?
>
> That kind of load is boring precisely because it doesn't care about
> interactivity.  CFS/SD aren't a whole lot different from mainline-
> without-interactivity in that respect, precisely because the latency
> of the network is between ten and a hundred times more than the
> latency of the scheduler.  The only time it really matters is with
> desktops where users care about smoothness.

Well, I've just seen some benchmarks of this kind with CFS and it does make a 
difference. The scalability problem with MySQL seems to get solved with this 
scheduler. However, the peak throughput is quite lower than with the same 
kernel/glibc version and mainline scheduler.

http://jeffr-tech.livejournal.com/

Anyway, this kind of testing seems to be useful. Linux is too widely deployed 
in servers to ignore the behavior of a CPU scheduler in that scenario.

Cheers,
Alberto.

Re: Question about fair schedulers

[Paolo Ornati]

On Sat, 23 Jun 2007 00:07:15 +0200
Alberto Gonzalez <info@gnebu.es> wrote:

> My conclusion is that SD behaves as expected: it's more fair. But for a 
> desktop, shouldn't an "intelligently unfair" scheduler be better?

"intelligently unfair" is what the current scheduler is (because of
interactivity estimator).

When it works (say 90% of the time) the desktop feels really good...
but when it doesn't it can be a disaster.


Look this for example:

http://groups.google.com/group/fa.linux.kernel/browse_thread/thread/6aa5c93c379ae9e1/98ab31c0e6fed2ee?&hl=en#98ab31c0e6fed2ee

-- 
	Paolo Ornati
	Linux 2.6.22-rc5-g0864a4e2 on x86_64

Re: Question about fair schedulers

[Alberto Gonzalez]

Thanks for your thoughts.

On Saturday 23 June 2007, Paolo Ornati wrote:
> On Sat, 23 Jun 2007 00:07:15 +0200
>
> Alberto Gonzalez wrote:
> > My conclusion is that SD behaves as expected: it's more fair. But for a
> > desktop, shouldn't an "intelligently unfair" scheduler be better?
>
> "intelligently unfair" is what the current scheduler is (because of
> interactivity estimator).
>
> When it works (say 90% of the time) the desktop feels really good...
> but when it doesn't it can be a disaster.

I see. So you mean that in 90% of the cases the mainline scheduler behaves 
better than fair schedulers, but when its "logic" fails it behaves much worse 
(the other 10% cases)? In my very simple test scenario the mainline scheduler 
did behave much better. Maybe the problem comes with very complex scenarios 
like the ones I've seen when testing these 2 fair schedulers (something like 
compiling a kernel while you open 5 instances of glxgears, write an email, 
play music in Amarok and watch 2 HD videos all at the same time). The 
question would then be if these kind of situations are likely to happen in 
real world, or even if it doesn't make more sense to try to improve the logic 
of the mainline scheduler so that those 10% cases are handled better instead 
of writing a new one that would behave worse in 90% of the cases and better 
in the other 10%.

	Alberto.

Re: Question about fair schedulers

[Willy Tarreau]

On Sat, Jun 23, 2007 at 10:01:02AM +0200, Alberto Gonzalez wrote:
> Thanks for your thoughts.
> 
> On Saturday 23 June 2007, Paolo Ornati wrote:
> > On Sat, 23 Jun 2007 00:07:15 +0200
> >
> > Alberto Gonzalez wrote:
> > > My conclusion is that SD behaves as expected: it's more fair. But for a
> > > desktop, shouldn't an "intelligently unfair" scheduler be better?
> >
> > "intelligently unfair" is what the current scheduler is (because of
> > interactivity estimator).
> >
> > When it works (say 90% of the time) the desktop feels really good...
> > but when it doesn't it can be a disaster.
> 
> I see. So you mean that in 90% of the cases the mainline scheduler behaves 
> better than fair schedulers, but when its "logic" fails it behaves much worse 
> (the other 10% cases)? In my very simple test scenario the mainline scheduler 
> did behave much better. Maybe the problem comes with very complex scenarios 
> like the ones I've seen when testing these 2 fair schedulers (something like 
> compiling a kernel while you open 5 instances of glxgears, write an email, 
> play music in Amarok and watch 2 HD videos all at the same time). The 
> question would then be if these kind of situations are likely to happen in 
> real world, or even if it doesn't make more sense to try to improve the logic 
> of the mainline scheduler so that those 10% cases are handled better instead 
> of writing a new one that would behave worse in 90% of the cases and better 
> in the other 10%.

No, in fact, the mainline scheduler is good only for the very particular
cases you propose here. But on some fairly trivial situations (such as two
processes consuming very close to 50% CPU), it can be a disaster. I've had
proxies under medium load slow down sshd so much that it was impossible to
log in. Most of the worst situations have been fixed since the early 2.6
versions (2.6.11 was still horrible), but it's the general design which
makes such cases hard to find and to fix.

Your situation is clearly something standard where only the user can decide
which application should get more CPU. You start two CPU hogs, and your mind
tells you that you would prefer the video not to skip and the encoding to
finish later. For other people, it might be the reverse because the video
will be there just for monitoring purposes. The nice command was invented
exactly for that, and people have been happily using it for the last 30
years. I don't see why it should become hard to use ! Many CPU-sensitive
applications already provide the ability to change their nice value
themselves using a command line parameter.

And if it is still too hard to assign nice values from within your
environment, then perhaps it's the environment which does not suit your
needs. Even Windows users can choose to assign more priorities to
foreground/background tasks, so it is something universally accepted,
and I would be really surprized that you would not have the ability
to proceed the same way.

> 	Alberto.

Regards,
Willy

Re: Question about fair schedulers

[Alberto Gonzalez]

On Saturday 23 June 2007, Willy Tarreau wrote:
> On Sat, Jun 23, 2007 at 10:01:02AM +0200, Alberto Gonzalez wrote:
> > I see. So you mean that in 90% of the cases the mainline scheduler
> > behaves better than fair schedulers, but when its "logic" fails it
> > behaves much worse (the other 10% cases)? In my very simple test scenario
> > the mainline scheduler did behave much better. Maybe the problem comes
> > with very complex scenarios like the ones I've seen when testing these 2
> > fair schedulers (something like compiling a kernel while you open 5
> > instances of glxgears, write an email, play music in Amarok and watch 2
> > HD videos all at the same time). The question would then be if these kind
> > of situations are likely to happen in real world, or even if it doesn't
> > make more sense to try to improve the logic of the mainline scheduler so
> > that those 10% cases are handled better instead of writing a new one that
> > would behave worse in 90% of the cases and better in the other 10%.
>
> No, in fact, the mainline scheduler is good only for the very particular
> cases you propose here. But on some fairly trivial situations (such as two
> processes consuming very close to 50% CPU), it can be a disaster. I've had
> proxies under medium load slow down sshd so much that it was impossible to
> log in. Most of the worst situations have been fixed since the early 2.6
> versions (2.6.11 was still horrible), but it's the general design which
> makes such cases hard to find and to fix.

I see. I was sure I was missing something, since everyone around here are 
praising these new schedulers. However, the example you provide is not so 
much about fair/unfair, but about good/bad scheduler. I mean, if a process 
consumes 48% CPU and the other one also consumes 48% (if each were running 
alone), when you run them at the same time a fair scheduler would give them 
48% to each, but an unfair one should also give them 48% to each. I still 
fail to see the advantages of the "fair scheduler" design (even if I accept 
that CFS is technically much better than mainline and thus work better in 
your example).

> Your situation is clearly something standard where only the user can decide
> which application should get more CPU. You start two CPU hogs, and your
> mind tells you that you would prefer the video not to skip and the encoding
> to finish later. For other people, it might be the reverse because the
> video will be there just for monitoring purposes. The nice command was
> invented exactly for that, and people have been happily using it for the
> last 30 years. I don't see why it should become hard to use ! Many
> CPU-sensitive applications already provide the ability to change their nice
> value themselves using a command line parameter.
>
> And if it is still too hard to assign nice values from within your
> environment, then perhaps it's the environment which does not suit your
> needs. Even Windows users can choose to assign more priorities to
> foreground/background tasks, so it is something universally accepted,
> and I would be really surprized that you would not have the ability
> to proceed the same way.

I think you're not considering normal users here. Believe it or not, 99% of 
desktop users in the world just click on a icon to watch a video. And they DO 
want watch them, not use them for monitoring purposes (whatever that means).

I acknowledge it's impossible to be inside a user's mind to decide what it's 
more important to him/her, but let's agree that clearly a audio/video player 
should have by default a higher priority than an audio/video encoder, for the 
simple reason that one task requires a certain amount of CPU to do the job 
correctly, while the other one can do the job correctly regardless of how 
much CPU time you give it. They are different in nature. What I don't know is 
if knowing this should belong to the CPU scheduler or to the application 
itself. But the bottom line is that on a desktop, tasks should receive 
different -unfair- amounts of CPU time to work correctly. The "fair" concept 
still looks wrong to me.

Nicing tasks might not be hard at all, but expecting normal users to do so is 
not realistic. Either the scheduler or the applications should make these 
decisions for them (us).

> Regards,
> Willy

Thanks,
Alberto.

Re: Question about fair schedulers

[Russell Harmon]

> I think you're not considering normal users here. Believe it or not, 99% of
> desktop users in the world just click on a icon to watch a video. And they DO
> want watch them, not use them for monitoring purposes (whatever that means).
>
> I acknowledge it's impossible to be inside a user's mind to decide what it's
> more important to him/her, but let's agree that clearly a audio/video player
> should have by default a higher priority than an audio/video encoder, for the
> simple reason that one task requires a certain amount of CPU to do the job
> correctly, while the other one can do the job correctly regardless of how
> much CPU time you give it. They are different in nature. What I don't know is
> if knowing this should belong to the CPU scheduler or to the application
> itself. But the bottom line is that on a desktop, tasks should receive
> different -unfair- amounts of CPU time to work correctly. The "fair" concept
> still looks wrong to me.
>
> Nicing tasks might not be hard at all, but expecting normal users to do so is
> not realistic. Either the scheduler or the applications should make these
> decisions for them (us).
While I agree in principle that less work for the end user who wants
it to "just work" is good (if done correctly), I think this is more an
issue of where the scheduler is being put to work. In a desktop
environment, I'd agree that you would want the player > encoder, but
in say a video authoring machine, you would definatley want the
encoder > player. It seems to me that the best solution would be a
compile time option to configure the scheduler for the environment it
will be working in.
I do think however that the default in most cases should be "it just
works" with the option of turning on the more advanced features.

Re: Question about fair schedulers

[Willy Tarreau]

On Sat, Jun 23, 2007 at 11:18:43AM +0200, Alberto Gonzalez wrote:
> On Saturday 23 June 2007, Willy Tarreau wrote:
> > On Sat, Jun 23, 2007 at 10:01:02AM +0200, Alberto Gonzalez wrote:
> > > I see. So you mean that in 90% of the cases the mainline scheduler
> > > behaves better than fair schedulers, but when its "logic" fails it
> > > behaves much worse (the other 10% cases)? In my very simple test scenario
> > > the mainline scheduler did behave much better. Maybe the problem comes
> > > with very complex scenarios like the ones I've seen when testing these 2
> > > fair schedulers (something like compiling a kernel while you open 5
> > > instances of glxgears, write an email, play music in Amarok and watch 2
> > > HD videos all at the same time). The question would then be if these kind
> > > of situations are likely to happen in real world, or even if it doesn't
> > > make more sense to try to improve the logic of the mainline scheduler so
> > > that those 10% cases are handled better instead of writing a new one that
> > > would behave worse in 90% of the cases and better in the other 10%.
> >
> > No, in fact, the mainline scheduler is good only for the very particular
> > cases you propose here. But on some fairly trivial situations (such as two
> > processes consuming very close to 50% CPU), it can be a disaster. I've had
> > proxies under medium load slow down sshd so much that it was impossible to
> > log in. Most of the worst situations have been fixed since the early 2.6
> > versions (2.6.11 was still horrible), but it's the general design which
> > makes such cases hard to find and to fix.
> 
> I see. I was sure I was missing something, since everyone around here are 
> praising these new schedulers. However, the example you provide is not so 
> much about fair/unfair, but about good/bad scheduler. I mean, if a process 
> consumes 48% CPU and the other one also consumes 48% (if each were running 
> alone), when you run them at the same time a fair scheduler would give them 
> 48% to each, but an unfair one should also give them 48% to each. I still 
> fail to see the advantages of the "fair scheduler" design (even if I accept 
> that CFS is technically much better than mainline and thus work better in 
> your example).
> 
> > Your situation is clearly something standard where only the user can decide
> > which application should get more CPU. You start two CPU hogs, and your
> > mind tells you that you would prefer the video not to skip and the encoding
> > to finish later. For other people, it might be the reverse because the
> > video will be there just for monitoring purposes. The nice command was
> > invented exactly for that, and people have been happily using it for the
> > last 30 years. I don't see why it should become hard to use ! Many
> > CPU-sensitive applications already provide the ability to change their nice
> > value themselves using a command line parameter.
> >
> > And if it is still too hard to assign nice values from within your
> > environment, then perhaps it's the environment which does not suit your
> > needs. Even Windows users can choose to assign more priorities to
> > foreground/background tasks, so it is something universally accepted,
> > and I would be really surprized that you would not have the ability
> > to proceed the same way.
> 
> I think you're not considering normal users here.

Yes I am.

> Believe it or not, 99% of 
> desktop users in the world just click on a icon to watch a video. And they DO 
> want watch them, not use them for monitoring purposes (whatever that means).

Of course, I was just trying to give you an example.

> I acknowledge it's impossible to be inside a user's mind to decide what it's 
> more important to him/her, but let's agree that clearly a audio/video player 
> should have by default a higher priority than an audio/video encoder, for the 
> simple reason that one task requires a certain amount of CPU to do the job 
> correctly, while the other one can do the job correctly regardless of how 
> much CPU time you give it. They are different in nature.

Yes, one of them works in real time, the other one does not.

> What I don't know is 
> if knowing this should belong to the CPU scheduler or to the application 
> itself.

In your opinion ? Who between a few hundred lines function designed to fit
all needs, and a specific MPEG player is the most aware that the MPEG player
is very likely to require real time scheduling by nature ? At least, the old
version of mplayer I have on my machine (1.0pre7) has a "-priority" option.
Also, I remember that cdrecord automatically tries to go realtime in order
to avoid underrunning the buffer.

> But the bottom line is that on a desktop, tasks should receive 
> different -unfair- amounts of CPU time to work correctly. The "fair" concept 
> still looks wrong to me.

"fair" means what it means : stop starving some tasks for no apparent reasons.
If one task adjusts its priority, it can get more CPU than others, but the
distribution will still be fair according to the priorities.

> Nicing tasks might not be hard at all, but expecting normal users to do so is 
> not realistic. Either the scheduler or the applications should make these 
> decisions for them (us).

No, I cannot agree with you. The users have to solutions to start their player:
  - typing "mplayer xxx.mpeg" on the command line ; then they can prepend
    "nice" in front of it

  - clicking on an icon in their windows-like window managers, which makes
    executes the command for them.

If they decide to use the second solution, it means that the default settings
assigned to the icon should fit the application (that applies to the nice
value too). And if their distro ships with those pre-defined icons with stupid
priorities, they should complain to the distro vendor or switch to another one.
And if the window manager by itself does not make it easy to adjust priorities
when starting processes, it's poorly designed because it is it and only it
which forces the user to open a command line and manually set "nice".

So there are plenty of really transparent solutions for the user, but maybe
there are a lot of wrong tools and configurations...

Regards,
Willy

Re: Question about fair schedulers

[Alberto Gonzalez]

On Saturday 23 June 2007, Willy Tarreau wrote:

> > But the bottom line is that on a desktop, tasks should receive
> > different -unfair- amounts of CPU time to work correctly. The "fair"
> > concept still looks wrong to me.
>
> "fair" means what it means : stop starving some tasks for no apparent
> reasons. If one task adjusts its priority, it can get more CPU than others,
> but the distribution will still be fair according to the priorities.

Ok, this was the kind of technical explanation I was looking for, thanks.

> > Nicing tasks might not be hard at all, but expecting normal users to do
> > so is not realistic. Either the scheduler or the applications should make
> > these decisions for them (us).
>
> No, I cannot agree with you. The users have to solutions to start their
> player: - typing "mplayer xxx.mpeg" on the command line ; then they can
> prepend "nice" in front of it
>
>   - clicking on an icon in their windows-like window managers, which makes
>     executes the command for them.
>
> If they decide to use the second solution, it means that the default
> settings assigned to the icon should fit the application (that applies to
> the nice value too). And if their distro ships with those pre-defined icons
> with stupid priorities, they should complain to the distro vendor or switch
> to another one. And if the window manager by itself does not make it easy
> to adjust priorities when starting processes, it's poorly designed because
> it is it and only it which forces the user to open a command line and
> manually set "nice".
>
> So there are plenty of really transparent solutions for the user, but maybe
> there are a lot of wrong tools and configurations...

Ok, so if I understand correctly, the problem I had in my simple test will be 
solved by distributions once a fair scheduler goes into mainline? This is 
fine then. As long as someone (but not end users) takes care of giving the 
right priority to tasks of different nature it should work fine.

> Regards,
> Willy

Thank you !
Alberto.

Re: Question about fair schedulers

[Willy Tarreau]

On Sat, Jun 23, 2007 at 12:45:30PM +0200, Alberto Gonzalez wrote:
> On Saturday 23 June 2007, Willy Tarreau wrote:
> 
> > > But the bottom line is that on a desktop, tasks should receive
> > > different -unfair- amounts of CPU time to work correctly. The "fair"
> > > concept still looks wrong to me.
> >
> > "fair" means what it means : stop starving some tasks for no apparent
> > reasons. If one task adjusts its priority, it can get more CPU than others,
> > but the distribution will still be fair according to the priorities.
> 
> Ok, this was the kind of technical explanation I was looking for, thanks.
> 
> > > Nicing tasks might not be hard at all, but expecting normal users to do
> > > so is not realistic. Either the scheduler or the applications should make
> > > these decisions for them (us).
> >
> > No, I cannot agree with you. The users have to solutions to start their
> > player: - typing "mplayer xxx.mpeg" on the command line ; then they can
> > prepend "nice" in front of it
> >
> >   - clicking on an icon in their windows-like window managers, which makes
> >     executes the command for them.
> >
> > If they decide to use the second solution, it means that the default
> > settings assigned to the icon should fit the application (that applies to
> > the nice value too). And if their distro ships with those pre-defined icons
> > with stupid priorities, they should complain to the distro vendor or switch
> > to another one. And if the window manager by itself does not make it easy
> > to adjust priorities when starting processes, it's poorly designed because
> > it is it and only it which forces the user to open a command line and
> > manually set "nice".
> >
> > So there are plenty of really transparent solutions for the user, but maybe
> > there are a lot of wrong tools and configurations...
> 
> Ok, so if I understand correctly, the problem I had in my simple test will be 
> solved by distributions once a fair scheduler goes into mainline?

No, there is no reason to wait for a fair scheduler at all. A *desktop* distro
*must* take process priorities into account, otherwise it's broken by design.
If you don't know where in your distro you can say "more CPU for the video"
and "less CPU for the encoder", then ask your vendor. It's something so much
basic and obvious that I cannot imagine not being supported. Even my mp3 player
script has been setting its prio for the last 8 years in order to avoid skips!

> This is 
> fine then. As long as someone (but not end users) takes care of giving the 
> right priority to tasks of different nature it should work fine.

Yes, it should.

Regards,
Willy

Re: Question about fair schedulers

[Alberto Gonzalez]

On Saturday 23 June 2007, Willy Tarreau wrote:
> On Sat, Jun 23, 2007 at 12:45:30PM +0200, Alberto Gonzalez wrote:
> > Ok, so if I understand correctly, the problem I had in my simple test
> > will be solved by distributions once a fair scheduler goes into mainline?
>
> No, there is no reason to wait for a fair scheduler at all. A *desktop*
> distro *must* take process priorities into account, otherwise it's broken
> by design.

Well, right now with the mainline scheduler it works fine. The player gets 
more CPU time than the encoder. It was when I switched to -ck kernel (with a 
fair scheduler) that it didn't work correctly, that's why I asked about 
this "fair" concept in the first place.

I guess once a fair scheduler goes mainline they will adapt things to work 
correctly again.

> Regards,
> Willy

Cheers,
Alberto.

Re: Question about fair schedulers

[Tom Spink]

On 23/06/07, Alberto Gonzalez <info@gnebu.es> wrote:
> On Saturday 23 June 2007, Willy Tarreau wrote:
> > On Sat, Jun 23, 2007 at 12:45:30PM +0200, Alberto Gonzalez wrote:
> > > Ok, so if I understand correctly, the problem I had in my simple test
> > > will be solved by distributions once a fair scheduler goes into mainline?
> >
> > No, there is no reason to wait for a fair scheduler at all. A *desktop*
> > distro *must* take process priorities into account, otherwise it's broken
> > by design.
>
> Well, right now with the mainline scheduler it works fine. The player gets
> more CPU time than the encoder. It was when I switched to -ck kernel (with a
> fair scheduler) that it didn't work correctly, that's why I asked about
> this "fair" concept in the first place.
>
> I guess once a fair scheduler goes mainline they will adapt things to work
> correctly again.
>
> > Regards,
> > Willy
>
> Cheers,
> Alberto.
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Alberto,

If you're feeling adventurous, grab the latest kernel and patch it
with Ingo's scheduler: CFS.

You may be pleasantly surprised.

-- 
Regards,
Tom Spink
University of Edinburgh

Re: Question about fair schedulers

[Alberto Gonzalez]

On Saturday 23 June 2007, Tom Spink wrote:
> Alberto,
>
> If you're feeling adventurous, grab the latest kernel and patch it
> with Ingo's scheduler: CFS.
>
> You may be pleasantly surprised.

Thanks, I might if I have to courage to patch and compile my own kernel :)

However, I'd also need to change all my applications to set them with the 
right priority to see the good results, so I think I might just wait until it 
lands in mainline.

Just to check if I understood everything correctly:

The mainline scheduler tries to be smart and guess the priority of each task, 
and while it mostly hits the nail right in the head, sometimes it hits you 
right in the thumb.

Fair schedulers, on the contrary, forget about trying to be smart and just 
care about being fair, leaving the priority settings to where they belong: 
applications.

Is this more or less correct?

Alberto.

Re: Question about fair schedulers

[Willy Tarreau]

On Sat, Jun 23, 2007 at 01:26:34PM +0200, Alberto Gonzalez wrote:
> On Saturday 23 June 2007, Tom Spink wrote:
> > Alberto,
> >
> > If you're feeling adventurous, grab the latest kernel and patch it
> > with Ingo's scheduler: CFS.
> >
> > You may be pleasantly surprised.
> 
> Thanks, I might if I have to courage to patch and compile my own kernel :)
> 
> However, I'd also need to change all my applications to set them with the 
> right priority to see the good results, so I think I might just wait until it 
> lands in mainline.
> 
> Just to check if I understood everything correctly:
> 
> The mainline scheduler tries to be smart and guess the priority of each task, 
> and while it mostly hits the nail right in the head, sometimes it hits you 
> right in the thumb.
> 
> Fair schedulers, on the contrary, forget about trying to be smart and just 
> care about being fair, leaving the priority settings to where they belong: 
> applications.
> 
> Is this more or less correct?

It is somewhat correct, except that nothing prevents them from trying to
be even smarter. For instance, your video player consumes some CPU, but
also calls X which itself consumes some CPU. There are tricks to track
what process makes X work so that both share the same slices, generally
resulting in improved smoothness without playing with applications priorities
yet.

It might be worth trying anyway.

Regards,
Willy

Re: Question about fair schedulers

[Bill Davidsen]

Alberto Gonzalez wrote:
> On Saturday 23 June 2007, Tom Spink wrote:
>> Alberto,
>>
>> If you're feeling adventurous, grab the latest kernel and patch it
>> with Ingo's scheduler: CFS.
>>
>> You may be pleasantly surprised.
> 
> Thanks, I might if I have to courage to patch and compile my own kernel :)
> 
> However, I'd also need to change all my applications to set them with the 
> right priority to see the good results, so I think I might just wait until it 
> lands in mainline.

In general not the case. I generally don't diddle my priorities, there's 
rarely a need.
> 
> Just to check if I understood everything correctly:
> 
> The mainline scheduler tries to be smart and guess the priority of each task, 
> and while it mostly hits the nail right in the head, sometimes it hits you 
> right in the thumb.
> 
> Fair schedulers, on the contrary, forget about trying to be smart and just 
> care about being fair, leaving the priority settings to where they belong: 
> applications.
> 
> Is this more or less correct?

Incomplete. The CFS scheduler seems to do better with latency, so you 
may get less CPU to a process but it doesn't wind up waiting a long time 
to get a fair share. So it "feels better" without micro tuning.

Face it, if you have more jobs than CPU no scheduler is going to make 
you really happy.
> 
> Alberto.
> 


-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Question about fair schedulers

[Paolo Ornati]

On Sat, 23 Jun 2007 10:01:02 +0200
Alberto Gonzalez <info@gnebu.es> wrote:

> I see. So you mean that in 90% of the cases the mainline scheduler behaves 
> better than fair schedulers, but when its "logic" fails it behaves much worse 
> (the other 10% cases)?

Yes and no... the "logic" is supposed to identify what processes are
somehow interactive and give them more priority / CPU time.

This makes the system behaves better when there are CPU hog processes
(like encoders etc...) because the interactive ones doesn't suffer too
much.

The big problem is that it can identify an almost CPU hog process as
interactive.... and giving him an insane amount of CPU starve the
others.

In my case it was "trancode", and I assure you... it wasn't funny.
Sometimes it happened that running it (at standard nice 0) made
the machine totally unusable!  (something like 30s to switch from X to
a virtual terminal... and I don't tell you how hard was doing login and
killing/renicing it).

So far I've seen these pathological behaviour only with trancode and
wine (only with particular programs I don't remember now).

But the fact is, the "interactivity estimator" is too fragile, and when
it fails it can do much damage.


Fair scheduler instead:
	- are robust
	- provide consistent behaviour
	- provide good interactivity within the bounds of fairness


> In my very simple test scenario the mainline scheduler 
> did behave much better.

Of course... because of the two competing processes the
"interactive" (for you) one needs 60%, that is more than it's 50% fair
share.

The real solution is to use nice levels so the scheduler doesn't have
to guess what process is more important.

And yes, programs/distributions should set good defaults for you... and
if they don't, just complain to them  :)

-- 
	Paolo Ornati
	Linux 2.6.22-rc5-g0864a4e2 on x86_64

Re: Question about fair schedulers

[Alberto Gonzalez]

On Saturday 23 June 2007, Paolo Ornati wrote:
> But the fact is, the "interactivity estimator" is too fragile, and when
> it fails it can do much damage.
>
>
> Fair scheduler instead:
> 	- are robust
> 	- provide consistent behaviour
> 	- provide good interactivity within the bounds of fairness

Yes, this is what I have concluded from this thread. Trying to guess which 
tasks should have higher priority is not the right approach. It's not a CPU 
scheduler's business to decide about priorities.

> > In my very simple test scenario the mainline scheduler
> > did behave much better.
>
> Of course... because of the two competing processes the
> "interactive" (for you) one needs 60%, that is more than it's 50% fair
> share.
>
> The real solution is to use nice levels so the scheduler doesn't have
> to guess what process is more important.

Yes, this seems to be the right approach from all the opinions I've heard.

> And yes, programs/distributions should set good defaults for you... and
> if they don't, just complain to them  :)

I'm sure they'll do once a fair scheduler goes into mainline :)

I guess what I was missing from the beginning is that "fair" means that the 
scheduler will be fair among tasks that have the same priority, but if a task 
has a higher priority, it _will_ get more CPU. So we'll just have to mark 
applications like video players, audio players or games with a high priority, 
others like encoders or compilers with low priority, and leave the rest 
(browsers, word processors, email readers, etc...) as normal priority. This 
way a fair scheduler would be able to give each task right amount of CPU.

That sounds like a good solution to me.

Thanks,
Alberto.

Re: Question about fair schedulers

[Paolo Ornati]

On Sat, 23 Jun 2007 15:56:36 +0200
Alberto Gonzalez <info@gnebu.es> wrote:

> > And yes, programs/distributions should set good defaults for you... and
> > if they don't, just complain to them  :)  
> 
> I'm sure they'll do once a fair scheduler goes into mainline :)

Some already does... for example the current version of:
	http://www.exit1.org/dvdrip/

it sets transcode nice to "+19" by default  :)

> 
> I guess what I was missing from the beginning is that "fair" means that the 
> scheduler will be fair among tasks that have the same priority, but if a task 
> has a higher priority, it _will_ get more CPU. So we'll just have to mark 
> applications like video players, audio players or games with a high priority, 
> others like encoders or compilers with low priority, and leave the rest 
> (browsers, word processors, email readers, etc...) as normal priority. This 
> way a fair scheduler would be able to give each task right amount of CPU.

Yes. I think that the more important thing is to nice background tasks
(like encoders etc..), then games / video players can run without
problems even without renicing (usually normal programs don't eat much
CPU).

-- 
	Paolo Ornati
	Linux 2.6.22-rc5-g0864a4e2 on x86_64