[REPORT] cfs-v5 vs sd-0.46

[REPORT] cfs-v5 vs sd-0.46

[Michael Gerdau]

[-- Attachment #1: Type: text/plain, Size: 8146 bytes --]

Hi list,

with cfs-v5 finally booting on my machine I have run my daily
numbercrunching jobs on both cfs-v5 and sd-0.46, 2.6.21-v7 on
top of a stock openSUSE 10.2 (X86_64). Config for both kernel
is the same except for the X boost option in cfs-v5 which on
my system didn't work (X still was @ -19; I understand this will
be fixed in -v6). HZ is 250 in both.

System is a Dell XPS M1710, Intel Core2 2.33GHz, 4GB,
NVIDIA GeForce Go 7950 GTX with proprietary driver 1.0-9755

I'm running three single threaded perl scripts that do double
precision floating point math with little i/o after initially
loading the data.

Both cfs and sd showed very similar behavior when monitored in top.
I'll show more or less representative excerpt from a 10 minutes
log, delay 3sec.

sd-0.46
top - 00:14:24 up  1:17,  9 users,  load average: 4.79, 4.95, 4.80
Tasks:   3 total,   3 running,   0 sleeping,   0 stopped,   0 zombie
Cpu(s): 99.8%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.2%hi,  0.0%si,  0.0%st
Mem:   3348628k total,  1648560k used,  1700068k free,    64392k buffers
Swap:  2097144k total,        0k used,  2097144k free,   828204k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                             
 6671 mgd       33   0 95508  22m 3652 R  100  0.7  44:28.11 perl                                                                                                                                                
 6669 mgd       31   0 95176  22m 3652 R   50  0.7  43:50.02 perl                                                                                                                                                
 6674
 mgd       31   0 95368  22m 3652 R   50  0.7  47:55.29 perl                                                                                                                                                

cfs-v5
top - 08:07:50 up 21 min,  9 users,  load average: 4.13, 4.16, 3.23
Tasks:   3 total,   3 running,   0 sleeping,   0 stopped,   0 zombie
Cpu(s): 99.5%us,  0.2%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   3348624k total,  1193500k used,  2155124k free,    32516k buffers
Swap:  2097144k total,        0k used,  2097144k free,   545568k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                             
 6357 mgd       20   0 92024  19m 3652 R  100  0.6   8:54.21 perl                                                                                                                                                
 6356 mgd       20   0 91652  18m 3652 R   50  0.6  10:35.52 perl                                                                                                                                                
 6359 mgd       20   0 91700  18m 3652 R   50  0.6   8:47.32 perl                                                                                                                                                

What did surprise me is that cpu utilization had been spread 100/50/50
(round robin) most of the time. I did expect 66/66/66 or so.

What I also don't understand is the difference in load average, sd
constantly had higher values, the above figures are representative
for the whole log. I don't know which is better though.


Here are excerpts from a concurrently run vmstat 3 200:

sd-0.46
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 5  0      0 1702928  63664 827876    0    0     0    67  458 1350 100  0  0  0
 3  0      0 1702928  63684 827876    0    0     0    89  468 1362 100  0  0  0
 5  0      0 1702680  63696 827876    0    0     0   132  461 1598 99  1  0  0
 8  0      0 1702680  63712 827892    0    0     0    80  465 1180 99  1  0  0
 3  0      0 1702712  63732 827884    0    0     0    67  453 1005 100  0  0  0
 4  0      0 1702792  63744 827920    0    0     0    41  461 1138 100  0  0  0
 3  0      0 1702792  63760 827916    0    0     0    57  456 1073 100  0  0  0
 3  0      0 1702808  63776 827928    0    0     0   111  473 1095 100  0  0  0
 3  0      0 1702808  63788 827928    0    0     0    81  461 1092 99  1  0  0
 3  0      0 1702188  63808 827928    0    0     0   160  463 1437 99  1  0  0
 3  0      0 1702064  63884 827900    0    0     0   229  479 1125 99  0  0  0
 4  0      0 1702064  63912 827972    0    0     1    77  460 1108 100  0  0  0
 7  0      0 1702032  63920 828000    0    0     0    40  463 1068 100  0  0  0
 4  0      0 1702048  63928 828008    0    0     0    68  454 1114 100  0  0  0
11  0      0 1702048  63928 828008    0    0     0     0  458 1001 100  0  0  0
 3  0      0 1701500  63960 828020    0    0     0   189  470 1538 99  1  0  0
 3  0      0 1701476  63968 828020    0    0     0    57  461 1111 100  0  0  0
 4  0      0 1701508  63996 828044    0    0     0   105  458 1093 99  1  0  0
 4  0      0 1701428  64012 828044    0    0     0   127  471 1341 100  0  0  0
 5  0      0 1701356  64028 828040    0    0     0    55  458 1344 100  0  0  0
 3  0      0 1701356  64028 828056    0    0     0    15  462 1291 100  0  0  0

cfs-v5
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 6  0      0 2157728  31816 545236    0    0     0   103  543  748 100  0  0  0
 4  0      0 2157780  31828 545256    0    0     0    63  435  752 100  0  0  0
 4  0      0 2157928  31852 545256    0    0     0   105  424  770 100  0  0  0
 4  0      0 2157928  31868 545268    0    0     0   261  457  763 100  0  0  0
 4  0      0 2157928  31884 545280    0    0     0   113  435  765 100  0  0  0
 4  0      0 2157928  31900 545288    0    0     0    52  422  745 100  0  0  0
 4  0      0 2157556  31932 545284    0    0     0   169  436 1010 100  0  0  0
 4  0      0 2157556  31952 545296    0    0     0    72  424  736 100  0  0  0
 4  0      0 2157556  31960 545304    0    0     0    35  428  743 100  0  0  0
 4  0      0 2157556  31984 545308    0    0     0    91  425  710 99  1  0  0
 4  0      0 2157556  31992 545320    0    0     0    35  428  738 100  0  0  0
 5  0      0 2157556  32016 545320    0    0     0   105  425  729 100  0  0  0
 4  0      0 2157432  32052 545336    0    0     0   197  434  989 99  1  0  0
 5  0      0 2157448  32060 545352    0    0     0    36  421  767 100  0  0  0
 4  0      0 2157448  32076 545356    0    0     0   127  441  752 100  0  0  0
 6  0      0 2157448  32092 545368    0    0     0    69  422  784 99  1  0  0
 4  0      0 2157324  32116 545388    0    0     0   191  445  734 100  0  0  0
 4  0      0 2157200  32148 545400    0    0     0   123  427  773 100  0  0  0
 4  0      0 2157200  32156 545412    0    0     0    39  428  713 100  0  0  0
 7  0      0 2156844  32184 545412    0    0     1   161  429 1360 99  1  0  0
 6  0      0 2156348  32192 545416    0    0     0    32  427  723 100  0  0  0


Last not least I'd like to add that at least on my system having X
niced to -19 does result in kind of "erratic" (for lack of a better
word) desktop behavior. I'll will reevaluate this with -v6 but for
now IMO nicing X to -19 is a regression at least on my machine despite
the claim that cfs doesn't suffer from it.

Best,
Michael

PS: Only learning how to test these things I'm happy to get pointed
out the shortcomings of what I tested above. Of course suggestions for
improvements are welcome.
-- 
 Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar
 Sitz Hamburg; HRB 89145 Amtsgericht Hamburg
 Vote against SPAM - see http://www.politik-digital.de/spam/
 Michael Gerdau       email: mgd@technosis.de
 GPG-keys available on request or at public keyserver

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [REPORT] cfs-v5 vs sd-0.46

[Ingo Molnar]

* Michael Gerdau <mgd@technosis.de> wrote:

> I'm running three single threaded perl scripts that do double 
> precision floating point math with little i/o after initially loading 
> the data.

thanks for the testing!

> What I also don't understand is the difference in load average, sd 
> constantly had higher values, the above figures are representative for 
> the whole log. I don't know which is better though.

hm, it's hard from here to tell that. What load average does the vanilla 
kernel report? I'd take that as a reference.

> Here are excerpts from a concurrently run vmstat 3 200:
> 
> sd-0.46
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
>  5  0      0 1702928  63664 827876    0    0     0    67  458 1350 100  0  0  0
>  3  0      0 1702928  63684 827876    0    0     0    89  468 1362 100  0  0  0
>  5  0      0 1702680  63696 827876    0    0     0   132  461 1598 99  1  0  0
>  8  0      0 1702680  63712 827892    0    0     0    80  465 1180 99  1  0  0

> cfs-v5
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
>  6  0      0 2157728  31816 545236    0    0     0   103  543  748 100  0  0  0
>  4  0      0 2157780  31828 545256    0    0     0    63  435  752 100  0  0  0
>  4  0      0 2157928  31852 545256    0    0     0   105  424  770 100  0  0  0
>  4  0      0 2157928  31868 545268    0    0     0   261  457  763 100  0  0  0

interesting - CFS has half the context-switch rate of SD. That is 
probably because on your workload CFS defaults to longer 'timeslices' 
than SD. You can influence the 'timeslice length' under SD via 
/proc/sys/kernel/rr_interval (milliseconds units) and under CFS via 
/proc/sys/kernel/sched_granularity_ns. On CFS the value is not 
necessarily the timeslice length you will observe - for example in your 
workload above the granularity is set to 5 msec, but your rescheduling 
rate is 13 msecs. SD default to a rr_interval value of 8 msecs, which in 
your workload produces a timeslice length of 6-7 msecs.

so to be totally 'fair' and get the same rescheduling 'granularity' you 
should probably lower CFS's sched_granularity_ns to 2 msecs.

> Last not least I'd like to add that at least on my system having X 
> niced to -19 does result in kind of "erratic" (for lack of a better 
> word) desktop behavior. I'll will reevaluate this with -v6 but for now 
> IMO nicing X to -19 is a regression at least on my machine despite the 
> claim that cfs doesn't suffer from it.

indeed with -19 the rescheduling limit is so high under CFS that it does 
not throttle X's scheduling rate enough and so it will make CFS behave 
as badly as other schedulers.

I retested this with -10 and it should work better with that. In -v6 i 
changed the default to -10 too.

> PS: Only learning how to test these things I'm happy to get pointed 
> out the shortcomings of what I tested above. Of course suggestions for 
> improvements are welcome.

your report was perfectly fine and useful. "no visible regressions" is 
valuable feedback too. [ In fact, such type of feedback is the one i 
find the easiest to resolve ;-) ]

Since you are running number-crunchers you might be able to give 
performacne feedback too: do you have any reliable 'performance metric' 
available for your number cruncher jobs (ops per minute, runtime, etc.) 
so that it would be possible to compare number-crunching performance of 
mainline to SD and to CFS as well? If that value is easy to get and 
reliable/stable enough to be meaningful. (And it would be nice to also 
establish some ballpark figure about how much noise there is in any 
performance metric, so that we can see whether any differences between 
schedulers are systematic or not.)

	Ingo

Re: [REPORT] cfs-v5 vs sd-0.46

[Michael Gerdau]

[-- Attachment #1: Type: text/plain, Size: 1819 bytes --]

> > What I also don't understand is the difference in load average, sd 
> > constantly had higher values, the above figures are representative for 
> > the whole log. I don't know which is better though.
> 
> hm, it's hard from here to tell that. What load average does the vanilla 
> kernel report? I'd take that as a reference.

I will redo this test with sd-0.46, cfs-v5 and mainline later today.

> interesting - CFS has half the context-switch rate of SD. That is 
> probably because on your workload CFS defaults to longer 'timeslices' 
> than SD. You can influence the 'timeslice length' under SD via 
> /proc/sys/kernel/rr_interval (milliseconds units) and under CFS via 
> /proc/sys/kernel/sched_granularity_ns. On CFS the value is not 
> necessarily the timeslice length you will observe - for example in your 
> workload above the granularity is set to 5 msec, but your rescheduling 
> rate is 13 msecs. SD default to a rr_interval value of 8 msecs, which in 
> your workload produces a timeslice length of 6-7 msecs.
> 
> so to be totally 'fair' and get the same rescheduling 'granularity' you 
> should probably lower CFS's sched_granularity_ns to 2 msecs.

I'll change default nice in cfs to -10.

I'm also happy to adjust /proc/sys/kernel/sched_granularity_ns to 2msec.
However checking /proc/sys/kernel/rr_interval reveals it is 16 (msec)
on my system.

Anyway, I'll have to do some urgent other work and won't be able to
do lots of testing until tonight (but then I will).

Best,
Michael
-- 
 Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar
 Sitz Hamburg; HRB 89145 Amtsgericht Hamburg
 Vote against SPAM - see http://www.politik-digital.de/spam/
 Michael Gerdau       email: mgd@technosis.de
 GPG-keys available on request or at public keyserver

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [REPORT] cfs-v5 vs sd-0.46

[Ingo Molnar]

* Michael Gerdau <mgd@technosis.de> wrote:

> > so to be totally 'fair' and get the same rescheduling 'granularity' 
> > you should probably lower CFS's sched_granularity_ns to 2 msecs.
> 
> I'll change default nice in cfs to -10.
> 
> I'm also happy to adjust /proc/sys/kernel/sched_granularity_ns to 
> 2msec. However checking /proc/sys/kernel/rr_interval reveals it is 16 
> (msec) on my system.

ah, yeah - there due to the SMP rule in SD:

       rr_interval *= 1 + ilog2(num_online_cpus());

and you have a 2-CPU system, so you get 8msec*2 == 16 msecs default 
interval. I find this a neat solution and i have talked to Con about 
this already and i'll adopt Con's idea in CFS too. Nevertheless, despite 
the settings, SD seems to be rescheduling every 6-7 msecs, while CFS 
reschedules only every 13 msecs.

Here i'm assuming that the vmstats are directly comparable: that your 
number-crunchers behave the same during the full runtime - is that 
correct? (If not then the vmstat result should be run at roughly the 
same type of "stage" of the workload, on all the schedulers.)

	Ingo

Re: [REPORT] cfs-v5 vs sd-0.46

[Michael Gerdau]

[-- Attachment #1: Type: text/plain, Size: 672 bytes --]

> Here i'm assuming that the vmstats are directly comparable: that your 
> number-crunchers behave the same during the full runtime - is that 
> correct?

Yes, basically it does (disregarding small fluctuations)

I'll see whether I can produce some type of absolute performance
measure as well. Thinking about it I guess this should be fairly
simple to implement.

Best,
Michael
-- 
 Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar
 Sitz Hamburg; HRB 89145 Amtsgericht Hamburg
 Vote against SPAM - see http://www.politik-digital.de/spam/
 Michael Gerdau       email: mgd@technosis.de
 GPG-keys available on request or at public keyserver

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [REPORT] cfs-v5 vs sd-0.46

[Ingo Molnar]

* Michael Gerdau <mgd@technosis.de> wrote:

> > Here i'm assuming that the vmstats are directly comparable: that 
> > your number-crunchers behave the same during the full runtime - is 
> > that correct?
> 
> Yes, basically it does (disregarding small fluctuations)

ok, good.

> I'll see whether I can produce some type of absolute performance 
> measure as well. Thinking about it I guess this should be fairly 
> simple to implement.

oh, you are writing the number-cruncher? In general the 'best' 
performance metrics for scheduler validation are the ones where you have 
immediate feedback: i.e. some ops/sec (or ops per minute) value in some 
readily accessible place, or some "milliseconds-per-100,000 ops" type of 
metric - whichever lends itself better to the workload at hand. If you 
measure time then the best is to use long long and nanoseconds and the 
monotonic clocksource:

 unsigned long long rdclock(void)
 {
	struct timespec ts;

	clock_gettime(CLOCK_MONOTONIC, &ts);

        return ts.tv_sec * 1000000000ULL + ts.tv_nsec;
 }

(link to librt via -lrt to pick up clock_gettime())

The cost of a clock_gettime() (or of a gettimeofday()) can be a couple 
of microseconds on some systems, so it shouldnt be done too frequently.

Plus an absolute metric of "the whole workload took X.Y seconds" is 
useful too.

	Ingo

Re: [REPORT] cfs-v5 vs sd-0.46

[Michael Gerdau]

[-- Attachment #1: Type: text/plain, Size: 1132 bytes --]

> oh, you are writing the number-cruncher?

Yep.

> In general the 'best'  
> performance metrics for scheduler validation are the ones where you have 
> immediate feedback: i.e. some ops/sec (or ops per minute) value in some 
> readily accessible place, or some "milliseconds-per-100,000 ops" type of 
> metric - whichever lends itself better to the workload at hand.

I'll have to see whether that works out. I don't have an easily
available ops/sec but I guess I could create something similar.

> If you  
> measure time then the best is to use long long and nanoseconds and the 
> monotonic clocksource:

[snip]
Thanks, I will implement that, for Linux anyway.

> Plus an absolute metric of "the whole workload took X.Y seconds" is 
> useful too.

That's the easiest to come by and is already available.

Best,
Michael
-- 
 Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar
 Sitz Hamburg; HRB 89145 Amtsgericht Hamburg
 Vote against SPAM - see http://www.politik-digital.de/spam/
 Michael Gerdau       email: mgd@technosis.de
 GPG-keys available on request or at public keyserver

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [ck] [REPORT] cfs-v5 vs sd-0.46

[Con Kolivas]

On Tuesday 24 April 2007 17:37, Michael Gerdau wrote:
> Hi list,
>
> with cfs-v5 finally booting on my machine I have run my daily
> numbercrunching jobs on both cfs-v5 and sd-0.46, 2.6.21-v7 on
> top of a stock openSUSE 10.2 (X86_64).

Thanks for testing.

> Both cfs and sd showed very similar behavior when monitored in top.
> I'll show more or less representative excerpt from a 10 minutes
> log, delay 3sec.
>
> sd-0.46
> top - 00:14:24 up  1:17,  9 users,  load average: 4.79, 4.95, 4.80
> Tasks:   3 total,   3 running,   0 sleeping,   0 stopped,   0 zombie
> Cpu(s): 99.8%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.2%hi,  0.0%si, 
> 0.0%st Mem:   3348628k total,  1648560k used,  1700068k free,    64392k
> buffers Swap:  2097144k total,        0k used,  2097144k free,   828204k
> cached
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  6671 mgd       33   0 95508  22m 3652 R  100  0.7  44:28.11 perl
>  6669 mgd       31   0 95176  22m 3652 R   50  0.7  43:50.02 perl
>  6674
>  mgd       31   0 95368  22m 3652 R   50  0.7  47:55.29 perl
>
> cfs-v5
> top - 08:07:50 up 21 min,  9 users,  load average: 4.13, 4.16, 3.23
> Tasks:   3 total,   3 running,   0 sleeping,   0 stopped,   0 zombie
> Cpu(s): 99.5%us,  0.2%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.3%si, 
> 0.0%st Mem:   3348624k total,  1193500k used,  2155124k free,    32516k
> buffers Swap:  2097144k total,        0k used,  2097144k free,   545568k
> cached
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  6357 mgd       20   0 92024  19m 3652 R  100  0.6   8:54.21 perl
>  6356 mgd       20   0 91652  18m 3652 R   50  0.6  10:35.52 perl
>  6359 mgd       20   0 91700  18m 3652 R   50  0.6   8:47.32 perl
>
> What did surprise me is that cpu utilization had been spread 100/50/50
> (round robin) most of the time. I did expect 66/66/66 or so.

You have 3 tasks and only 2 cpus. The %cpu is the percentage of the cpu the 
task is currently on that it is using; it is not the percentage of 
the "overall cpu available on the machine". Since you have 3 tasks and 2 
cpus, the extra task will always be on one or the other cpu taking half of 
the cpu but never on both cpus.

> What I also don't understand is the difference in load average, sd
> constantly had higher values, the above figures are representative
> for the whole log. I don't know which is better though.

There isn't much useful to say about the load average in isolation. It may be 
meaningful or not depending on whether it just shows the timing of when the 
cpu load is determined, or whether there is more time waiting in runqueues. 
Only throughput measurements can really tell them apart.

What is important is that if all three tasks are fully cpu bound and started 
at the same time at the same nice level, that they all receive close to the 
same total cpu time overall showing some fairness is working as well. This 
should be the case no matter how many cpus you have.

-- 
-ck

Re: [ck] [REPORT] cfs-v5 vs sd-0.46

[Michael Gerdau]

[-- Attachment #1: Type: text/plain, Size: 1996 bytes --]

> > with cfs-v5 finally booting on my machine I have run my daily
> > numbercrunching jobs on both cfs-v5 and sd-0.46, 2.6.21-v7 on
> > top of a stock openSUSE 10.2 (X86_64).
> 
> Thanks for testing.

I actually enjoyed it -- the more extensive test I had promised
two days ago is almost finished. There is just one test I have
yet to (re)run and I will have a slot for it later today so I'll
mail out the results comparing
    2.6.21-rc7 (mainline)
    2.6.21-rc7-sd046
    2.6.21-rc7-cfs-v6-rc2 (X @ nice 0)
    2.6.21-rc7-cfs-v6-rc2 (X @ nice -10)
during the early afternoon (my time).

> You have 3 tasks and only 2 cpus. The %cpu is the percentage of the cpu the 
> task is currently on that it is using; it is not the percentage of 
> the "overall cpu available on the machine". Since you have 3 tasks and 2 
> cpus, the extra task will always be on one or the other cpu taking half of 
> the cpu but never on both cpus.

I had assumed that given the interval of 3 sec the three tasks would
be evenly distributed among the 2 CPUs thus resulting in a CPU% of
66 each because that's what they get in the long run anyway.

Apparently 3 sec is too short an interval to see this.

> What is important is that if all three tasks are fully cpu bound and started 
> at the same time at the same nice level, that they all receive close to the 
> same total cpu time overall showing some fairness is working as well. This 
> should be the case no matter how many cpus you have.

They are started via 'make -j3' which implies they start at the
same time (i.e. within a few msec). They initially load some data
and then perform extensive computations on that data.

Best,
Michael
-- 
 Technosis GmbH, Geschäftsführer: Michael Gerdau, Tobias Dittmar
 Sitz Hamburg; HRB 89145 Amtsgericht Hamburg
 Vote against SPAM - see http://www.politik-digital.de/spam/
 Michael Gerdau       email: mgd@technosis.de
 GPG-keys available on request or at public keyserver

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]