scheduler vs hardware? (was Re: another i7 (linux) bug?)

scheduler vs hardware? (was Re: another i7 (linux) bug?)

[Alex Chiang]

Scheduler folks cc'ed.

* Luca Zini <luca.zini@gmail.com>:
> Dear all, 
> I find your addresses in the thread 
> (http://bugzilla.kernel.org/show_bug.cgi?id=15064) that I followed.

In general, it's better to send mail to the mailing lists in case
individual developers don't have time/expertise to help you.

Which kernel are you using?

This may be an issue for the scheduler folks since in your
experiment, the only thing you're changing are the nice levels,
not hardware.

> I had the same problem that now seems to be solved. There is
> also another peculiarity in my system:
>
> if I run a cpu intensive process with the lowest priority (19
> from man nice) I obtain much better performance that with the
> highest priority available (-20 from man nice).
>
> For example the same file is processed by lame in 8.7 seconds
> at the lowest priority, and in 12 seconds at the highest
> priority.  Before posting a bug I wold like to understand if
> this is a problem related to the i7 mobile (my processor is a
> i7 Q720).
> 
> As far as I tested on the same laptop series (dell studio 15),
> with the same kernel this problem does not exists.
> 
> Can you test your i7 mobile system to help me to understand if
> it is related to the processor?
> 
> Regards, Luca Zini
> 

Re: scheduler vs hardware? (was Re: another i7 (linux) bug?)

[Mike Galbraith]

> if I run a cpu intensive process with the lowest priority (19
> from man nice) I obtain much better performance that with the
> highest priority available (-20 from man nice).
>
> For example the same file is processed by lame in 8.7 seconds
> at the lowest priority, and in 12 seconds at the highest
> priority.  Before posting a bug I wold like to understand if
> this is a problem related to the i7 mobile (my processor is a
> i7 Q720).
> 
> As far as I tested on the same laptop series (dell studio 15),
> with the same kernel this problem does not exists.

So you only see this on the i7.  That's odd.  Can you try 33-rc5?

Posting a reliable reproducer would be nice.  It'd also be nice to see
what all is running when you see this, and where.

	-Mike

Re: scheduler vs hardware? (was Re: another i7 (linux) bug?)

[Peter Zijlstra]

On Fri, 2010-01-22 at 08:19 +0100, Mike Galbraith wrote:
> > if I run a cpu intensive process with the lowest priority (19
> > from man nice) I obtain much better performance that with the
> > highest priority available (-20 from man nice).
> >
> > For example the same file is processed by lame in 8.7 seconds
> > at the lowest priority, and in 12 seconds at the highest
> > priority.  Before posting a bug I wold like to understand if
> > this is a problem related to the i7 mobile (my processor is a
> > i7 Q720).
> > 
> > As far as I tested on the same laptop series (dell studio 15),
> > with the same kernel this problem does not exists.
> 
> So you only see this on the i7.  That's odd.  Can you try 33-rc5?
> 
> Posting a reliable reproducer would be nice.  It'd also be nice to see
> what all is running when you see this, and where.

Using a sample from: http://lame.sourceforge.net/quality.php

My laptop does:


# time nice -n 19 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null
LAME 3.98.2 64bits (http://www.mp3dev.org/)
Using polyphase lowpass filter, transition band: 19383 Hz - 19916 Hz
Encoding youcantdothat.wav to <stdout>
Encoding as 44.1 kHz j-stereo MPEG-1 Layer III VBR(q=0)
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA
  1147/1147  (100%)|    0:03/    0:03|    0:03/    0:03|   9.3341x|    0:00
 32 [   1] *
256 [1009] %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
320 [ 137] %%%%%%%%%
----------------------------------------------------------------------------
   kbps        LR    MS  %     long switch short %
  263.4       99.1   0.9        90.5   5.5   4.0

real    0m3.273s
user    0m3.217s
sys     0m0.022s


# time nice -n 0 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null 
LAME 3.98.2 64bits (http://www.mp3dev.org/)
Using polyphase lowpass filter, transition band: 19383 Hz - 19916 Hz
Encoding youcantdothat.wav to <stdout>
Encoding as 44.1 kHz j-stereo MPEG-1 Layer III VBR(q=0)
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA
  1147/1147  (100%)|    0:01/    0:01|    0:01/    0:01|   27.239x|    0:00
 32 [   1] *
256 [1009] %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
320 [ 137] %%%%%%%%%
----------------------------------------------------------------------------
   kbps        LR    MS  %     long switch short %
  263.4       99.1   0.9        90.5   5.5   4.0

real    0m1.121s
user    0m1.102s
sys     0m0.013s


# time nice -n -20 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null
LAME 3.98.2 64bits (http://www.mp3dev.org/)
Using polyphase lowpass filter, transition band: 19383 Hz - 19916 Hz
Encoding youcantdothat.wav to <stdout>
Encoding as 44.1 kHz j-stereo MPEG-1 Layer III VBR(q=0)
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA
  1147/1147  (100%)|    0:01/    0:01|    0:01/    0:01|   27.239x|    0:00
 32 [   1] *
256 [1009] %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
320 [ 137] %%%%%%%%%
----------------------------------------------------------------------------
   kbps        LR    MS  %     long switch short %
  263.4       99.1   0.9        90.5   5.5   4.0

real    0m1.112s
user    0m1.093s
sys     0m0.018s



On a Nehalem class server machine it does:


# time nice -n 19 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null
LAME 3.98.2 64bits (http://www.mp3dev.org/)
Using polyphase lowpass filter, transition band: 19383 Hz - 19916 Hz
Encoding youcantdothat.wav to <stdout>
Encoding as 44.1 kHz j-stereo MPEG-1 Layer III VBR(q=0)
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA
  1147/1147  (100%)|    0:00/    0:00|    0:00/    0:00|   32.926x|    0:00
 32 [   1] *
256 [1009] %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
320 [ 137] %%%%%%%%%
----------------------------------------------------------------------------
   kbps        LR    MS  %     long switch short %
  263.4       99.1   0.9        90.5   5.5   4.0

real    0m0.932s
user    0m0.917s
sys     0m0.005s


# time nice -n 0 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null
LAME 3.98.2 64bits (http://www.mp3dev.org/)
Using polyphase lowpass filter, transition band: 19383 Hz - 19916 Hz
Encoding youcantdothat.wav to <stdout>
Encoding as 44.1 kHz j-stereo MPEG-1 Layer III VBR(q=0)
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA
  1147/1147  (100%)|    0:00/    0:00|    0:00/    0:00|   32.568x|    0:00
 32 [   1] *
256 [1009] %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
320 [ 137] %%%%%%%%%
----------------------------------------------------------------------------
   kbps        LR    MS  %     long switch short %
  263.4       99.1   0.9        90.5   5.5   4.0

real    0m0.927s
user    0m0.922s
sys     0m0.003s


# time nice -n -20 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null
LAME 3.98.2 64bits (http://www.mp3dev.org/)
Using polyphase lowpass filter, transition band: 19383 Hz - 19916 Hz
Encoding youcantdothat.wav to <stdout>
Encoding as 44.1 kHz j-stereo MPEG-1 Layer III VBR(q=0)
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA
  1147/1147  (100%)|    0:00/    0:00|    0:00/    0:00|   32.926x|    0:00
 32 [   1] *
256 [1009] %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
320 [ 137] %%%%%%%%%
----------------------------------------------------------------------------
   kbps        LR    MS  %     long switch short %
  263.4       99.1   0.9        90.5   5.5   4.0

real    0m0.919s
user    0m0.914s
sys     0m0.005s

Re: scheduler vs hardware? (was Re: another i7 (linux) bug?)

[Mike Galbraith]

On Fri, 2010-01-22 at 09:51 +0100, Peter Zijlstra wrote:
> On Fri, 2010-01-22 at 08:19 +0100, Mike Galbraith wrote:
> > > if I run a cpu intensive process with the lowest priority (19
> > > from man nice) I obtain much better performance that with the
> > > highest priority available (-20 from man nice).
> > >
> > > For example the same file is processed by lame in 8.7 seconds
> > > at the lowest priority, and in 12 seconds at the highest
> > > priority.  Before posting a bug I wold like to understand if
> > > this is a problem related to the i7 mobile (my processor is a
> > > i7 Q720).
> > > 
> > > As far as I tested on the same laptop series (dell studio 15),
> > > with the same kernel this problem does not exists.
> > 
> > So you only see this on the i7.  That's odd.  Can you try 33-rc5?
> > 
> > Posting a reliable reproducer would be nice.  It'd also be nice to see
> > what all is running when you see this, and where.
> 
> Using a sample from: http://lame.sourceforge.net/quality.php
> 
> My laptop does:
> 
> 
> # time nice -n 19 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null

> real    0m3.273s
> user    0m3.217s
> sys     0m0.022s
> 
> 
> # time nice -n 0 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null 

> real    0m1.121s
> user    0m1.102s
> sys     0m0.013s
> 
> 
> # time nice -n -20 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null

> real    0m1.112s
> user    0m1.093s
> sys     0m0.018s
> 
> 
> 
> On a Nehalem class server machine it does:
> 
> 
> # time nice -n 19 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null

> real    0m0.932s
> user    0m0.917s
> sys     0m0.005s
> 
> 
> # time nice -n 0 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null

> real    0m0.927s
> user    0m0.922s
> sys     0m0.003s
> 
> 
> # time nice -n -20 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null

> real    0m0.919s
> user    0m0.914s
> sys     0m0.005s

Weird.  Here there is zip squat difference, as expected with 1 thread.

time nice -n 19 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null

real    0m0.912s
user    0m0.908s
sys     0m0.000s

time nice -n 0 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null

real    0m0.912s
user    0m0.904s
sys     0m0.004s

time nice -n -20 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null

real    0m0.912s
user    0m0.904s
sys     0m0.004s

(bah, who needs a nehalem;)

	-Mike

Re: scheduler vs hardware? (was Re: another i7 (linux) bug?)

[Mike Galbraith]

On Fri, 2010-01-22 at 09:51 +0100, Peter Zijlstra wrote:

> My laptop does:
> 
> 
> # time nice -n 19 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null

> real    0m3.273s
> user    0m3.217s
> sys     0m0.022s
> 
> 
> # time nice -n 0 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null

> real    0m1.121s
> user    0m1.102s
> sys     0m0.013s

I don't see any way that can be scheduler (knocks wood).  Could it be
cpufreq?  IIRC there is or was an ignore nice gizmo in there.

	-Mike 

Re: scheduler vs hardware? (was Re: another i7 (linux) bug?)

[Peter Zijlstra]

On Fri, 2010-01-22 at 10:25 +0100, Mike Galbraith wrote:
> On Fri, 2010-01-22 at 09:51 +0100, Peter Zijlstra wrote:
> 
> > My laptop does:
> > 
> > 
> > # time nice -n 19 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null
> 
> > real    0m3.273s
> > user    0m3.217s
> > sys     0m0.022s
> > 
> > 
> > # time nice -n 0 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null
> 
> > real    0m1.121s
> > user    0m1.102s
> > sys     0m0.013s
> 
> I don't see any way that can be scheduler (knocks wood).  Could it be
> cpufreq?  IIRC there is or was an ignore nice gizmo in there.

That's a laptop with full desktop bloat, I'm sure either firefox or evo
farted, or both.

Re: scheduler vs hardware? (was Re: another i7 (linux) bug?)

[Luca Zini]

[-- Attachment #1: Type: Text/Plain, Size: 1959 bytes --]

First of all sorry for misunderstanding with Alex Chiang, I was trying to 
collect some data from other i7 mobile users to try to isolate the problem 
before posting.

I tried different things: I disabled speedstep from the bios and the results 
where more sensible (higher priority lower execution time). The same thing 
happens if I disable throttling  by software selecting "aggressive powersave" 
settings.

So I suppose that is something related directly or indirectly to frequency 
scaling or turbo boost (unfortunately I have no bios option to disable only 
turbo boost) 
Here are the results of the same test of Peter Zijlstra

time  sudo nice -n 19 lame -b 256 -V0 -h youcantdothat.wav  2&> /dev/null

real    0m1.105s
user    0m1.090s
sys     0m0.010s

time  sudo nice -n 0 lame -b 256 -V0 -h youcantdothat.wav  2&> /dev/null

real    0m1.108s
user    0m1.100s
sys     0m0.010s

time  sudo nice -n -20 lame -b 256 -V0 -h youcantdothat.wav  2&> /dev/null

real    0m1.354s
user    0m1.330s
sys     0m0.000s

They are almost the same results that I obtained before.
Looking at top there are no other process that is using the cpu  (main are 
Xorg and kopete).

I tested this on  2.6.31-17-server (ubuntu 9.10) 2.6.31-17-generic (ubuntu 
9.10) and the default kernel of fedora 12 (live cd).
As soon as possible I'll try 33-rc5 as requested by Mike Galbraith (now I 
can't shut down the computer).

Ok, a last minute update! 
I slightly modified i7z to save a log to disk. If i7z output are reliable I 
suppose that it can show is the problem:
with nice set to 19 a processor 3 reach the maximum speed,  with nice set to 
-20 its maximum value is 300-400 mhz under the maximum value. (I attach the 
two logs named as the nice level used obtained running lame on a bigger file)
Please note that logs may have some value read before and/or after the 
start/end of the running process (lame).

The next update will be the results with 33-rc5 kernel.

Luca

[-- Attachment #2: log-20 --]
[-- Type: text/plain, Size: 1449 bytes --]

	Processor 0:  1062.13 (7.99x)	   1	99.7	   0	98.5
	Processor 1:  1246.75 (9.37x)	   1	99.8	   0	  99
	Processor 2:  1113.41 (8.37x)	   1	99.8	   0	98.4
	Processor 3:  1477.30 (11.11x)	4.67	95.7	   0	85.7
	Processor 0:  1109.70 (8.34x)	   1	99.7	2.93	95.8
	Processor 1:  2316.51 (17.42x)	 5.5	  92	   0	91.7
	Processor 2:  1114.96 (8.38x)	   1	99.8	14.9	83.7
	Processor 3:  1225.48 (9.21x)	9.95	92.4	1.33	67.6
	Processor 0:  1877.15 (14.11x)	   1	99.8	29.6	69.9
	Processor 1:  1914.75 (14.40x)	   1	99.2	   1	98.1
	Processor 2:  1955.22 (14.70x)	   1	99.8	   0	99.4
	Processor 3:  2402.49 (18.06x)	38.8	41.6	   0	   0
	Processor 0:  2114.28 (15.90x)	  12	84.1	   0	35.4
	Processor 1:  1528.62 (11.49x)	   1	99.8	25.4	73.2
	Processor 2:  1582.10 (11.90x)	   1	99.9	74.2	25.2
	Processor 3:  2359.66 (17.74x)	14.6	78.3	18.6	  56
	Processor 0:  2242.78 (16.86x)	16.3	77.1	   0	   0
	Processor 1:  1591.62 (11.97x)	   1	99.9	87.9	10.9
	Processor 2:  1885.84 (14.18x)	   1	 100	99.2	   1
	Processor 3:  2035.81 (15.31x)	1.77	97.7	  15	75.4
	Processor 0:  2280.59 (17.15x)	18.3	73.8	   1	70.4
	Processor 1:  1906.72 (14.34x)	   1	99.7	26.9	  72
	Processor 2:  1778.56 (13.37x)	   1	99.9	70.9	28.6
	Processor 3:  1077.58 (8.10x)	8.56	94.2	3.66	4.79
	Processor 0:  2009.19 (15.11x)	7.07	91.1	   1	88.3
	Processor 1:  1708.96 (12.85x)	   1	99.8	37.3	61.5
	Processor 2:  1673.01 (12.58x)	   1	99.9	13.4	85.9
	Processor 3:  1421.09 (10.68x)	12.1	89.2	   1	21.8

[-- Attachment #3: log19 --]
[-- Type: text/plain, Size: 1238 bytes --]

	Processor 0:  935.30 (7.03x)	   1	99.8	   0	73.4
	Processor 1:  938.46 (7.06x)	   1	99.9	   0	98.3
	Processor 2:  932.33 (7.01x)	   1	99.9	   0	97.7
	Processor 3:  1017.69 (7.65x)	6.09	96.1	   0	92.1
	Processor 0:  1040.73 (7.83x)	   1	99.8	   0	99.1
	Processor 1:  1062.37 (7.99x)	   1	99.7	   0	96.1
	Processor 2:  1047.94 (7.88x)	   1	99.8	   1	97.7
	Processor 3:  1941.75 (14.60x)	11.1	86.5	2.98	81.5
	Processor 0:  1940.64 (14.59x)	   1	99.5	   0	72.7
	Processor 1:  1600.11 (12.03x)	   1	99.8	11.5	  86
	Processor 2:  2141.58 (16.10x)	   1	 100	99.6	   1
	Processor 3:  2760.11 (20.75x)	83.2	   0	4.33	11.6
	Processor 0:  2331.59 (17.53x)	   1	99.4	5.98	  93
	Processor 1:  2287.80 (17.20x)	   1	99.8	   0	97.6
	Processor 2:  2122.39 (15.96x)	   1	 100	88.5	11.1
	Processor 3:  2774.36 (20.86x)	99.9	   0	   0	   0
	Processor 0:  2322.64 (17.46x)	   1	99.5	   1	98.6
	Processor 1:  2275.29 (17.11x)	   1	99.8	   1	97.9
	Processor 2:  2268.43 (17.06x)	   1	99.8	   0	99.5
	Processor 3:  2779.36 (20.90x)	99.9	   0	   0	   0
	Processor 0:  2277.58 (17.12x)	   1	99.8	   0	99.6
	Processor 1:  2372.66 (17.84x)	   1	99.5	   0	99.2
	Processor 2:  2346.90 (17.65x)	   1	99.8	   0	99.5
	Processor 3:  2783.34 (20.93x)	99.9	   0	   0	   0

Re: scheduler vs hardware? (was Re: another i7 (linux) bug?)

[Chris Friesen]

On 01/22/2010 05:22 AM, Luca Zini wrote:

> with nice set to 19 a processor 3 reach the maximum speed,  with nice
set to
> -20 its maximum value is 300-400 mhz under the maximum value.

I wonder if at lower nice levels the timing on something changes,
forcing multiple cores to be active just often enough to keep the i7
from doing the turbo boost thing, whereas with higher nice levels
everything is serialized allowing the boost to be enabled.

Chris

Re: scheduler vs hardware? (was Re: another i7 (linux) bug?)

[Luca Zini]

vanilla 33-rc5 runs ok:
with nice set to -20
real    0m8.649s
user    0m8.600s
sys     0m0.030s


with nice set to 19:
real    0m8.658s
user    0m8.600s
sys     0m0.040s

thank you!
	Luca


On Friday 22 January 2010 08:19:18 am Mike Galbraith wrote:
> > if I run a cpu intensive process with the lowest priority (19
> > from man nice) I obtain much better performance that with the
> > highest priority available (-20 from man nice).
> >
> > For example the same file is processed by lame in 8.7 seconds
> > at the lowest priority, and in 12 seconds at the highest
> > priority.  Before posting a bug I wold like to understand if
> > this is a problem related to the i7 mobile (my processor is a
> > i7 Q720).
> >
> > As far as I tested on the same laptop series (dell studio 15),
> > with the same kernel this problem does not exists.
> 
> So you only see this on the i7.  That's odd.  Can you try 33-rc5?
> 
> Posting a reliable reproducer would be nice.  It'd also be nice to see
> what all is running when you see this, and where.
> 
> 	-Mike
> 

Re: scheduler vs hardware? (was Re: another i7 (linux) bug?)

[Ingo Molnar]

* Peter Zijlstra <peterz@infradead.org> wrote:

> On Fri, 2010-01-22 at 08:19 +0100, Mike Galbraith wrote:
> > > if I run a cpu intensive process with the lowest priority (19
> > > from man nice) I obtain much better performance that with the
> > > highest priority available (-20 from man nice).
> > >
> > > For example the same file is processed by lame in 8.7 seconds
> > > at the lowest priority, and in 12 seconds at the highest
> > > priority.  Before posting a bug I wold like to understand if
> > > this is a problem related to the i7 mobile (my processor is a
> > > i7 Q720).
> > > 
> > > As far as I tested on the same laptop series (dell studio 15),
> > > with the same kernel this problem does not exists.
> > 
> > So you only see this on the i7.  That's odd.  Can you try 33-rc5?
> > 
> > Posting a reliable reproducer would be nice.  It'd also be nice to see
> > what all is running when you see this, and where.
> 
> Using a sample from: http://lame.sourceforge.net/quality.php
> 
> My laptop does:
> 
> 
> # time nice -n 19 lame -b 256 -V0 -h youcantdothat.wav - > /dev/null

A 'perf stat --repeat 3' run would tell us more about the underlying reasons i 
think.

	Ingo