Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Rafael J. Wysocki]

On Friday, April 08, 2016 08:50:54 AM Mike Galbraith wrote:
> On Fri, 2016-04-08 at 08:45 +0200, Peter Zijlstra wrote:
> 
> > Cute, I thought you used governor=performance for your runs?
> 
> I do, and those numbers are with it thus set.

Well, this is a trade-off.

4.5 introduced a power regression here so this one goes back to the previous
state of things.

Thanks,
Rafael

RE: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Doug Smythies]

On 2016.04.08 14:00 Rafael J. Wysocki wrote:
> On Friday, April 08, 2016 08:50:54 AM Mike Galbraith wrote:
>> On Fri, 2016-04-08 at 08:45 +0200, Peter Zijlstra wrote:
>> 
>>> Cute, I thought you used governor=performance for your runs?
>> 
>> I do, and those numbers are with it thus set.

> Well, this is a trade-off.
>
> 4.5 introduced a power regression here so this one goes back to the previous
> state of things.

Mike:

Could you send me, or point me to, the program "pipe-test"?
So far, I have only found one, but it is both old and not
the same program you are running (based on print statements).

I realize I might not be to recreate your problem scenario anyhow,
I just want to try.

... Doug

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Mike Galbraith]

On Fri, 2016-04-08 at 15:19 -0700, Doug Smythies wrote:

> Could you send me, or point me to, the program "pipe-test"?
> So far, I have only found one, but it is both old and not
> the same program you are running (based on print statements).

It's the same old pipe-test, just bent up a little to suit my usage.

	-Mike

RE: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Doug Smythies]

On 2016.04.08 15:19 Doug Smythies wrote:
> On 2016.04.08 14:00 Rafael J. Wysocki wrote:
>> On Friday, April 08, 2016 08:50:54 AM Mike Galbraith wrote:
>>> On Fri, 2016-04-08 at 08:45 +0200, Peter Zijlstra wrote:
>>> 
>>>> Cute, I thought you used governor=performance for your runs?
>>> 
>>> I do, and those numbers are with it thus set.

>> Well, this is a trade-off.
>>
>> 4.5 introduced a power regression here so this one goes back to the previous
>> state of things.

> Mike:
>
> Could you send me, or point me to, the program "pipe-test"?
> So far, I have only found one, but it is both old and not
> the same program you are running (based on print statements).
>
> I realize I might not be to recreate your problem scenario anyhow,
> I just want to try.

I still didn't find the exact same program, but I think I found some
earlier version of the correct test.

I get (long term average):
Kernel 4.4.0-17: Powersave 3.93 usecs/loop ; Performance 3.93 usecs/loop 0.89
Kernel 4.5-rc7: Powersave 3.47 usecs/loop ; Performance 3.51 usecs/loop  1.00
Kernel 4.6-rc1: Powersave 3.84 usecs/loop ; Performance 3.88 usecs/loop  0.90

So, similar results (so far, I didn't try reverted yet).

... Doug

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Mike Galbraith]

On Sat, 2016-04-09 at 00:17 -0700, Doug Smythies wrote:

> I still didn't find the exact same program, but I think I found some
> earlier version of the correct test.
> 
> I get (long term average):
> Kernel 4.4.0-17: Powersave 3.93 usecs/loop ; Performance 3.93 usecs/loop 0.89
> Kernel 4.5-rc7: Powersave 3.47 usecs/loop ; Performance 3.51 usecs/loop  1.00
> Kernel 4.6-rc1: Powersave 3.84 usecs/loop ; Performance 3.88 usecs/loop  0.90
> 
> So, similar results (so far, I didn't try reverted yet).

I likely see a bit more go missing because I throttle no_hz when idle
is being hammered at high frequency.

	-Mike

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Peter Zijlstra]

On Fri, Apr 08, 2016 at 03:19:14PM -0700, Doug Smythies wrote:
> Could you send me, or point me to, the program "pipe-test"?
> So far, I have only found one, but it is both old and not
> the same program you are running (based on print statements).

The latest public one lives as: perf bench sched pipe

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Mike Galbraith]

On Fri, 2016-04-08 at 22:59 +0200, Rafael J. Wysocki wrote:
> On Friday, April 08, 2016 08:50:54 AM Mike Galbraith wrote:
> > On Fri, 2016-04-08 at 08:45 +0200, Peter Zijlstra wrote:
> > 
> > > Cute, I thought you used governor=performance for your runs?
> > 
> > I do, and those numbers are with it thus set.
> 
> Well, this is a trade-off.
> 
> 4.5 introduced a power regression here so this one goes back to the previous
> state of things.

That sounds somewhat reasonable.  Too bad I don't have a super duper
watt meter handy.. seeing that you really really are saving me money
would perhaps make me less fond of those prettier numbers.

	-Mike

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Rafael J. Wysocki]

On Sat, Apr 9, 2016 at 8:40 AM, Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> On Fri, 2016-04-08 at 22:59 +0200, Rafael J. Wysocki wrote:
>> On Friday, April 08, 2016 08:50:54 AM Mike Galbraith wrote:
>> > On Fri, 2016-04-08 at 08:45 +0200, Peter Zijlstra wrote:
>> >
>> > > Cute, I thought you used governor=performance for your runs?
>> >
>> > I do, and those numbers are with it thus set.
>>
>> Well, this is a trade-off.
>>
>> 4.5 introduced a power regression here so this one goes back to the previous
>> state of things.
>
> That sounds somewhat reasonable.  Too bad I don't have a super duper
> watt meter handy.. seeing that you really really are saving me money
> would perhaps make me less fond of those prettier numbers.

You can look at the turbostat Watts numbers ("turbostat --debug" and
the last three columns of the output in turbostat as included in the
kernel source).

That requires an Intel CPU with RAPL.

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Mike Galbraith]

On Sat, 2016-04-09 at 14:33 +0200, Rafael J. Wysocki wrote:
> On Sat, Apr 9, 2016 at 8:40 AM, Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> > On Fri, 2016-04-08 at 22:59 +0200, Rafael J. Wysocki wrote:
> > > On Friday, April 08, 2016 08:50:54 AM Mike Galbraith wrote:
> > > > On Fri, 2016-04-08 at 08:45 +0200, Peter Zijlstra wrote:
> > > > 
> > > > > Cute, I thought you used governor=performance for your runs?
> > > > 
> > > > I do, and those numbers are with it thus set.
> > > 
> > > Well, this is a trade-off.
> > > 
> > > 4.5 introduced a power regression here so this one goes back to the previous
> > > state of things.
> > 
> > That sounds somewhat reasonable.  Too bad I don't have a super duper
> > watt meter handy.. seeing that you really really are saving me money
> > would perhaps make me less fond of those prettier numbers.
> 
> You can look at the turbostat Watts numbers ("turbostat --debug" and
> the last three columns of the output in turbostat as included in the
> kernel source).

Hm.  I think I want my prettier numbers back.

714KHz/877KHz = 0.81
25Watt/30Watt = 0.83

	-Mike

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Mike Galbraith]

Hm, setting gov=performance, and taking the average of 3 30 second
interval PkgWatt samples as pipe-test runs..

714KHz/28.03Ws = 25.46
877KHz/30.28Ws = 28.96

..for pipe-test, the tradeoff look a bit more like red than green.

	-Mike

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Rafael J. Wysocki]

On Sat, Apr 9, 2016 at 6:39 PM, Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
>
> Hm, setting gov=performance, and taking the average of 3 30 second
> interval PkgWatt samples as pipe-test runs..
>
> 714KHz/28.03Ws = 25.46
> 877KHz/30.28Ws = 28.96
>
> ..for pipe-test, the tradeoff look a bit more like red than green.

Well, fair enough, but that's just pipe-test, and what about the
people who don't see the performance gain and see the energy loss,
like Doug?

Essentially, this trades performance gains in somewhat special
workloads for increased energy consumption in idle.  Those workloads
need not be run by everybody, but idle is.

That said I applied the patch you're complaining about mostly because
the commit that introduced the change in question in 4.5 claimed that
it wouldn't affect idle power on systems with reasonably fast C1, but
that didn't pass the reality test.  I'm not totally against restoring
that change, but it would need to be based on very solid evidence.

RE: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Doug Smythies]

On 2106.04.09 20:45 Rafael J. Wysocki wrote:
>On Sat, Apr 9, 2016 at 6:39 PM, Mike Galbraith wrote:
>>
>> Hm, setting gov=performance, and taking the average of 3 30 second
>> interval PkgWatt samples as pipe-test runs..
>>
>> 714KHz/28.03Ws = 25.46
>> 877KHz/30.28Ws = 28.96
>>
>> ..for pipe-test, the tradeoff look a bit more like red than green.
>
> Well, fair enough, but that's just pipe-test, and what about the
> people who don't see the performance gain and see the energy loss,
> like Doug?

Some numbers from my computer:

Pipe-test (100 seconds):

Kernel 4.6-rc2 gov=powersave:
Stock: 3.86 uSecs/loop and 3148.05 Joules
Reverted: 3.34 uSecs/loop and 3567.43 Joules

Reverted is 13% faster at a cost of 13% more energy.

Idle stats (done separately and for 20e6 loops)

State	k46rc2-ps (sec)	k46rc2-rev-ps(sec)
0.00	0.01			4.09
1.00	38.68			0.00
2.00	0.46			0.27
3.00	0.01			0.00
4.00	464.23		380.23
		
total	503.38		384.60

Kernel 4.6-rc2 gov=performance:
Stock: 3.89 uSecs/loop and 3154.72 Joules
Reverted: 3.25 uSecs/loop and 3445.90 Joules

Reverted is 16% faster at a cost of 9% more energy.

Idle stats (done separately and for 20e6 loops)

State	k46rc2-pf (sec)	k46rc2-rev-pf (sec)
0.00	0.00			1.43
1.00	38.89			0.04
2.00	2.08			0.03
3.00	0.01			0.01
4.00	463.05		381.54
		
total	504.03		383.05

9 incremental kernel compiles, with no changes:
(the reference test from last cycle):
(2000 seconds turbostat package energy sample time):
There is no detectable consistent change in compile times:

Kernel 4.6-rc2 gov=powersave:
Stock: 48557 Joules
Reverted: 65439 Joules

Reverted costs 34% more energy.
(note: this result is unusually high. There are variations test to test)

Kernel 4.6-rc2 gov=performance:
Stock: 49965 Joules
Reverted: 59232 Joules

Reverted costs 19% more energy.
(note: never tested gov=performance before)

Idle stats not re-done (we had several samples last cycle).

> Essentially, this trades performance gains in somewhat special
> workloads for increased energy consumption in idle.  Those workloads
> need not be run by everybody, but idle is.
>
> That said I applied the patch you're complaining about mostly because
> the commit that introduced the change in question in 4.5 claimed that
> it wouldn't affect idle power on systems with reasonably fast C1, but
> that didn't pass the reality test.  I'm not totally against restoring
> that change, but it would need to be based on very solid evidence.

... Doug

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Mike Galbraith]

[-- Attachment #1: Type: text/plain, Size: 2099 bytes --]

On Sun, 2016-04-10 at 05:44 +0200, Rafael J. Wysocki wrote:
> On Sat, Apr 9, 2016 at 6:39 PM, Mike Galbraith <
> umgwanakikbuti@gmail.com> wrote:
> > 
> > Hm, setting gov=performance, and taking the average of 3 30 second
> > interval PkgWatt samples as pipe-test runs..
> > 
> > 714KHz/28.03Ws = 25.46
> > 877KHz/30.28Ws = 28.96
> > 
> > ..for pipe-test, the tradeoff look a bit more like red than green.
> 
> Well, fair enough, but that's just pipe-test, and what about the
> people who don't see the performance gain and see the energy loss,
> like Doug?

Perhaps Doug sees increased power because he's not throttling no_hz,
whereas I am, so he burns more power getting _to_ idle?  Dunno, maybe
he'll try the attached.  If it's a general case energy loser, so be it,
numbers talk, bs walks and all that ;-)

> Essentially, this trades performance gains in somewhat special
> workloads for increased energy consumption in idle.  Those workloads
> need not be run by everybody, but idle is.

Cross core scheduling is routine business, we do truckloads of that for
good reason, and lots of stuff does wakeups at high frequency.

> That said I applied the patch you're complaining about mostly because
> the commit that introduced the change in question in 4.5 claimed that
> it wouldn't affect idle power on systems with reasonably fast C1, but
> that didn't pass the reality test.  I'm not totally against restoring
> that change, but it would need to be based on very solid evidence.

Understood.  My box seems to be saying we can hug the trees hardest by
telling the CPU get work done as quickly as possible, but I don't have
much experience at tree hugging measurement.  Performance wise, tasks
talking via localhost is definitely not special.

tbench     1      2      4     8
base     752   1283   2250  3362

select_idle_sibling() off
         735   1344   2080  2884
delta   .977  1.047   .924  .857

select_idle_sibling() on, 0c313cb20732 reverted
         816   1317   2240  3388
delta  1.085  1.026   .995 1.007 vs base
delta  1.110   .979  1.076 1.174 vs off
               (^hm)

	-Mike

[-- Attachment #2: sched-throttle-nohz.patch --]
[-- Type: text/x-patch, Size: 1535 bytes --]

sched: ratelimit nohz

Entering nohz code on every micro-idle is too expensive to bear.

Signed-off-by: Mike Galbraith <efault@gmx.de>
---
 include/linux/sched.h    |    5 +++++
 kernel/sched/core.c      |    8 ++++++++
 kernel/time/tick-sched.c |    2 +-
 3 files changed, 14 insertions(+), 1 deletion(-)

--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -2286,6 +2286,11 @@ static inline int set_cpus_allowed_ptr(s
 #ifdef CONFIG_NO_HZ_COMMON
 void calc_load_enter_idle(void);
 void calc_load_exit_idle(void);
+#ifdef CONFIG_SMP
+extern int sched_needs_cpu(int cpu);
+#else
+static inline int sched_needs_cpu(int cpu) { return 0; }
+#endif
 #else
 static inline void calc_load_enter_idle(void) { }
 static inline void calc_load_exit_idle(void) { }
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -577,6 +577,14 @@ static inline bool got_nohz_idle_kick(vo
 	return false;
 }
 
+int sched_needs_cpu(int cpu)
+{
+	if (tick_nohz_full_cpu(cpu))
+		return 0;
+
+	return  cpu_rq(cpu)->avg_idle < sysctl_sched_migration_cost;
+}
+
 #else /* CONFIG_NO_HZ_COMMON */
 
 static inline bool got_nohz_idle_kick(void)
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -676,7 +676,7 @@ static ktime_t tick_nohz_stop_sched_tick
 	} while (read_seqretry(&jiffies_lock, seq));
 	ts->last_jiffies = basejiff;
 
-	if (rcu_needs_cpu(basemono, &next_rcu) ||
+	if (sched_needs_cpu(cpu) || rcu_needs_cpu(basemono, &next_rcu) ||
 	    arch_needs_cpu() || irq_work_needs_cpu()) {
 		next_tick = basemono + TICK_NSEC;
 	} else {

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Mike Galbraith]

On Sun, 2016-04-10 at 11:35 +0200, Mike Galbraith wrote:
> On Sun, 2016-04-10 at 05:44 +0200, Rafael J. Wysocki wrote:
> > On Sat, Apr 9, 2016 at 6:39 PM, Mike Galbraith <
> > umgwanakikbuti@gmail.com> wrote:
> > > 
> > > Hm, setting gov=performance, and taking the average of 3 30 second
> > > interval PkgWatt samples as pipe-test runs..
> > > 
> > > 714KHz/28.03Ws = 25.46
> > > 877KHz/30.28Ws = 28.96
> > > 
> > > ..for pipe-test, the tradeoff look a bit more like red than green.
> > 
> > Well, fair enough, but that's just pipe-test, and what about the
> > people who don't see the performance gain and see the energy loss,
> > like Doug?
> 
> Perhaps Doug sees increased power because he's not throttling no_hz,
> whereas I am, so he burns more power getting _to_ idle?  Dunno, maybe
> he'll try the attached.  If it's a general case energy loser, so be it,
> numbers talk, bs walks and all that ;-)

And here are the rest of my numbers..

> tbench     1      2      4     8
> base     752   1283   2250  3362
> 
> select_idle_sibling() off
>          735   1344   2080  2884
> delta   .977  1.047   .924  .857
> 
> select_idle_sibling() on, 0c313cb20732 reverted
>          816   1317   2240  3388
> delta  1.085  1.026   .995 1.007 vs base
> delta  1.110   .979  1.076 1.174 vs off
>                (^hm)

tbench 2 turboboost off
base          1215  1.00   1215/32.24=37.68
revert        1252  1.03   1252/35.82=34.95=loser

tbench 2 throughput hm is apparently a turboboost oddity, and..

tbench (turboboost back on)
power      1      2      4     8
base   23.88  37.41  54.64 62.25
revert 31.25  42.53  55.11 62.66

MB/s/Ws    1      2      4     8
base   31.49  34.29  41.17 54.00
revert 26.11  30.96  40.64 54.06

..while single pipe-test pair said green/green, tbench numbers say
throughput green, but energy efficiency red across the board.

	-Mike

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Peter Zijlstra]

On Fri, Apr 08, 2016 at 10:59:59PM +0200, Rafael J. Wysocki wrote:
> On Friday, April 08, 2016 08:50:54 AM Mike Galbraith wrote:
> > On Fri, 2016-04-08 at 08:45 +0200, Peter Zijlstra wrote:
> > 
> > > Cute, I thought you used governor=performance for your runs?
> > 
> > I do, and those numbers are with it thus set.
> 
> Well, this is a trade-off.
> 
> 4.5 introduced a power regression here so this one goes back to the previous
> state of things.

Just for my elucidation; how can gov=performance have a 'power'
regression?

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Rafael J. Wysocki]

On Sat, Apr 9, 2016 at 1:07 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Fri, Apr 08, 2016 at 10:59:59PM +0200, Rafael J. Wysocki wrote:
>> On Friday, April 08, 2016 08:50:54 AM Mike Galbraith wrote:
>> > On Fri, 2016-04-08 at 08:45 +0200, Peter Zijlstra wrote:
>> >
>> > > Cute, I thought you used governor=performance for your runs?
>> >
>> > I do, and those numbers are with it thus set.
>>
>> Well, this is a trade-off.
>>
>> 4.5 introduced a power regression here so this one goes back to the previous
>> state of things.
>
> Just for my elucidation; how can gov=performance have a 'power'
> regression?

Because of what is used as the "default" idle state most of the time.

C1 was used before 4.5 and that changed to polling in 4.5.

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Mike Galbraith]

On Sat, 2016-04-09 at 14:31 +0200, Rafael J. Wysocki wrote:
> On Sat, Apr 9, 2016 at 1:07 PM, Peter Zijlstra <peterz@infradead.org>
> wrote:
> > On Fri, Apr 08, 2016 at 10:59:59PM +0200, Rafael J. Wysocki wrote:
> > > On Friday, April 08, 2016 08:50:54 AM Mike Galbraith wrote:
> > > > On Fri, 2016-04-08 at 08:45 +0200, Peter Zijlstra wrote:
> > > > 
> > > > > Cute, I thought you used governor=performance for your runs?
> > > > 
> > > > I do, and those numbers are with it thus set.
> > > 
> > > Well, this is a trade-off.
> > > 
> > > 4.5 introduced a power regression here so this one goes back to
> > > the previous
> > > state of things.
> > 
> > Just for my elucidation; how can gov=performance have a 'power'
> > regression?
> 
> Because of what is used as the "default" idle state most of the time.
> 
> C1 was used before 4.5 and that changed to polling in 4.5.

Should the default idle state not then be governor dependent?  When I
set gov=performance, I'm expecting box to go just as fast as it can go
without melting.  Does polling risk CPU -> lava conversion?

	-Mike

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Rik van Riel]

[-- Attachment #1: Type: text/plain, Size: 1591 bytes --]

On Sun, 2016-04-10 at 17:39 +0200, Mike Galbraith wrote:
> On Sat, 2016-04-09 at 14:31 +0200, Rafael J. Wysocki wrote:
> > 
> > On Sat, Apr 9, 2016 at 1:07 PM, Peter Zijlstra <peterz@infradead.or
> > g>
> > wrote:
> > > 
> > > On Fri, Apr 08, 2016 at 10:59:59PM +0200, Rafael J. Wysocki
> > > wrote:
> > > > 
> > > > On Friday, April 08, 2016 08:50:54 AM Mike Galbraith wrote:
> > > > > 
> > > > > On Fri, 2016-04-08 at 08:45 +0200, Peter Zijlstra wrote:
> > > > > 
> > > > > > 
> > > > > > Cute, I thought you used governor=performance for your
> > > > > > runs?
> > > > > I do, and those numbers are with it thus set.
> > > > Well, this is a trade-off.
> > > > 
> > > > 4.5 introduced a power regression here so this one goes back to
> > > > the previous
> > > > state of things.
> > > Just for my elucidation; how can gov=performance have a 'power'
> > > regression?
> > Because of what is used as the "default" idle state most of the
> > time.
> > 
> > C1 was used before 4.5 and that changed to polling in 4.5.
> Should the default idle state not then be governor dependent?  When I
> set gov=performance, I'm expecting box to go just as fast as it can
> go
> without melting.  Does polling risk CPU -> lava conversion?

Current CPUs can only have some cores run at full speed
(turbo mode) if other cores are idling and/or running at
lower speeds.

It may be time to stop pretending that gov=performance
actually results in better performance on current CPUs,
since it may inhibit entire levels of turbo mode.

-- 
All Rights Reversed.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Mike Galbraith]

On Sun, 2016-04-10 at 16:24 -0400, Rik van Riel wrote:
> On Sun, 2016-04-10 at 17:39 +0200, Mike Galbraith wrote:

> > Should the default idle state not then be governor dependent?  When I
> > set gov=performance, I'm expecting box to go just as fast as it can
> > go
> > without melting.  Does polling risk CPU -> lava conversion?
> 
> Current CPUs can only have some cores run at full speed
> (turbo mode) if other cores are idling and/or running at
> lower speeds.

The real world is very unlikely to miss the prettier numbers I'm
grieving over one tiny bit.  Knowing that doesn't make giving them up
any easier though.. byebye cycles (sniff) ;-)

	-Mike

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Rik van Riel]

[-- Attachment #1: Type: text/plain, Size: 1537 bytes --]

On Mon, 2016-04-11 at 05:04 +0200, Mike Galbraith wrote:
> On Sun, 2016-04-10 at 16:24 -0400, Rik van Riel wrote:
> > 
> > On Sun, 2016-04-10 at 17:39 +0200, Mike Galbraith wrote:
> > 
> > > 
> > > Should the default idle state not then be governor
> > > dependent?  When I
> > > set gov=performance, I'm expecting box to go just as fast as it
> > > can
> > > go
> > > without melting.  Does polling risk CPU -> lava conversion?
> > Current CPUs can only have some cores run at full speed
> > (turbo mode) if other cores are idling and/or running at
> > lower speeds.
> The real world is very unlikely to miss the prettier numbers I'm
> grieving over one tiny bit.  Knowing that doesn't make giving them up
> any easier though.. byebye cycles (sniff) ;-)

I suspect your pipe benchmark could be very relevant to
network performance numbers, too.

I would like to go into polling a little bit more aggressively
in a future kernel, and I think we can get away with it if we
teach the polling loop to exit after we have spent enough time
there that the menu governor will pick HLT after a few timed
out poll loops.

That way while we run a workload that actually benefits from
polling, we will get polling, but once we run a workload that
actually sleeps longer than the HLT threshold, we will quickly
fall back to HLT.

With 10Gbps network traffic, it could make a real difference
whether or not the CPU can wake up immediately, or takes a
microsecond to wake up...

-- 
All Rights Reversed.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Rafael J. Wysocki]

On Mon, Apr 11, 2016 at 2:38 PM, Rik van Riel <riel@redhat.com> wrote:
> On Mon, 2016-04-11 at 05:04 +0200, Mike Galbraith wrote:
>> On Sun, 2016-04-10 at 16:24 -0400, Rik van Riel wrote:
>> >
>> > On Sun, 2016-04-10 at 17:39 +0200, Mike Galbraith wrote:
>> >
>> > >
>> > > Should the default idle state not then be governor
>> > > dependent?  When I
>> > > set gov=performance, I'm expecting box to go just as fast as it
>> > > can
>> > > go
>> > > without melting.  Does polling risk CPU -> lava conversion?
>> > Current CPUs can only have some cores run at full speed
>> > (turbo mode) if other cores are idling and/or running at
>> > lower speeds.
>> The real world is very unlikely to miss the prettier numbers I'm
>> grieving over one tiny bit.  Knowing that doesn't make giving them up
>> any easier though.. byebye cycles (sniff) ;-)
>
> I suspect your pipe benchmark could be very relevant to
> network performance numbers, too.
>
> I would like to go into polling a little bit more aggressively
> in a future kernel,

Agreed, but ->

> and I think we can get away with it if we
> teach the polling loop to exit after we have spent enough time
> there that the menu governor will pick HLT after a few timed
> out poll loops.

-> my concern about this approach is that it would add an artificial
point to the menu governor statistics at whatever the timeout is
chosen to be.

Re: [regression] cross core scheduling frequency drop bisected to 0c313cb20732

[Rik van Riel]

[-- Attachment #1: Type: text/plain, Size: 2070 bytes --]

On Mon, 2016-04-11 at 15:21 +0200, Rafael J. Wysocki wrote:
> On Mon, Apr 11, 2016 at 2:38 PM, Rik van Riel <riel@redhat.com>
> wrote:
> > 
> > On Mon, 2016-04-11 at 05:04 +0200, Mike Galbraith wrote:
> > > 
> > > On Sun, 2016-04-10 at 16:24 -0400, Rik van Riel wrote:
> > > > 
> > > > 
> > > > On Sun, 2016-04-10 at 17:39 +0200, Mike Galbraith wrote:
> > > > 
> > > > > 
> > > > > 
> > > > > Should the default idle state not then be governor
> > > > > dependent?  When I
> > > > > set gov=performance, I'm expecting box to go just as fast as
> > > > > it
> > > > > can
> > > > > go
> > > > > without melting.  Does polling risk CPU -> lava conversion?
> > > > Current CPUs can only have some cores run at full speed
> > > > (turbo mode) if other cores are idling and/or running at
> > > > lower speeds.
> > > The real world is very unlikely to miss the prettier numbers I'm
> > > grieving over one tiny bit.  Knowing that doesn't make giving
> > > them up
> > > any easier though.. byebye cycles (sniff) ;-)
> > I suspect your pipe benchmark could be very relevant to
> > network performance numbers, too.
> > 
> > I would like to go into polling a little bit more aggressively
> > in a future kernel,
> Agreed, but ->
> 
> > 
> > and I think we can get away with it if we
> > teach the polling loop to exit after we have spent enough time
> > there that the menu governor will pick HLT after a few timed
> > out poll loops.
> -> my concern about this approach is that it would add an artificial
> point to the menu governor statistics at whatever the timeout is
> chosen to be.

I would set the threshold to at least the HLT target residency +
exit latency, so 3 poll timeouts in 8 wakeups would cause
us to fall back to HLT.

On the other hand, if the system is legitimately very busy,
and we break out of the HLT loop due to activities happening
before the timeout most of the time, we should automatically
pick polling.

Does that make sense?

Am I overlooking something?

-- 
All Rights Reversed.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]