measuring and/or dealing with "idleness" and variable frequency

measuring and/or dealing with "idleness" and variable frequency

[Rick Jones]

I am coming to "perf" (record -e cycles and report) from an old-time 
background where one could see the "idle routine" in a profile and know 
how "idle" (in terms of classic reporting a la top) a CPU was.  Back 
then the frequency was fixed, the idle loop was always "running" when 
the CPU was idle and there were no hardware threads.  Life was simple 
and good.

Now I have to realign to the present and am wondering where/how to get 
started.  I've looked at https://perf.wiki.kernel.org/index.php/Tutorial

which talks about:

> By default, perf record uses the cycles event as the sampling event.
> This is a generic hardware event that is mapped to a
> hardware-specific PMU event by the kernel. For Intel, it is mapped to
> UNHALTED_CORE_CYCLES. This event does not maintain a constant
> correlation to time in the presence of CPU frequency scaling. Intel
> provides another event, called UNHALTED_REFERENCE_CYCLES but this
> event is NOT currently available with perf_events.
>
> On AMD systems, the event is mapped to CPU_CLK_UNHALTED and this
> event is also subject to frequency scaling. On any Intel or AMD
> processor, the cycle event does not count when the processor is idle,
> i.e., when it calls mwait().

And I am wondering if those are all still true.

I am running a 3.5.0 kernel in an Ubuntu environment (3.5.0-23-generic 
#35~precise1-Ubuntu SMP), and among the things I wish to do is sanity 
check the Idle % being reported by top on one system which is passing 
traffic back and forth between two others.  Compared to the CPU 
utilization reported by top on those two systems, the reports for this 
one look extraordinarily low.

The CPU involved here is:

Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz

I am also interested in seeing how the CPU consumption changes as I 
tweak the setup of the middle box, does it go more or less idle which 
routines have their consumption change etc etc.  and I am wondering if I 
can get that all with just perf or if I have to correlate things between 
multiple tools.

thanks, and happy benchmarking,

rick jones

Re: measuring and/or dealing with "idleness" and variable frequency

[David Ahern]

On 6/6/13 11:46 AM, Rick Jones wrote:
> I am coming to "perf" (record -e cycles and report) from an old-time
> background where one could see the "idle routine" in a profile and know
> how "idle" (in terms of classic reporting a la top) a CPU was.  Back
> then the frequency was fixed, the idle loop was always "running" when
> the CPU was idle and there were no hardware threads.  Life was simple
> and good.

You could use the scheduling events or context-switches to know when a 
CPU is idle. Idle time for a CPU is one of the stats my perf-timehist 
command shows. I just dumped it to LKML:
    https://lkml.org/lkml/2013/6/7/656

It's an RFC from an inclusion upstream but it has been used for over 2 
years.

David


>
> Now I have to realign to the present and am wondering where/how to get
> started.  I've looked at https://perf.wiki.kernel.org/index.php/Tutorial
>
> which talks about:
>
>> By default, perf record uses the cycles event as the sampling event.
>> This is a generic hardware event that is mapped to a
>> hardware-specific PMU event by the kernel. For Intel, it is mapped to
>> UNHALTED_CORE_CYCLES. This event does not maintain a constant
>> correlation to time in the presence of CPU frequency scaling. Intel
>> provides another event, called UNHALTED_REFERENCE_CYCLES but this
>> event is NOT currently available with perf_events.
>>
>> On AMD systems, the event is mapped to CPU_CLK_UNHALTED and this
>> event is also subject to frequency scaling. On any Intel or AMD
>> processor, the cycle event does not count when the processor is idle,
>> i.e., when it calls mwait().
>
> And I am wondering if those are all still true.
>
> I am running a 3.5.0 kernel in an Ubuntu environment (3.5.0-23-generic
> #35~precise1-Ubuntu SMP), and among the things I wish to do is sanity
> check the Idle % being reported by top on one system which is passing
> traffic back and forth between two others.  Compared to the CPU
> utilization reported by top on those two systems, the reports for this
> one look extraordinarily low.
>
> The CPU involved here is:
>
> Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
>
> I am also interested in seeing how the CPU consumption changes as I
> tweak the setup of the middle box, does it go more or less idle which
> routines have their consumption change etc etc.  and I am wondering if I
> can get that all with just perf or if I have to correlate things between
> multiple tools.
>
> thanks, and happy benchmarking,
>
> rick jones
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-perf-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: measuring and/or dealing with "idleness" and variable frequency

[Rick Jones]

On 06/08/2013 08:21 PM, David Ahern wrote:
> On 6/6/13 11:46 AM, Rick Jones wrote:
>> I am coming to "perf" (record -e cycles and report) from an old-time
>> background where one could see the "idle routine" in a profile and know
>> how "idle" (in terms of classic reporting a la top) a CPU was.  Back
>> then the frequency was fixed, the idle loop was always "running" when
>> the CPU was idle and there were no hardware threads.  Life was simple
>> and good.
>
> You could use the scheduling events or context-switches to know when a
> CPU is idle. Idle time for a CPU is one of the stats my perf-timehist
> command shows. I just dumped it to LKML:
>     https://lkml.org/lkml/2013/6/7/656
>
> It's an RFC from an inclusion upstream but it has been used for over 2
> years.

I think I could get what I want from that - assuming I could get that to 
a kernel I can use in my test env, but am guessing it is rather more 
than I"m looking for at the moment.  I'm wondering if there might be a 
simpler way to go - just get the idle loop to keep looping rather than 
halt (?) the thread, so there would still be cycle events in the PMU? 
(And perhaps set the system to a static, high-performance mode to avoid 
frequency changes and maybe even disable HT?)

rick

>
> David
>
>
>>
>> Now I have to realign to the present and am wondering where/how to get
>> started.  I've looked at https://perf.wiki.kernel.org/index.php/Tutorial
>>
>> which talks about:
>>
>>> By default, perf record uses the cycles event as the sampling event.
>>> This is a generic hardware event that is mapped to a
>>> hardware-specific PMU event by the kernel. For Intel, it is mapped to
>>> UNHALTED_CORE_CYCLES. This event does not maintain a constant
>>> correlation to time in the presence of CPU frequency scaling. Intel
>>> provides another event, called UNHALTED_REFERENCE_CYCLES but this
>>> event is NOT currently available with perf_events.
>>>
>>> On AMD systems, the event is mapped to CPU_CLK_UNHALTED and this
>>> event is also subject to frequency scaling. On any Intel or AMD
>>> processor, the cycle event does not count when the processor is idle,
>>> i.e., when it calls mwait().
>>
>> And I am wondering if those are all still true.
>>
>> I am running a 3.5.0 kernel in an Ubuntu environment (3.5.0-23-generic
>> #35~precise1-Ubuntu SMP), and among the things I wish to do is sanity
>> check the Idle % being reported by top on one system which is passing
>> traffic back and forth between two others.  Compared to the CPU
>> utilization reported by top on those two systems, the reports for this
>> one look extraordinarily low.
>>
>> The CPU involved here is:
>>
>> Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
>>
>> I am also interested in seeing how the CPU consumption changes as I
>> tweak the setup of the middle box, does it go more or less idle which
>> routines have their consumption change etc etc.  and I am wondering if I
>> can get that all with just perf or if I have to correlate things between
>> multiple tools.
>>
>> thanks, and happy benchmarking,
>>
>> rick jones
>> --
>> To unsubscribe from this list: send the line "unsubscribe
>> linux-perf-users" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: measuring and/or dealing with "idleness" and variable frequency

[Rick Jones]

On 06/10/2013 01:34 PM, Rick Jones wrote:
> On 06/08/2013 08:21 PM, David Ahern wrote:
>> On 6/6/13 11:46 AM, Rick Jones wrote:
>>> I am coming to "perf" (record -e cycles and report) from an old-time
>>> background where one could see the "idle routine" in a profile and know
>>> how "idle" (in terms of classic reporting a la top) a CPU was.  Back
>>> then the frequency was fixed, the idle loop was always "running" when
>>> the CPU was idle and there were no hardware threads.  Life was simple
>>> and good.
>>
>> You could use the scheduling events or context-switches to know when a
>> CPU is idle. Idle time for a CPU is one of the stats my perf-timehist
>> command shows. I just dumped it to LKML:
>>     https://lkml.org/lkml/2013/6/7/656
>>
>> It's an RFC from an inclusion upstream but it has been used for over 2
>> years.
>
> I think I could get what I want from that - assuming I could get that to
> a kernel I can use in my test env, but am guessing it is rather more
> than I"m looking for at the moment.  I'm wondering if there might be a
> simpler way to go - just get the idle loop to keep looping rather than
> halt (?) the thread, so there would still be cycle events in the PMU?
> (And perhaps set the system to a static, high-performance mode to avoid
> frequency changes and maybe even disable HT?)

Or, after having found 
https://www.kernel.org/doc/Documentation/kernel-parameters.txt, perhaps 
idle=poll?

rick jones

Re: measuring and/or dealing with "idleness" and variable frequency

[David Ahern]

On 6/10/13 4:28 PM, Rick Jones wrote:
>> I think I could get what I want from that - assuming I could get that to
>> a kernel I can use in my test env, but am guessing it is rather more
>> than I"m looking for at the moment.  I'm wondering if there might be a
>> simpler way to go - just get the idle loop to keep looping rather than
>> halt (?) the thread, so there would still be cycle events in the PMU?
>> (And perhaps set the system to a static, high-performance mode to avoid
>> frequency changes and maybe even disable HT?)
>
> Or, after having found
> https://www.kernel.org/doc/Documentation/kernel-parameters.txt, perhaps
> idle=poll?

Yes, that should turn the server into a power-sucking space heater and 
keep the cycles counter firing.

David

Re: measuring and/or dealing with "idleness" and variable frequency

[Rick Jones]

On 06/10/2013 03:33 PM, David Ahern wrote:
> On 6/10/13 4:28 PM, Rick Jones wrote:
>>> I think I could get what I want from that - assuming I could get that to
>>> a kernel I can use in my test env, but am guessing it is rather more
>>> than I"m looking for at the moment.  I'm wondering if there might be a
>>> simpler way to go - just get the idle loop to keep looping rather than
>>> halt (?) the thread, so there would still be cycle events in the PMU?
>>> (And perhaps set the system to a static, high-performance mode to avoid
>>> frequency changes and maybe even disable HT?)
>>
>> Or, after having found
>> https://www.kernel.org/doc/Documentation/kernel-parameters.txt, perhaps
>> idle=poll?
>
> Yes, that should turn the server into a power-sucking space heater and
> keep the cycles counter firing.

Excellent - I can live with the server being a power-sucking space 
heater for the duration of my profiling.  Heck, many of the servers I 
used to work on were always power-sucking space heaters :)

happy benchmarking,

rick jones

Interestingly enough, on the old Centrino-based laptop where I've tried 
it (on AC power), read_hpet has gone away completely.  I'd noticed that 
was visible in several examples in the tutorial on the wiki, but it 
wasn't explained.

Re: measuring and/or dealing with "idleness" and variable frequency

[Rick Jones]

On 06/10/2013 03:39 PM, Rick Jones wrote:
> On 06/10/2013 03:33 PM, David Ahern wrote:
>> On 6/10/13 4:28 PM, Rick Jones wrote:
>>>> I think I could get what I want from that - assuming I could get
>>>> that to
>>>> a kernel I can use in my test env, but am guessing it is rather more
>>>> than I"m looking for at the moment.  I'm wondering if there might be a
>>>> simpler way to go - just get the idle loop to keep looping rather than
>>>> halt (?) the thread, so there would still be cycle events in the PMU?
>>>> (And perhaps set the system to a static, high-performance mode to avoid
>>>> frequency changes and maybe even disable HT?)
>>>
>>> Or, after having found
>>> https://www.kernel.org/doc/Documentation/kernel-parameters.txt, perhaps
>>> idle=poll?
>>
>> Yes, that should turn the server into a power-sucking space heater and
>> keep the cycles counter firing.
>
> Excellent - I can live with the server being a power-sucking space
> heater for the duration of my profiling.  Heck, many of the servers I
> used to work on were always power-sucking space heaters :)

Just for the archives and anyone looking to go "space heater" in the 
future - it is a Good Idea (tm) to disable hyper threads (at least with 
this sort of hyper threading in E5-2670 processors) or you can have what 
I presume are two HT's battling one another in the idle loop.

rick jones