Wiki docs on counting and tracing KVM perf events

Wiki docs on counting and tracing KVM perf events

[Stefan Hajnoczi]

How to count and trace KVM perf events:

http://www.linux-kvm.org/page/Perf_events

I want to draw attention to this because traditional kvm_stat and
kvm_trace use has been moving over to the debugfs based tracing
mechanisms.  Perhaps we can flesh out documentation and examples of
common perf event usage.

Stefan

Re: Wiki docs on counting and tracing KVM perf events

[Avi Kivity]

On 05/13/2010 05:35 PM, Stefan Hajnoczi wrote:
> How to count and trace KVM perf events:
>
> http://www.linux-kvm.org/page/Perf_events
>
> I want to draw attention to this because traditional kvm_stat and
> kvm_trace use has been moving over to the debugfs based tracing
> mechanisms.  Perhaps we can flesh out documentation and examples of
> common perf event usage.
>
>    

Two things are missing to make this really useful:

- a continuously updating difference mode like kvm_stat
- subevents; for example kvm:kvm_exit is an aggregate of all exit types 
that can be split using filters to show individual exit reason statistics

The plan is to eventually remove kvm_stat based performance monitoring 
in favour of perf events.

-- 
error compiling committee.c: too many arguments to function

Re: Wiki docs on counting and tracing KVM perf events

[Jes Sorensen]

On 05/13/10 16:57, Avi Kivity wrote:
> On 05/13/2010 05:35 PM, Stefan Hajnoczi wrote:
>> How to count and trace KVM perf events:
>>
>> http://www.linux-kvm.org/page/Perf_events
>>
>> I want to draw attention to this because traditional kvm_stat and
>> kvm_trace use has been moving over to the debugfs based tracing
>> mechanisms.  Perhaps we can flesh out documentation and examples of
>> common perf event usage.
>>
>>    
> 
> Two things are missing to make this really useful:
> 
> - a continuously updating difference mode like kvm_stat
> - subevents; for example kvm:kvm_exit is an aggregate of all exit types
> that can be split using filters to show individual exit reason statistics

Third missing item, which I find really useful:
- run once spit out raw counters

For some operations, like file system benchmarking, it is useful to
sample the counters before and after and then divide the raw number of
events by the number of IOPS performed by the benchmark. If perf spits
out events/sec it's kinda hard to get this.

Cheers,
Jes

Re: Wiki docs on counting and tracing KVM perf events

[Avi Kivity]

On 05/20/2010 11:15 AM, Jes Sorensen wrote:
> On 05/13/10 16:57, Avi Kivity wrote:
>    
>> On 05/13/2010 05:35 PM, Stefan Hajnoczi wrote:
>>      
>>> How to count and trace KVM perf events:
>>>
>>> http://www.linux-kvm.org/page/Perf_events
>>>
>>> I want to draw attention to this because traditional kvm_stat and
>>> kvm_trace use has been moving over to the debugfs based tracing
>>> mechanisms.  Perhaps we can flesh out documentation and examples of
>>> common perf event usage.
>>>
>>>
>>>        
>> Two things are missing to make this really useful:
>>
>> - a continuously updating difference mode like kvm_stat
>> - subevents; for example kvm:kvm_exit is an aggregate of all exit types
>> that can be split using filters to show individual exit reason statistics
>>      
> Third missing item, which I find really useful:
> - run once spit out raw counters
>
> For some operations, like file system benchmarking, it is useful to
> sample the counters before and after and then divide the raw number of
> events by the number of IOPS performed by the benchmark. If perf spits
> out events/sec it's kinda hard to get this.
>    

That's 'perf stat -a sleep 2'

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Wiki docs on counting and tracing KVM perf events

[Jes Sorensen]

On 05/20/10 10:19, Avi Kivity wrote:
> On 05/20/2010 11:15 AM, Jes Sorensen wrote:       
>>> Two things are missing to make this really useful:
>>>
>>> - a continuously updating difference mode like kvm_stat
>>> - subevents; for example kvm:kvm_exit is an aggregate of all exit types
>>> that can be split using filters to show individual exit reason
>>> statistics
>>>      
>> Third missing item, which I find really useful:
>> - run once spit out raw counters
>>
>> For some operations, like file system benchmarking, it is useful to
>> sample the counters before and after and then divide the raw number of
>> events by the number of IOPS performed by the benchmark. If perf spits
>> out events/sec it's kinda hard to get this.
>>    
> 
> That's 'perf stat -a sleep 2'

sleep 2 doesn't really cut it, I guess you could do it with

perf stat -a sleep 0.1

but to be honest, that is pretty ugly. Something like this would be a
lot nicer:

perf stat -a -raw

Jes

Re: Wiki docs on counting and tracing KVM perf events

[Stefan Hajnoczi]

8330  kvm:kvm_entry            #      0.000 M/sec
^--- count since starting perf

The 8330 number means that kvm_entry has fired 8330 times since perf
was started.  Like Avi says, you need to keep the perf process
running.  I run benchmarks using a script that kills perf after the
benchmark completes.

Jes, you're right, something like "perf stat -e kvm:* --start" and
"perf stat --stop" would be more usable for system-wide monitoring.  I
wonder if it is possible to support this or whether the perf process
needs to periodically accumulate the counters (i.e. babysit the kernel
infrastructure)?

Stefan

Re: Wiki docs on counting and tracing KVM perf events

[Avi Kivity]

On 05/20/2010 02:05 PM, Stefan Hajnoczi wrote:
> 8330  kvm:kvm_entry            #      0.000 M/sec
> ^--- count since starting perf
>
> The 8330 number means that kvm_entry has fired 8330 times since perf
> was started.  Like Avi says, you need to keep the perf process
> running.  I run benchmarks using a script that kills perf after the
> benchmark completes.
>
> Jes, you're right, something like "perf stat -e kvm:* --start" and
> "perf stat --stop" would be more usable for system-wide monitoring.  I
> wonder if it is possible to support this or whether the perf process
> needs to periodically accumulate the counters (i.e. babysit the kernel
> infrastructure)?
>    

perf needs to be running to pull data out of the kernel (and since 
profiling is tied to an fd life cycle).

What's wrong with starting perf after the warm-up period and stopping it 
before it's done?

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Wiki docs on counting and tracing KVM perf events

[Jes Sorensen]

On 05/20/10 13:10, Avi Kivity wrote:
> On 05/20/2010 02:05 PM, Stefan Hajnoczi wrote:
>> Jes, you're right, something like "perf stat -e kvm:* --start" and
>> "perf stat --stop" would be more usable for system-wide monitoring.  I
>> wonder if it is possible to support this or whether the perf process
>> needs to periodically accumulate the counters (i.e. babysit the kernel
>> infrastructure)?
>>    
> 
> perf needs to be running to pull data out of the kernel (and since
> profiling is tied to an fd life cycle).
> 
> What's wrong with starting perf after the warm-up period and stopping it
> before it's done?

It's pretty hard to script.

I have scripts that do 'read_stats() ; run_test() ; read_stats() ;
calc_average()'

Autotest would like to do this kinda stuff too.

Jes

Re: Wiki docs on counting and tracing KVM perf events

[Stefan Hajnoczi]

On Thu, May 20, 2010 at 12:16 PM, Jes Sorensen <Jes.Sorensen@redhat.com> wrote:
> On 05/20/10 13:10, Avi Kivity wrote:
>> What's wrong with starting perf after the warm-up period and stopping it
>> before it's done?
>
> It's pretty hard to script.

I use the following.  It ain't pretty:

#!/bin/bash
cleanup() {
        trap - 2
        kill -2 $sleep_pid
        echo 0 >/sys/kernel/debug/tracing/events/kvm/enable
        kill $cat_pid
}

perf stat -a -e 'kvm:*' sleep 1h >results/perf_stat 2>&1 &
sleep_pid=$(sleep 1 && pgrep -x -f "sleep 1h")   # sleep 1 is to avoid
race with forked perf process
trap cleanup 2
echo 1 >/sys/kernel/debug/tracing/events/kvm/enable
cat /sys/kernel/debug/tracing/trace_pipe >results/trace &
cat_pid=$!

# ...do stuff here...

cleanup

Stefan

Re: Wiki docs on counting and tracing KVM perf events

[Avi Kivity]

On 05/20/2010 02:23 PM, Stefan Hajnoczi wrote:
> On Thu, May 20, 2010 at 12:16 PM, Jes Sorensen<Jes.Sorensen@redhat.com>  wrote:
>    
>> On 05/20/10 13:10, Avi Kivity wrote:
>>      
>>> What's wrong with starting perf after the warm-up period and stopping it
>>> before it's done?
>>>        
>> It's pretty hard to script.
>>      
> I use the following.  It ain't pretty:
>
> #!/bin/bash
> cleanup() {
>          trap - 2
>          kill -2 $sleep_pid
>          echo 0>/sys/kernel/debug/tracing/events/kvm/enable
>          kill $cat_pid
> }
>
> perf stat -a -e 'kvm:*' sleep 1h>results/perf_stat 2>&1&
> sleep_pid=$(sleep 1&&  pgrep -x -f "sleep 1h")   # sleep 1 is to avoid
> race with forked perf process
> trap cleanup 2
> echo 1>/sys/kernel/debug/tracing/events/kvm/enable
> cat /sys/kernel/debug/tracing/trace_pipe>results/trace&
> cat_pid=$!
>
> # ...do stuff here...
>
> cleanup
>    

perf will enable the events by itself (no?), so all you need is is the 
perf call in the middle.

What's missing is vmstat-like or kvm_stat-like output, but that's 
another thing.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Wiki docs on counting and tracing KVM perf events

[Stefan Hajnoczi]

On Thu, May 20, 2010 at 1:14 PM, Avi Kivity <avi@redhat.com> wrote:
>> echo 1>/sys/kernel/debug/tracing/events/kvm/enable
>> cat /sys/kernel/debug/tracing/trace_pipe>results/trace&
>
> perf will enable the events by itself (no?), so all you need is is the perf
> call in the middle.

Yes, it will enable events.  However, I am also generating a
kvm_trace-like log using trace_pipe.  On this box I couldn't get perf
trace working so I used trace_pipe for the kvm_trace-equivalent and
perf stat for the kvm_stat equivalent.

Stefan

Re: Wiki docs on counting and tracing KVM perf events

[Avi Kivity]

On 05/20/2010 03:24 PM, Stefan Hajnoczi wrote:
> On Thu, May 20, 2010 at 1:14 PM, Avi Kivity<avi@redhat.com>  wrote:
>    
>>> echo 1>/sys/kernel/debug/tracing/events/kvm/enable
>>> cat /sys/kernel/debug/tracing/trace_pipe>results/trace&
>>>        
>> perf will enable the events by itself (no?), so all you need is is the perf
>> call in the middle.
>>      
> Yes, it will enable events.  However, I am also generating a
> kvm_trace-like log using trace_pipe.  On this box I couldn't get perf
> trace working so I used trace_pipe for the kvm_trace-equivalent and
> perf stat for the kvm_stat equivalent.
>    

Ah.  There's the new shiny trace_cmd as well.

If you're getting a (binary) trace, runing perf stat is redundant since 
it contains the same data.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.