Interrupt caused by software events

Interrupt caused by software events

[Martin Ichilevici de Oliveira]

[-- Attachment #1: Type: text/plain, Size: 682 bytes --]

Hello,

I'm learning the perf_events programming interface and I'm not sure if
what I'm trying to do is possible. I could only find the perf_event_open
documentation, but unfortunately that's not enough for my needs, as it
only counts the number of events that happened.

I'm looking for a way to determine which instruction caused some events:
something similar to what perf annotate does, but in an online fashion. 
Ideally, I'd like to have some kind of handler that gets called whenever
a perf event occurs. Is that even possible? If not, is there any way I 
could get this information in a real-time way?

I'll be using PEBS or IBS events.

Thank you,
-Martin

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Interrupt caused by software events

[Jiri Olsa]

On Tue, Jul 08, 2014 at 11:42:42AM -0300, Martin Ichilevici de Oliveira wrote:
> Hello,
> 
> I'm learning the perf_events programming interface and I'm not sure if
> what I'm trying to do is possible. I could only find the perf_event_open
> documentation, but unfortunately that's not enough for my needs, as it
> only counts the number of events that happened.
> 
> I'm looking for a way to determine which instruction caused some events:
> something similar to what perf annotate does, but in an online fashion. 
> Ideally, I'd like to have some kind of handler that gets called whenever
> a perf event occurs. Is that even possible? If not, is there any way I 
> could get this information in a real-time way?

hi,
you could use perf_event_create_kernel_counter interface,
check the lockup watchdog in kernel/watchdog.c

jirka

Re: Interrupt caused by software events

[Vince Weaver]

On Tue, 8 Jul 2014, Martin Ichilevici de Oliveira wrote:

> I'm looking for a way to determine which instruction caused some events:
> something similar to what perf annotate does, but in an online fashion. 
> Ideally, I'd like to have some kind of handler that gets called whenever
> a perf event occurs. Is that even possible? If not, is there any way I 
> could get this information in a real-time way?

Not really sure by what you mean by "real-time", but you can theoretically 
set up the perf_event mmap buffer with a low threshold, and have it 
configured to send a signal to your program whenever an event comes in.
In theory you can get the PEBS-like info out that way.  It's going to have 
fairly high overhead though, especially if you set the sample rate high.

It's even worse if you are trying to get full info on *every* event.
That's probably not possible with any even that happens more than a few 
times a second.

Vince

Re: Interrupt caused by software events

[Martin Ichilevici de Oliveira]

[-- Attachment #1: Type: text/plain, Size: 2213 bytes --]

On Wed, Jul 09, 2014 at 10:05:55AM -0400, Vince Weaver wrote:
> Not really sure by what you mean by "real-time", but you can theoretically 
> set up the perf_event mmap buffer with a low threshold, and have it 
> configured to send a signal to your program whenever an event comes in.
> In theory you can get the PEBS-like info out that way.  It's going to have 
> fairly high overhead though, especially if you set the sample rate high.

On Wed, Jul 09, 2014 at 03:54:18PM +0200, Jiri Olsa wrote:
> hi,
> you could use perf_event_create_kernel_counter interface,
> check the lockup watchdog in kernel/watchdog.c

Vince and Jiri,

Thank you very much for your replies.

I re-read my email and I'm not sure I was clear, so just in case I
didn't made myself clear, here's a high-level example of what I'm
looking for.

/*******************/
/* Gets called whenever my program generates a cache-miss */
void my_func()
{
    unsigned long addr = get_ip_that_generated_event();
    printf("A cache-miss occurred at IP: %x!\n", addr);
}

int main()
{
    struct perf_event_attr pe;
    pe.type = PERF_TYPE_HARDWARE.
    pe.config = PERF_COUNT_HW_CACHE_MISSES;
    ...
    register_handler(&pe, my_func);
    // code to be monitored
    unregister_handler();
    return 0;
}
/*******************/

Every time an event occurs (a cache-miss in my example), my_func should
get called. The get_ip_that_generated_event() function should return the
Instruction Pointer that generated the event.

My impression is that this could be done with a low overhead (other then
that introduced by what I do in my_func), because 
"perf record -e cache-misses ./a.out"
can pinpoint with little overhead how many times the event was caused by
each instruction. But maybe I'm just missing something.

Vince:
I'm unsure how to configure the mmap buffer you mentioned, and I
couldn't find any examples online. Could you please point me to some
reference, or maybe provide a little example?

Jiri:
I'm under the impression I'd have to be in a kernel module (or in kernel
mode in general) to use perf_event_create_kernel_counter(). Is that
correct?

Thank you once again,
-Martin

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: Interrupt caused by software events

[Vince Weaver]

On Fri, 11 Jul 2014, Martin Ichilevici de Oliveira wrote:

> /* Gets called whenever my program generates a cache-miss */
> void my_func()
> {
>     unsigned long addr = get_ip_that_generated_event();
>     printf("A cache-miss occurred at IP: %x!\n", addr);
> }
> 
> int main()
> {
>     struct perf_event_attr pe;
>     pe.type = PERF_TYPE_HARDWARE.
>     pe.config = PERF_COUNT_HW_CACHE_MISSES;
>     ...
>     register_handler(&pe, my_func);
>     // code to be monitored
>     unregister_handler();
>     return 0;
> }
> /*******************/
> 
> Every time an event occurs (a cache-miss in my example), my_func should
> get called. The get_ip_that_generated_event() function should return the
> Instruction Pointer that generated the event.

In general you can't do this.  Cache misses happen way too often, your 
program would not be able to keep up, and likely it would be causing cache 
misses itself.

Also you should read up on skid, it's very hard to track an event to an 
exact IP, although if your CPU has PEBS it's possible with certain 
limitations.

> My impression is that this could be done with a low overhead (other then
> that introduced by what I do in my_func), because 
> "perf record -e cache-misses ./a.out"
> can pinpoint with little overhead how many times the event was caused by
> each instruction. But maybe I'm just missing something.

No, what perf record is doing is sampling.  It's getting interrupted say 
every 100,000 cache misses and reporting the values, and hopefully the 
ones that trigger are statistically relevant enough to extract the full 
program behavior.

> I'm unsure how to configure the mmap buffer you mentioned, and I
> couldn't find any examples online. Could you please point me to some
> reference, or maybe provide a little example?

There are some examples in my perf_event_tests package.

See for example
	https://github.com/deater/perf_event_tests/blob/master/tests/x86_intel/pebs.c

although that example really isn't using PEBS yet, I'm currently working 
on that code, so beware I might be re-writing and moving around the code 
in that directory in the next few days.

Vince

Re: Interrupt caused by software events

[Jiri Olsa]

On Fri, Jul 11, 2014 at 11:20:58AM -0300, Martin Ichilevici de Oliveira wrote:
> On Wed, Jul 09, 2014 at 10:05:55AM -0400, Vince Weaver wrote:
> > Not really sure by what you mean by "real-time", but you can theoretically 
> > set up the perf_event mmap buffer with a low threshold, and have it 
> > configured to send a signal to your program whenever an event comes in.
> > In theory you can get the PEBS-like info out that way.  It's going to have 
> > fairly high overhead though, especially if you set the sample rate high.

SNIP

> 
> Jiri:
> I'm under the impression I'd have to be in a kernel module (or in kernel
> mode in general) to use perf_event_create_kernel_counter(). Is that
> correct?

yep, thats kernel space only

jirka

Re: Interrupt caused by software events

[Andi Kleen]

Martin Ichilevici de Oliveira <martin.i.oliveira@gmail.com> writes:
>
> /*******************/
> /* Gets called whenever my program generates a cache-miss */
> void my_func()
> {
>     unsigned long addr = get_ip_that_generated_event();
>     printf("A cache-miss occurred at IP: %x!\n", addr);
> }

This will almost certainly not work with any useful workload.
Just compare the overhead of the instrumentation with the total
run time. The program will be doing nothing but such interrupts
and perf will eventually throttle the interrupts. Besides
PEBS is not designed to catch every event anyways.

If you really want to instrument all cache misses you most likely
want to use some memory trace mechanism and a offline cache model
or a simulator.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only