Re: Perf event for Wall-time based sampling?

[Milian Wolff <mail@milianw.de>]

On Friday 19 September 2014 10:11:21 Milian Wolff wrote:
> On Thursday 18 September 2014 17:36:25 Arnaldo Carvalho de Melo wrote:
> > Em Thu, Sep 18, 2014 at 02:17:49PM -0600, David Ahern escreveu:
> > > On 9/18/14, 1:17 PM, Arnaldo Carvalho de Melo wrote:
> > > >>This was also why I asked my initial question, which I want to repeat
> > > >>once
> > > >>
> > > >>>more: Is there a technical reason to not offer a "timer" software
> > > >>>event
> > > >>>to
> > > >>>perf? I'm a complete layman when it comes to Kernel internals, but
> > > >>>from
> > > >>>a user point of view this would be awesome:
> > > >>>
> > > >>>perf record --call-graph dwarf -e sw-timer -F 100 someapplication
> > > >>>
> > > >>>This command would then create a timer in the kernel with a 100Hz
> > > >>>frequency. Whenever it fires, the callgraphs of all threads in
> > > >>>$someapplication are sampled and written to perf.data. Is this
> > > >>>technically not feasible? Or is it simply not implemented?
> > > >>>I'm experimenting with a libunwind based profiler, and with some ugly
> > > >>>signal hackery I can now grab backtraces by sending my application
> > > >>>SIGUSR1. Based on> >
> > > >
> > > >Humm, can't you do the same thing with perf? I.e. you send SIGUSR1 to
> > > >your app with the frequency you want, and then hook a 'perf probe' into
> > > >your signal... /me tries some stuff, will get back with results...
> 
> That is actually a very good idea. With the more powerful scripting
> abilities in perf now, that could/should do the job indeed. I'll also try
> this out.

Indeed, this does not seem to work as intended:

~~~~~~~~~~~~~~~~~~

#include <csignal>

extern "C" {

void signalHandler(int sig)
{
}

}

struct Pace
{
    Pace()
    {
        signal(SIGUSR1, &signalHandler);
    }
};

static Pace initializeLibPace;
~~~~~~~~~~~~~~~~~~~

# as user, run:
g++ -shared -fPIC -o libpace.so libpace.cpp
LD_PRELOAD=$(readlink -f libpace.so) somebinary
while true; do killall -s SIGUSR1 somebinary; done

# as root, run:
perf probe -x libpace.so signalHandler
perf record --call-graph dwarf -e probe_libpace:signalHandler -p $(pidof 
yourApp)
perf report -g graph --no-children

Nice! But there are multiple issues with this approach, compared to 
potentially offering it in perf directly:

1) you need root/elevated access to not only create the probe, but also to use 
it then during record. this also requires you to to attach to the debuggee, as 
you don't want to run most user-space apps as root. while runtime-attaching is 
a powerful feature, it is cumbersome to use for the simple case

2) all of the issues of signal handling. Try the above on my initial test 
application:

int main() {
  sleep(10);
  return 0;
}

After sending it the first SIGUSR1, the sleep will return. So we change the 
behavior of the application by profiling it, which is of course very bad. 
Sure, most of the time such sleeps are run in a loop, but even then one will 
influence the runtime behavior. And you will never find the spurious 
"sleep(100)" that you forgot in a debug session in your application...

3) What about multiple threads? One could potentially get it working by 
overwriting pthread_create/exit via LD_PRELOAD and propagating a custom signal 
via pthread_kill to all running threads when the external SIGUSR1 comes in.

So, I hope this shows you all that having such a feature in perf directly 
would be a good thing.

-- 
Milian Wolff
mail@milianw.de
http://milianw.de