Re: [RFC resend] Perf: Trigger and dump sample info to perf.data from user space ring buffer

[David Ahern <dsahern@gmail.com>]

On 9/21/15 9:16 PM, Yunlong Song wrote:
> [Problem Background]
>
> We want to run perf in daemon mode and collect the traces when the exception
> (e.g., machine crashes, app performance goes down) appears. Perf may run for a
> long time (from days to weeks or even months), since we do not know when the
> exception will appear at all, however it will appear at some time (especially
> for a beta product). If we simply use “perf record” as usual, here come two
> problems as time goes by: 1 there will be large amounts of IOs created for writing
> perf.data which may affects the performance a lot; 2 the size of perf.data will
> be larger and larger as well. Although we can use eBPF to reduce the traces in
> normal case, but in our case, the perf runs in daemon mode for a long time and
> that will accumulate the traces as time goes by.

This is a perf-based scheduling daemon I wrote a few years ago:

https://github.com/dsahern/linux/blob/perf/full-monty/tools/perf/schedmon.c

It solves a similar problem by holding the last N-seconds or M-bytes of 
events in memory. When something of significance happens it is notified 
to dump events to a file. The events are scheduling tracepoints but 
could easily be anything else.

David