Re: Perf: Question about continuous background data collection

[Jiri Olsa <jolsa@redhat.com>]

On Thu, Oct 07, 2021 at 06:17:29AM -0300, Arnaldo Carvalho de Melo wrote:
> 
> 
> On October 6, 2021 6:25:57 PM GMT-03:00, Renjith Ponnappan <renjithponnapps@gmail.com> wrote:
> >Hello Arnaldo & Jiri,
> >
> >Thank you for your response.
> >
> >The perf daemon is the closest implementation for what I was looking for.
> >Here we run perf in the back-ground, keep overwriting the samples being
> >collected and use the SIGUSR2 to signal to the perf daemon to collect the
> >perf data to a file. This fulfills the following requirements:
> >
> >   1. Run perf in the background to collect data
> >   2. A method to signal perf to collect the samples for the current cycle.
> >
> >The last part of the requirement which  was:
> >
> >   1. The data-collection in step 2 above, rather than writing the data to
> >   file store it in in-memory datastructure via pointer manipulation. We can
> >   have a list of such samples stored in memory until the next step. This
> >   helps free up the CPU cycles used by perf for writing to file for
> >   applications.
> >   2. A method to signal perf to dump all the collected samples into
> >   separate files. This way the user can collect the stored samples when the
> >   CPU is relatively free.
> >
> >Let me know whether we have support for storing samples as in-memory
> >samples.
> 
> Have you looked at the URL I pointed to you?
> 
> https://www.spinics.net/lists/linux-perf-users/msg11455.html
> 
> More specifically, read about 'perf record --overwrite"

also there's example with flight recorder in perf-daemon man page:
https://man7.org/linux/man-pages/man1/perf-daemon.1.html

jirka

> 
> - Arnaldo
> >
> >Thanks in advance for your help!
> >
> >*cheers,*
> >*Renjith*
> >
> >
> >On Tue, Oct 5, 2021 at 12:00 AM Jiri Olsa <jolsa@redhat.com> wrote:
> >
> >> On Thu, Sep 30, 2021 at 10:39:21PM -0300, Arnaldo Carvalho de Melo wrote:
> >> >
> >> >
> >> > On September 30, 2021 10:28:28 PM GMT-03:00, Renjith Ponnappan <
> >> renjithponnapps@gmail.com> wrote:
> >> > >Hello Peter/Ingo/Arnaldo,
> >> > >
> >> > >First of all, apologies if I bombarded you with an irrelevant question
> >> in
> >> > >your busy day and ignore this if the question is irrelevant.
> >> > >
> >> > >I had a question about continuous background data-collection with perf
> >> and
> >> > >hope you are the right person to answer this. If not, it would be great
> >> if
> >> > >you can redirect me to the right person.
> >> > >
> >> > >I am trying to build a CPU profiling system (on an embedded ARM Platform
> >> > >with CPU/memory constraints) which has CPU Samples already collected
> >> when
> >> > >an application CPU starvation scenario occurs based on perf. The
> >> > >implementation I am trying to use is:
> >> > >
> >> > >   1. Run perf in the background collecting samples for the entire
> >> system
> >> >
> >> >
> >> > This is already in perf:
> >> >
> >> > https://www.spinics.net/lists/linux-perf-users/msg11455.html
> >> >
> >> >
> >> > Reply adding linux-perf-users@vger.kernel.org.
> >> >
> >> > - Arnaldo
> >> >
> >> >
> >> > >   with a sleep period of 60 seconds
> >> > >   2. When an application CPU starvation scenario occurs (detected and
> >> > >   raised by applications) notify the collection process to store the
> >> last
> >> > >   perf collection as data to be analyzed offline.
> >> > >
> >> > >Have you come across such a scenario and any recommendations on this?
> >> > >
> >> > >The following are the two implementations I have on the above:
> >> > >
> >> > >   1. An external process which instructs perf to record the 60 seconds
> >> by
> >> > >   providing unique filenames each time. This approach was taking
> >> around 40%
> >> > >   CPU of a CPU core, everytime the perf record was getting written
> >> (once for
> >> > >   each 60 seconds cycle). This isn't okay as it could cause
> >> aggravation of
> >> > >   the CPU starvation situation.
> >> > >   2. I tinkered with Perf Code to add the logic of looping and writing
> >> the
> >> > >   file incase of an event only. This did reduce the CPU to only the
> >> case when
> >> > >   an event was detected.
> >>
> >> hi,
> >> and there's also perf daemon to run perf sessions on background:
> >> https://lore.kernel.org/lkml/20210130234856.271282-19-jolsa@kernel.org/
> >>
> >> jirka
> >>
> >> > >
> >> > >Would like to hear your opinion on whether approach 2 is the right way
> >> here
> >> > >and any suggestion/guidance you may have.
> >> > >
> >> > >Thanks in advance for this help!
> >> > >
> >> > >*cheers,*
> >> > >*Renjith*
> >> >
> >>
> >>
>