Re: Perf: Question about continuous background data collection

Re: Perf: Question about continuous background data collection

[Renjith Ponnappan]

Hello Arnaldo & Jiri,

Thank you for your response.

The perf daemon is the closest implementation for what I was looking
for. Here we run perf in the back-ground, keep overwriting the samples
being collected and use the SIGUSR2 to signal to the perf daemon to
collect the perf data to a file. This fulfills the following
requirements:

1. Run perf in the background to collect data
2. A method to signal perf to collect the samples for the current cycle.

The last part of the requirement which  was:

1. The data-collection in step 2 above, rather than writing the data
to file store it in in-memory datastructure via pointer manipulation.
We can have a list of such samples stored in memory until the next
step. This helps free up the CPU cycles used by perf for writing to
file for applications.
2. A method to signal perf to dump all the collected samples into
separate files. This way the user can collect the stored samples when
the CPU is relatively free.

Let me know whether we have support for storing samples as in-memory samples.

Thanks in advance for your help!

cheers,
Renjith
cheers,
Renjith

On Tue, Oct 5, 2021 at 12:00 AM Jiri Olsa <jolsa@redhat.com> wrote:
>
> On Thu, Sep 30, 2021 at 10:39:21PM -0300, Arnaldo Carvalho de Melo wrote:
> >
> >
> > On September 30, 2021 10:28:28 PM GMT-03:00, Renjith Ponnappan <renjithponnapps@gmail.com> wrote:
> > >Hello Peter/Ingo/Arnaldo,
> > >
> > >First of all, apologies if I bombarded you with an irrelevant question in
> > >your busy day and ignore this if the question is irrelevant.
> > >
> > >I had a question about continuous background data-collection with perf and
> > >hope you are the right person to answer this. If not, it would be great if
> > >you can redirect me to the right person.
> > >
> > >I am trying to build a CPU profiling system (on an embedded ARM Platform
> > >with CPU/memory constraints) which has CPU Samples already collected when
> > >an application CPU starvation scenario occurs based on perf. The
> > >implementation I am trying to use is:
> > >
> > >   1. Run perf in the background collecting samples for the entire system
> >
> >
> > This is already in perf:
> >
> > https://www.spinics.net/lists/linux-perf-users/msg11455.html
> >
> >
> > Reply adding linux-perf-users@vger.kernel.org.
> >
> > - Arnaldo
> >
> >
> > >   with a sleep period of 60 seconds
> > >   2. When an application CPU starvation scenario occurs (detected and
> > >   raised by applications) notify the collection process to store the last
> > >   perf collection as data to be analyzed offline.
> > >
> > >Have you come across such a scenario and any recommendations on this?
> > >
> > >The following are the two implementations I have on the above:
> > >
> > >   1. An external process which instructs perf to record the 60 seconds by
> > >   providing unique filenames each time. This approach was taking around 40%
> > >   CPU of a CPU core, everytime the perf record was getting written (once for
> > >   each 60 seconds cycle). This isn't okay as it could cause aggravation of
> > >   the CPU starvation situation.
> > >   2. I tinkered with Perf Code to add the logic of looping and writing the
> > >   file incase of an event only. This did reduce the CPU to only the case when
> > >   an event was detected.
>
> hi,
> and there's also perf daemon to run perf sessions on background:
> https://lore.kernel.org/lkml/20210130234856.271282-19-jolsa@kernel.org/
>
> jirka
>
> > >
> > >Would like to hear your opinion on whether approach 2 is the right way here
> > >and any suggestion/guidance you may have.
> > >
> > >Thanks in advance for this help!
> > >
> > >*cheers,*
> > >*Renjith*
> >
>

Re: Perf: Question about continuous background data collection

[Arnaldo Carvalho de Melo]

On October 6, 2021 6:25:57 PM GMT-03:00, Renjith Ponnappan <renjithponnapps@gmail.com> wrote:
>Hello Arnaldo & Jiri,
>
>Thank you for your response.
>
>The perf daemon is the closest implementation for what I was looking for.
>Here we run perf in the back-ground, keep overwriting the samples being
>collected and use the SIGUSR2 to signal to the perf daemon to collect the
>perf data to a file. This fulfills the following requirements:
>
>   1. Run perf in the background to collect data
>   2. A method to signal perf to collect the samples for the current cycle.
>
>The last part of the requirement which  was:
>
>   1. The data-collection in step 2 above, rather than writing the data to
>   file store it in in-memory datastructure via pointer manipulation. We can
>   have a list of such samples stored in memory until the next step. This
>   helps free up the CPU cycles used by perf for writing to file for
>   applications.
>   2. A method to signal perf to dump all the collected samples into
>   separate files. This way the user can collect the stored samples when the
>   CPU is relatively free.
>
>Let me know whether we have support for storing samples as in-memory
>samples.

Have you looked at the URL I pointed to you?

https://www.spinics.net/lists/linux-perf-users/msg11455.html

More specifically, read about 'perf record --overwrite"

- Arnaldo
>
>Thanks in advance for your help!
>
>*cheers,*
>*Renjith*
>
>
>On Tue, Oct 5, 2021 at 12:00 AM Jiri Olsa <jolsa@redhat.com> wrote:
>
>> On Thu, Sep 30, 2021 at 10:39:21PM -0300, Arnaldo Carvalho de Melo wrote:
>> >
>> >
>> > On September 30, 2021 10:28:28 PM GMT-03:00, Renjith Ponnappan <
>> renjithponnapps@gmail.com> wrote:
>> > >Hello Peter/Ingo/Arnaldo,
>> > >
>> > >First of all, apologies if I bombarded you with an irrelevant question
>> in
>> > >your busy day and ignore this if the question is irrelevant.
>> > >
>> > >I had a question about continuous background data-collection with perf
>> and
>> > >hope you are the right person to answer this. If not, it would be great
>> if
>> > >you can redirect me to the right person.
>> > >
>> > >I am trying to build a CPU profiling system (on an embedded ARM Platform
>> > >with CPU/memory constraints) which has CPU Samples already collected
>> when
>> > >an application CPU starvation scenario occurs based on perf. The
>> > >implementation I am trying to use is:
>> > >
>> > >   1. Run perf in the background collecting samples for the entire
>> system
>> >
>> >
>> > This is already in perf:
>> >
>> > https://www.spinics.net/lists/linux-perf-users/msg11455.html
>> >
>> >
>> > Reply adding linux-perf-users@vger.kernel.org.
>> >
>> > - Arnaldo
>> >
>> >
>> > >   with a sleep period of 60 seconds
>> > >   2. When an application CPU starvation scenario occurs (detected and
>> > >   raised by applications) notify the collection process to store the
>> last
>> > >   perf collection as data to be analyzed offline.
>> > >
>> > >Have you come across such a scenario and any recommendations on this?
>> > >
>> > >The following are the two implementations I have on the above:
>> > >
>> > >   1. An external process which instructs perf to record the 60 seconds
>> by
>> > >   providing unique filenames each time. This approach was taking
>> around 40%
>> > >   CPU of a CPU core, everytime the perf record was getting written
>> (once for
>> > >   each 60 seconds cycle). This isn't okay as it could cause
>> aggravation of
>> > >   the CPU starvation situation.
>> > >   2. I tinkered with Perf Code to add the logic of looping and writing
>> the
>> > >   file incase of an event only. This did reduce the CPU to only the
>> case when
>> > >   an event was detected.
>>
>> hi,
>> and there's also perf daemon to run perf sessions on background:
>> https://lore.kernel.org/lkml/20210130234856.271282-19-jolsa@kernel.org/
>>
>> jirka
>>
>> > >
>> > >Would like to hear your opinion on whether approach 2 is the right way
>> here
>> > >and any suggestion/guidance you may have.
>> > >
>> > >Thanks in advance for this help!
>> > >
>> > >*cheers,*
>> > >*Renjith*
>> >
>>
>>

Re: Perf: Question about continuous background data collection

[Jiri Olsa]

On Thu, Oct 07, 2021 at 06:17:29AM -0300, Arnaldo Carvalho de Melo wrote:
> 
> 
> On October 6, 2021 6:25:57 PM GMT-03:00, Renjith Ponnappan <renjithponnapps@gmail.com> wrote:
> >Hello Arnaldo & Jiri,
> >
> >Thank you for your response.
> >
> >The perf daemon is the closest implementation for what I was looking for.
> >Here we run perf in the back-ground, keep overwriting the samples being
> >collected and use the SIGUSR2 to signal to the perf daemon to collect the
> >perf data to a file. This fulfills the following requirements:
> >
> >   1. Run perf in the background to collect data
> >   2. A method to signal perf to collect the samples for the current cycle.
> >
> >The last part of the requirement which  was:
> >
> >   1. The data-collection in step 2 above, rather than writing the data to
> >   file store it in in-memory datastructure via pointer manipulation. We can
> >   have a list of such samples stored in memory until the next step. This
> >   helps free up the CPU cycles used by perf for writing to file for
> >   applications.
> >   2. A method to signal perf to dump all the collected samples into
> >   separate files. This way the user can collect the stored samples when the
> >   CPU is relatively free.
> >
> >Let me know whether we have support for storing samples as in-memory
> >samples.
> 
> Have you looked at the URL I pointed to you?
> 
> https://www.spinics.net/lists/linux-perf-users/msg11455.html
> 
> More specifically, read about 'perf record --overwrite"

also there's example with flight recorder in perf-daemon man page:
https://man7.org/linux/man-pages/man1/perf-daemon.1.html

jirka

> 
> - Arnaldo
> >
> >Thanks in advance for your help!
> >
> >*cheers,*
> >*Renjith*
> >
> >
> >On Tue, Oct 5, 2021 at 12:00 AM Jiri Olsa <jolsa@redhat.com> wrote:
> >
> >> On Thu, Sep 30, 2021 at 10:39:21PM -0300, Arnaldo Carvalho de Melo wrote:
> >> >
> >> >
> >> > On September 30, 2021 10:28:28 PM GMT-03:00, Renjith Ponnappan <
> >> renjithponnapps@gmail.com> wrote:
> >> > >Hello Peter/Ingo/Arnaldo,
> >> > >
> >> > >First of all, apologies if I bombarded you with an irrelevant question
> >> in
> >> > >your busy day and ignore this if the question is irrelevant.
> >> > >
> >> > >I had a question about continuous background data-collection with perf
> >> and
> >> > >hope you are the right person to answer this. If not, it would be great
> >> if
> >> > >you can redirect me to the right person.
> >> > >
> >> > >I am trying to build a CPU profiling system (on an embedded ARM Platform
> >> > >with CPU/memory constraints) which has CPU Samples already collected
> >> when
> >> > >an application CPU starvation scenario occurs based on perf. The
> >> > >implementation I am trying to use is:
> >> > >
> >> > >   1. Run perf in the background collecting samples for the entire
> >> system
> >> >
> >> >
> >> > This is already in perf:
> >> >
> >> > https://www.spinics.net/lists/linux-perf-users/msg11455.html
> >> >
> >> >
> >> > Reply adding linux-perf-users@vger.kernel.org.
> >> >
> >> > - Arnaldo
> >> >
> >> >
> >> > >   with a sleep period of 60 seconds
> >> > >   2. When an application CPU starvation scenario occurs (detected and
> >> > >   raised by applications) notify the collection process to store the
> >> last
> >> > >   perf collection as data to be analyzed offline.
> >> > >
> >> > >Have you come across such a scenario and any recommendations on this?
> >> > >
> >> > >The following are the two implementations I have on the above:
> >> > >
> >> > >   1. An external process which instructs perf to record the 60 seconds
> >> by
> >> > >   providing unique filenames each time. This approach was taking
> >> around 40%
> >> > >   CPU of a CPU core, everytime the perf record was getting written
> >> (once for
> >> > >   each 60 seconds cycle). This isn't okay as it could cause
> >> aggravation of
> >> > >   the CPU starvation situation.
> >> > >   2. I tinkered with Perf Code to add the logic of looping and writing
> >> the
> >> > >   file incase of an event only. This did reduce the CPU to only the
> >> case when
> >> > >   an event was detected.
> >>
> >> hi,
> >> and there's also perf daemon to run perf sessions on background:
> >> https://lore.kernel.org/lkml/20210130234856.271282-19-jolsa@kernel.org/
> >>
> >> jirka
> >>
> >> > >
> >> > >Would like to hear your opinion on whether approach 2 is the right way
> >> here
> >> > >and any suggestion/guidance you may have.
> >> > >
> >> > >Thanks in advance for this help!
> >> > >
> >> > >*cheers,*
> >> > >*Renjith*
> >> >
> >>
> >>
>