Toggle perf stat on/off periodically and keep each sample

Toggle perf stat on/off periodically and keep each sample

[Fengkai Sun]

Hi all,

I've been using control fd and delay=-1 to toggle perf stat on until a
certain point of the program, and it works like a charm.

Now, I want to do this periodically and keep all the results.
Specifically, I have a performance-critical network program that reads
the packets in batches and processes them, and I want to record the
hardware counters on each batch, for future comparison.

Intuitively, this requires two functionalities I have no idea how to achieve:

1. The ability to zero out all recorded counters when the current
sample is finished and
2. The ability to keep individual samples, instead of outputting a
single overall result when the process finishes.

I'm wondering if this can be achieved under current version of perf,
and if such recording makes sense (e.g., does not incur too much
overhead). Thank you!


Thanks,
Fengkai

Re: Toggle perf stat on/off periodically and keep each sample

[Namhyung Kim]

Hello,

On Wed, Oct 16, 2024 at 09:28:33AM +0800, Fengkai Sun wrote:
> Hi all,
> 
> I've been using control fd and delay=-1 to toggle perf stat on until a
> certain point of the program, and it works like a charm.
> 
> Now, I want to do this periodically and keep all the results.
> Specifically, I have a performance-critical network program that reads
> the packets in batches and processes them, and I want to record the
> hardware counters on each batch, for future comparison.
> 
> Intuitively, this requires two functionalities I have no idea how to achieve:
> 
> 1. The ability to zero out all recorded counters when the current
> sample is finished and
> 2. The ability to keep individual samples, instead of outputting a
> single overall result when the process finishes.
> 
> I'm wondering if this can be achieved under current version of perf,
> and if such recording makes sense (e.g., does not incur too much
> overhead). Thank you!

I think you are looking for something like interval mode like -I 1000.

  $ sudo perf stat -a -I 1000 -e cycles,instructions sleep 5
  #           time             counts unit events
       1.001064439        644,313,943      cycles                                                                
       1.001064439        548,175,426      instructions                     #    0.85  insn per cycle            
       2.002682133        570,454,723      cycles                                                                
       2.002682133        426,891,566      instructions                     #    0.75  insn per cycle            
       3.004412672        565,162,161      cycles                                                                
       3.004412672        351,875,655      instructions                     #    0.62  insn per cycle            
       4.006131920        463,748,461      cycles                                                                
       4.006131920        239,702,866      instructions                     #    0.52  insn per cycle            
       5.001878221        466,409,911      cycles                                                                
       5.001878221        235,684,130      instructions                     #    0.51  insn per cycle

Of course it doesn't turn the events on and off.  But is it something
similar to what you want?

Thanks,
Namhyung

Re: Toggle perf stat on/off periodically and keep each sample

[Fengkai Sun]

Hello Namhyung, thank you very much for your reply!

On Fri, Oct 18, 2024 at 6:26 AM Namhyung Kim <namhyung@kernel.org> wrote:
>
> I think you are looking for something like interval mode like -I 1000.
>
>   $ sudo perf stat -a -I 1000 -e cycles,instructions sleep 5
>   #           time             counts unit events
>        1.001064439        644,313,943      cycles
>        1.001064439        548,175,426      instructions                     #    0.85  insn per cycle
>        2.002682133        570,454,723      cycles
>        2.002682133        426,891,566      instructions                     #    0.75  insn per cycle
>        3.004412672        565,162,161      cycles
>        3.004412672        351,875,655      instructions                     #    0.62  insn per cycle
>        4.006131920        463,748,461      cycles
>        4.006131920        239,702,866      instructions                     #    0.52  insn per cycle
>        5.001878221        466,409,911      cycles
>        5.001878221        235,684,130      instructions                     #    0.51  insn per cycle
>
> Of course it doesn't turn the events on and off.  But is it something
> similar to what you want?
>
> Thanks,
> Namhyung
>

Yes, this is similar to what I want. Of course, I want the interval to
be variable and can be set from the user program. I guess this cannot
be achieved by current perf, right?

---
Thanks,
Fengkai

Re: Toggle perf stat on/off periodically and keep each sample

[Ian Rogers]

On Fri, Oct 18, 2024 at 7:19 AM Fengkai Sun <qcloud1014@gmail.com> wrote:
>
> Hello Namhyung, thank you very much for your reply!
>
> On Fri, Oct 18, 2024 at 6:26 AM Namhyung Kim <namhyung@kernel.org> wrote:
> >
> > I think you are looking for something like interval mode like -I 1000.
> >
> >   $ sudo perf stat -a -I 1000 -e cycles,instructions sleep 5
> >   #           time             counts unit events
> >        1.001064439        644,313,943      cycles
> >        1.001064439        548,175,426      instructions                     #    0.85  insn per cycle
> >        2.002682133        570,454,723      cycles
> >        2.002682133        426,891,566      instructions                     #    0.75  insn per cycle
> >        3.004412672        565,162,161      cycles
> >        3.004412672        351,875,655      instructions                     #    0.62  insn per cycle
> >        4.006131920        463,748,461      cycles
> >        4.006131920        239,702,866      instructions                     #    0.52  insn per cycle
> >        5.001878221        466,409,911      cycles
> >        5.001878221        235,684,130      instructions                     #    0.51  insn per cycle
> >
> > Of course it doesn't turn the events on and off.  But is it something
> > similar to what you want?
> >
> > Thanks,
> > Namhyung
> >
>
> Yes, this is similar to what I want. Of course, I want the interval to
> be variable and can be set from the user program. I guess this cannot
> be achieved by current perf, right?

You could gather more data than you need and discard the ones you
don't? There's a json output to perf stat that may be useful for
scripting on top of this.

Fwiw, in the future writing python code to do this kind of thing
should be more straightforward:
https://lore.kernel.org/lkml/20241016042415.7552-17-irogers@google.com/

Thanks,
Ian