Re: [PATCH v2 0/3] perf-stat: share hardware PMCs with BPF

[Jiri Olsa <jolsa@redhat.com>]

On Thu, Mar 18, 2021 at 03:52:51AM +0000, Song Liu wrote:
> 
> 
> > On Mar 17, 2021, at 6:11 AM, Arnaldo Carvalho de Melo <acme@kernel.org> wrote:
> > 
> > Em Wed, Mar 17, 2021 at 02:29:28PM +0900, Namhyung Kim escreveu:
> >> Hi Song,
> >> 
> >> On Wed, Mar 17, 2021 at 6:18 AM Song Liu <songliubraving@fb.com> wrote:
> >>> 
> >>> perf uses performance monitoring counters (PMCs) to monitor system
> >>> performance. The PMCs are limited hardware resources. For example,
> >>> Intel CPUs have 3x fixed PMCs and 4x programmable PMCs per cpu.
> >>> 
> >>> Modern data center systems use these PMCs in many different ways:
> >>> system level monitoring, (maybe nested) container level monitoring, per
> >>> process monitoring, profiling (in sample mode), etc. In some cases,
> >>> there are more active perf_events than available hardware PMCs. To allow
> >>> all perf_events to have a chance to run, it is necessary to do expensive
> >>> time multiplexing of events.
> >>> 
> >>> On the other hand, many monitoring tools count the common metrics (cycles,
> >>> instructions). It is a waste to have multiple tools create multiple
> >>> perf_events of "cycles" and occupy multiple PMCs.
> >> 
> >> Right, it'd be really helpful when the PMCs are frequently or mostly shared.
> >> But it'd also increase the overhead for uncontended cases as BPF programs
> >> need to run on every context switch.  Depending on the workload, it may
> >> cause a non-negligible performance impact.  So users should be aware of it.
> > 
> > Would be interesting to, humm, measure both cases to have a firm number
> > of the impact, how many instructions are added when sharing using
> > --bpf-counters?
> > 
> > I.e. compare the "expensive time multiplexing of events" with its
> > avoidance by using --bpf-counters.
> > 
> > Song, have you perfmormed such measurements?
> 
> I have got some measurements with perf-bench-sched-messaging:
> 
> The system: x86_64 with 23 cores (46 HT)
> 
> The perf-stat command:
> perf stat -e cycles,cycles,instructions,instructions,ref-cycles,ref-cycles <target, etc.>
> 
> The benchmark command and output:
> ./perf bench sched messaging -g 40 -l 50000 -t
> # Running 'sched/messaging' benchmark:
> # 20 sender and receiver threads per group
> # 40 groups == 1600 threads run
>      Total time: 10X.XXX [sec]
> 
> 
> I use the "Total time" as measurement, so smaller number is better. 
> 
> For each condition, I run the command 5 times, and took the median of 
> "Total time". 
> 
> Baseline (no perf-stat)			104.873 [sec]
> # global
> perf stat -a				107.887 [sec]
> perf stat -a --bpf-counters		106.071 [sec]
> # per task
> perf stat 				106.314 [sec]
> perf stat --bpf-counters 		105.965 [sec]
> # per cpu
> perf stat -C 1,3,5 			107.063 [sec]
> perf stat -C 1,3,5 --bpf-counters 	106.406 [sec]

I can't see why it's actualy faster than normal perf ;-)
would be worth to find out

jirka

> 
> From the data, --bpf-counters is slightly better than the regular event
> for all targets. I noticed that the results are not very stable. There 
> are a couple 108.xx runs in some of the conditions (w/ and w/o 
> --bpf-counters).
> 
> 
> I also measured the average runtime of the BPF programs, with 
> 
> 	sysctl kernel.bpf_stats_enabled=1
> 
> For each event, if we have one leader and two followers, the total run 
> time is about 340ns. IOW, 340ns for two perf-stat reading instructions, 
> 340ns for two perf-stat reading cycles, etc. 
> 
> Thanks,
> Song
>