perf sampling count/freq/period strange bahavior

[Michael Petlan <mpetlan@redhat.com>]

Hello all...

We have encountered a strange behavior related to sampling. Generally
said, the number of samples is usually much lower than it should be.

Let's say, for load L, `perf stat -e instructions -- L` shows around
250,000 event hits. If I understand the concept of `-c` option for
`perf record`, with `-c 250` it should make the counter overflow and
generate a sample for every 250th event hit, so, it should result into
~1000 samples.

Instead I am getting something like ~62-150 samples, for example 124.
I have checked in dmesg that kernel has not throttled anything. When
running `perf report --stdio`, I see in the header that there were 0
samples lost (so the buffer was consumed fast enough). Additionally,
`perf report --stdio` tries to estimate the actual event count:

  Event count (approx.): 31000

(Which is because 250 * 124 = 31000, so perf itself considers this
formula (n_of_samples * sampling_rate = n_of_events) valid.)

But in reality, this is not the case, `perf stat`, as said, never
shows that low number of events for the same load.

What else is involved in the sampling rates and why the number of
samples is so much lower than it should be?

Thank you for ideas!

Michael