Re: perf utility question: "Sky high iTLB-load-misses"

["Jan Pokorný" <poki@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 3134 bytes --]

On 09/11/17 06:38 -0800, Andi Kleen wrote:
> On Thu, Nov 09, 2017 at 10:53:25AM -0300, Arnaldo Carvalho de Melo wrote:
>> Em Wed, Nov 08, 2017 at 11:55:02AM -0500, Vince Weaver escreveu:
>>> On Wed, 8 Nov 2017, Jan Pokorný wrote:
>>> 
>>>> Is iTLB-load-misses > 100% reported because of some deficiency
>>>> of the platform, a bug in perf, or an expected behaviour?
>>> 
>>> If you look into the kernel, the two events being measured are
>>> 	[ C(RESULT_ACCESS) ] = 0x2085,  /* ITLB_MISSES.STLB_HIT */
>>> 	[ C(RESULT_MISS)   ] = 0xe85,   /* ITLB_MISSES.WALK_COMPLETED */
>>> 
>>> If you look this up in Intel Vol3b you can see that the "access" metric
>>> measures First level ITLB misses that hit in the Second-level TLB
>>> 
>>> Wheras the miss metric is a list of accesses from any level that caused a 
>>> walk of the page tables.
>>> 
>>> So the events don't necessarily sound like they match up very well, so 
>>> it's beleivable you will get odd results when trying to calculate 
>>> percentages based on them.
>> 
>> Andi, is this something you can help in figuring out a fix? Peter?
> 
> See "perf list itlb"
> 
> itlb access could be replaced with instructions retired,
> but I suspect that is also not what you're looking for.

Not sure I am getting the exact events right, but there are some
results concentrating just on these 3:

$ perf stat -r 100 -e itlb_misses.stlb_hit \
  -e frontend_retired.itlb_miss -e itlb_misses.walk_completed \
  --log-fd=1 ./log_callsite_bench_sectionless 2>/dev/null

> [...]
>          115,178      itlb_misses.stlb_hit                                          ( +-  2.53% )
>           46,199      frontend_retired.itlb_miss                                     ( +-  1.93% )
>          291,411      itlb_misses.walk_completed                                     ( +- 37.72% )
> [...]

but the ratio is flapping for particular runs, from three subsequent ones:

>           86,124      itlb_misses.stlb_hit
>           31,595      frontend_retired.itlb_miss
>            5,509      itlb_misses.walk_completed

vs.

>           97,578      itlb_misses.stlb_hit
>           33,673      frontend_retired.itlb_miss
>           15,961      itlb_misses.walk_completed

vs.

>          121,994      itlb_misses.stlb_hit
>           43,408      frontend_retired.itlb_miss
>          124,473      itlb_misses.walk_completed

I am by no mean knowledgeable in this field, didn't meant to make any real
conclusions based on the results, and I now suspect there's a reason behind
hiding iTLB-load-misses behind doubled -dd switch of the default execution
mode.  Was mainly curious why such obvious outliers on this Skylake machine
were observed.

> Only other alternative would be to disable the events.

Perhaps depends on the target audience, whether it's rather amateurs like me or
hardware-literate (I mean down to archicture manuals' level) experts that can
finely reason about all these metrics :)

(Nevertheless, I think keeping the basic mode easily consumable by the former
group is a noble goal.)

-- 
Jan (Poki)

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]