perf utility question: "Sky high iTLB-load-misses"

perf utility question: "Sky high iTLB-load-misses"

[Jan Pokorný]

[-- Attachment #1: Type: text/plain, Size: 679 bytes --]

Hello,

I was recently comparing the effect of using orphan section offloading
of some data in the libqb project[*] and basically run into the
same rather strange reports by perf as asked on StackOverflow:
https://stackoverflow.com/q/47107255
(except that I'm on Skylake processor).

Is iTLB-load-misses > 100% reported because of some deficiency
of the platform, a bug in perf, or an expected behaviour?

Thanks and cheers.
Please, keep me in the loop, I am not subscribed to the list.

[*] comparing log_callsite_bench_section{full,less} from
https://github.com/ClusterLabs/libqb/pull/266/commits/a2b96193ec9a677e18824736c363df35ee0c68dd    

-- 
Jan (Poki)

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: perf utility question: "Sky high iTLB-load-misses"

[Vince Weaver]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

On Wed, 8 Nov 2017, Jan Pokorný wrote:

> Is iTLB-load-misses > 100% reported because of some deficiency
> of the platform, a bug in perf, or an expected behaviour?

If you look into the kernel, the two events being measured are
	[ C(RESULT_ACCESS) ] = 0x2085,  /* ITLB_MISSES.STLB_HIT */
	[ C(RESULT_MISS)   ] = 0xe85,   /* ITLB_MISSES.WALK_COMPLETED */

If you look this up in Intel Vol3b you can see that the "access" metric
measures First level ITLB misses that hit in the Second-level TLB

Wheras the miss metric is a list of accesses from any level that caused a 
walk of the page tables.

So the events don't necessarily sound like they match up very well, so 
it's beleivable you will get odd results when trying to calculate 
percentages based on them.

Vince

Re: perf utility question: "Sky high iTLB-load-misses"

[Arnaldo Carvalho de Melo]

Em Wed, Nov 08, 2017 at 11:55:02AM -0500, Vince Weaver escreveu:
> On Wed, 8 Nov 2017, Jan Pokorný wrote:
> 
> > Is iTLB-load-misses > 100% reported because of some deficiency
> > of the platform, a bug in perf, or an expected behaviour?
> 
> If you look into the kernel, the two events being measured are
> 	[ C(RESULT_ACCESS) ] = 0x2085,  /* ITLB_MISSES.STLB_HIT */
> 	[ C(RESULT_MISS)   ] = 0xe85,   /* ITLB_MISSES.WALK_COMPLETED */
> 
> If you look this up in Intel Vol3b you can see that the "access" metric
> measures First level ITLB misses that hit in the Second-level TLB
> 
> Wheras the miss metric is a list of accesses from any level that caused a 
> walk of the page tables.
> 
> So the events don't necessarily sound like they match up very well, so 
> it's beleivable you will get odd results when trying to calculate 
> percentages based on them.

Andi, is this something you can help in figuring out a fix? Peter?

- Arnaldo

Re: perf utility question: "Sky high iTLB-load-misses"

[Andi Kleen]

On Thu, Nov 09, 2017 at 10:53:25AM -0300, Arnaldo Carvalho de Melo wrote:
> Em Wed, Nov 08, 2017 at 11:55:02AM -0500, Vince Weaver escreveu:
> > On Wed, 8 Nov 2017, Jan Pokorný wrote:
> > 
> > > Is iTLB-load-misses > 100% reported because of some deficiency
> > > of the platform, a bug in perf, or an expected behaviour?
> > 
> > If you look into the kernel, the two events being measured are
> > 	[ C(RESULT_ACCESS) ] = 0x2085,  /* ITLB_MISSES.STLB_HIT */
> > 	[ C(RESULT_MISS)   ] = 0xe85,   /* ITLB_MISSES.WALK_COMPLETED */
> > 
> > If you look this up in Intel Vol3b you can see that the "access" metric
> > measures First level ITLB misses that hit in the Second-level TLB
> > 
> > Wheras the miss metric is a list of accesses from any level that caused a 
> > walk of the page tables.
> > 
> > So the events don't necessarily sound like they match up very well, so 
> > it's beleivable you will get odd results when trying to calculate 
> > percentages based on them.
> 
> Andi, is this something you can help in figuring out a fix? Peter?

See "perf list itlb"

itlb access could be replaced with instructions retired,
but I suspect that is also not what you're looking for.

Only other alternative would be to disable the events.

-Andi

Re: perf utility question: "Sky high iTLB-load-misses"

[Jan Pokorný]

[-- Attachment #1: Type: text/plain, Size: 3134 bytes --]

On 09/11/17 06:38 -0800, Andi Kleen wrote:
> On Thu, Nov 09, 2017 at 10:53:25AM -0300, Arnaldo Carvalho de Melo wrote:
>> Em Wed, Nov 08, 2017 at 11:55:02AM -0500, Vince Weaver escreveu:
>>> On Wed, 8 Nov 2017, Jan Pokorný wrote:
>>> 
>>>> Is iTLB-load-misses > 100% reported because of some deficiency
>>>> of the platform, a bug in perf, or an expected behaviour?
>>> 
>>> If you look into the kernel, the two events being measured are
>>> 	[ C(RESULT_ACCESS) ] = 0x2085,  /* ITLB_MISSES.STLB_HIT */
>>> 	[ C(RESULT_MISS)   ] = 0xe85,   /* ITLB_MISSES.WALK_COMPLETED */
>>> 
>>> If you look this up in Intel Vol3b you can see that the "access" metric
>>> measures First level ITLB misses that hit in the Second-level TLB
>>> 
>>> Wheras the miss metric is a list of accesses from any level that caused a 
>>> walk of the page tables.
>>> 
>>> So the events don't necessarily sound like they match up very well, so 
>>> it's beleivable you will get odd results when trying to calculate 
>>> percentages based on them.
>> 
>> Andi, is this something you can help in figuring out a fix? Peter?
> 
> See "perf list itlb"
> 
> itlb access could be replaced with instructions retired,
> but I suspect that is also not what you're looking for.

Not sure I am getting the exact events right, but there are some
results concentrating just on these 3:

$ perf stat -r 100 -e itlb_misses.stlb_hit \
  -e frontend_retired.itlb_miss -e itlb_misses.walk_completed \
  --log-fd=1 ./log_callsite_bench_sectionless 2>/dev/null

> [...]
>          115,178      itlb_misses.stlb_hit                                          ( +-  2.53% )
>           46,199      frontend_retired.itlb_miss                                     ( +-  1.93% )
>          291,411      itlb_misses.walk_completed                                     ( +- 37.72% )
> [...]

but the ratio is flapping for particular runs, from three subsequent ones:

>           86,124      itlb_misses.stlb_hit
>           31,595      frontend_retired.itlb_miss
>            5,509      itlb_misses.walk_completed

vs.

>           97,578      itlb_misses.stlb_hit
>           33,673      frontend_retired.itlb_miss
>           15,961      itlb_misses.walk_completed

vs.

>          121,994      itlb_misses.stlb_hit
>           43,408      frontend_retired.itlb_miss
>          124,473      itlb_misses.walk_completed

I am by no mean knowledgeable in this field, didn't meant to make any real
conclusions based on the results, and I now suspect there's a reason behind
hiding iTLB-load-misses behind doubled -dd switch of the default execution
mode.  Was mainly curious why such obvious outliers on this Skylake machine
were observed.

> Only other alternative would be to disable the events.

Perhaps depends on the target audience, whether it's rather amateurs like me or
hardware-literate (I mean down to archicture manuals' level) experts that can
finely reason about all these metrics :)

(Nevertheless, I think keeping the basic mode easily consumable by the former
group is a noble goal.)

-- 
Jan (Poki)

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]

Re: perf utility question: "Sky high iTLB-load-misses"

[Bram Stolk]

Vince mentioned that two values are compared which do not relate well.
I would say the two possible approaches would then be:
(1) report absolute numbers only, no percentage.
(2) don't report it.

Re: perf utility question: "Sky high iTLB-load-misses"

[Vince Weaver]

On Thu, 9 Nov 2017, Bram Stolk wrote:

> Vince mentioned that two values are compared which do not relate well.
> I would say the two possible approaches would then be:
> (1) report absolute numbers only, no percentage.
> (2) don't report it.

Well they possibly don't relate well.  You would need an advanced degree 
in computer architecture and/or an NDA with Intel to know for sure what's 
really going on.

On modern procesors any event more complicated than "cycles" and 
"instructions" can be nearly impossible to understand, and even cycles has 
a lot of corner cases.

Vince