* Re: [Questions] perf c2c: What's the current status of perf c2c?
2015-12-09 9:34 ` Peter Zijlstra
@ 2015-12-09 10:58 ` Peter Zijlstra
2015-12-09 11:09 ` Joe Mario
` (2 subsequent siblings)
3 siblings, 0 replies; 14+ messages in thread
From: Peter Zijlstra @ 2015-12-09 10:58 UTC (permalink / raw)
To: Jiri Olsa
Cc: Yunlong Song, dzickus, dsahern, fweisbec, jmario, efault, paulus,
rfowles, eranian,
acme@kernel.org >> Arnaldo Carvalho de Melo, mingo,
Linux Kernel Mailing List, Jiri Olsa,
wangnan0@huawei.com >> Wang Nan, fowles, Namhyung Kim, andi
On Wed, Dec 09, 2015 at 10:34:02AM +0100, Peter Zijlstra wrote:
> Which we can imagine could result in code like:
>
> foobar:
> mov (%rax), %rax # load bar::foo
> mov (%rax,1,4), %rax # load foo::val
>
>
> And DWARFs should know this, so by knowing the instruction we can know
> which load missed the cache.
>
> Once you have this information, you can use pahole like structure output
> and heat colour them or whatnot. Bright red if you miss lots etc..
>
> Now currently this is possible but a bit of work because the DWARF
> annotations are not exactly following these data types, that is you
> might need to decode previous instructions and infer some bits.
To clarify, current DWARFs might only know the argument to foobar is of
type struct bar *, and we'll have to infer the rest.
> I think Stephane was working with GCC people to allow more/better DWARF
> annotations and allow easier retrieval of this information.
And even if that gets sorted, it might still make sense to implement the
hard case as per the above, because it'll take a long time before
everything is build with the fancy new GCC/dwarf output.
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [Questions] perf c2c: What's the current status of perf c2c?
2015-12-09 9:34 ` Peter Zijlstra
2015-12-09 10:58 ` Peter Zijlstra
@ 2015-12-09 11:09 ` Joe Mario
[not found] ` <1891297138.17374838.1449658964938.JavaMail.zimbra@redhat.com>
2015-12-09 16:58 ` Andi Kleen
3 siblings, 0 replies; 14+ messages in thread
From: Joe Mario @ 2015-12-09 11:09 UTC (permalink / raw)
To: Peter Zijlstra, Jiri Olsa
Cc: Yunlong Song, dzickus, dsahern, fweisbec, efault, paulus, rfowles,
eranian, acme@kernel.org >> Arnaldo Carvalho de Melo, mingo,
Linux Kernel Mailing List, Jiri Olsa,
wangnan0@huawei.com >> Wang Nan, fowles, Namhyung Kim, andi
[RESEND - this time w/o html junk]
On 12/09/2015 04:34 AM, Peter Zijlstra wrote:
> On Wed, Dec 09, 2015 at 09:04:40AM +0100, Jiri Olsa wrote:
>> On Wed, Dec 09, 2015 at 12:06:44PM +0800, Yunlong Song wrote:
>>> Hi, Don,
>>> I am interested in the perf c2c tool, which is introduced in: http://lwn.net/Articles/588866/
>>> However, I found that this tool has not been applied to the mainline tree of perf, Why? It was first
>>> introduced in Feb. 2014. What's its current status now? Does it have a new version or a repository
>>> somewhere else? And does it support Haswell?
>>
>> hi,
>> not sure Don made any progress on this field, but I'm having
>> his c2c sources rebased current perf sources ATM.
>>
<snip>
> So I never really liked the c2c tool because it was so narrowly
> focussed, it only works on NUMA thingies IIRC.
>
> I would much rather see a tool that uses PEBS events and does a dwarf
> decode of the exact instruction's data reference -- without relying on
> data address bits.
Peter:
Yes, that would be a great enhancement, but is it any reason to hold up the current implementation?
I've been using "perf c2c" heavily with customers over the past two years and after they see what it can do, their first question is why it hasn't been checked in upstream yet.
Joe
^ permalink raw reply [flat|nested] 14+ messages in thread[parent not found: <1891297138.17374838.1449658964938.JavaMail.zimbra@redhat.com>]
* Re: [Questions] perf c2c: What's the current status of perf c2c?
[not found] ` <1891297138.17374838.1449658964938.JavaMail.zimbra@redhat.com>
@ 2015-12-09 14:03 ` Peter Zijlstra
0 siblings, 0 replies; 14+ messages in thread
From: Peter Zijlstra @ 2015-12-09 14:03 UTC (permalink / raw)
To: Joe Mario
Cc: Jiri Olsa, Yunlong Song, dzickus, dsahern, fweisbec, efault,
paulus, rfowles, eranian,
acme@kernel.org >> Arnaldo Carvalho de Melo, mingo,
Linux Kernel Mailing List, Jiri Olsa,
wangnan0@huawei.com >> Wang Nan, fowles, Namhyung Kim, andi
On Wed, Dec 09, 2015 at 06:02:44AM -0500, Joe Mario wrote:
> Yes, that would be a great enhancement,
This is hardly new though; I've outlined the very same the first time
the c2c thing got mentioned.
> but is it any reason to hold up the current implementation?
I just wonder how much of c2c is still useful once we get it done
proper. And once such a tool is out there, its hard to kill, leaving us
with a maintenance burden we could do without.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Questions] perf c2c: What's the current status of perf c2c?
2015-12-09 9:34 ` Peter Zijlstra
` (2 preceding siblings ...)
[not found] ` <1891297138.17374838.1449658964938.JavaMail.zimbra@redhat.com>
@ 2015-12-09 16:58 ` Andi Kleen
2015-12-09 17:15 ` Stephane Eranian
3 siblings, 1 reply; 14+ messages in thread
From: Andi Kleen @ 2015-12-09 16:58 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Jiri Olsa, Yunlong Song, dzickus, dsahern, fweisbec, jmario,
efault, paulus, rfowles, eranian,
acme@kernel.org >> Arnaldo Carvalho de Melo, mingo,
Linux Kernel Mailing List, Jiri Olsa,
wangnan0@huawei.com >> Wang Nan, fowles, Namhyung Kim, andi
> > the plan for me is to to use it some more to prove it's useful
> > and kick it to be merged with perf at some point
>
> So I never really liked the c2c tool because it was so narrowly
> focussed, it only works on NUMA thingies IIRC.
It should work on all systems with an Intel Core (not Atom)
However it was never clear to me if the tool was any better
than simply sampling for
mem_load_uops_l3_hit_retired.xsnp_hitm:pp (local socket)
mem_load_uops_l3_miss_retired.remote_hitm:pp (remote socket)
which gives you instructions that reference bouncing cache lines.
-Andi
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [Questions] perf c2c: What's the current status of perf c2c?
2015-12-09 16:58 ` Andi Kleen
@ 2015-12-09 17:15 ` Stephane Eranian
2015-12-09 17:21 ` Andi Kleen
2015-12-09 20:41 ` Joe Mario
0 siblings, 2 replies; 14+ messages in thread
From: Stephane Eranian @ 2015-12-09 17:15 UTC (permalink / raw)
To: Andi Kleen
Cc: Peter Zijlstra, Jiri Olsa, Yunlong Song, Don Zickus, David Ahern,
Frédéric Weisbecker, Joe Mario, Mike Galbraith,
Paul Mackerras, Richard Fowles,
acme@kernel.org >> Arnaldo Carvalho de Melo,
mingo@redhat.com, Linux Kernel Mailing List, Jiri Olsa,
wangnan0@huawei.com >> Wang Nan, Richard Fowles,
Namhyung Kim
Hi,
On Wed, Dec 9, 2015 at 8:58 AM, Andi Kleen <andi@firstfloor.org> wrote:
>> > the plan for me is to to use it some more to prove it's useful
>> > and kick it to be merged with perf at some point
>>
>> So I never really liked the c2c tool because it was so narrowly
>> focussed, it only works on NUMA thingies IIRC.
>
> It should work on all systems with an Intel Core (not Atom)
>
> However it was never clear to me if the tool was any better
> than simply sampling for
>
> mem_load_uops_l3_hit_retired.xsnp_hitm:pp (local socket)
> mem_load_uops_l3_miss_retired.remote_hitm:pp (remote socket)
>
> which gives you instructions that reference bouncing cache lines.
>
If I recall the c2c tool is giving you more than the bouncing line. It shows you
the offset inside the line and the participating CPUs.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Questions] perf c2c: What's the current status of perf c2c?
2015-12-09 17:15 ` Stephane Eranian
@ 2015-12-09 17:21 ` Andi Kleen
2015-12-09 19:48 ` Stephane Eranian
2015-12-09 20:41 ` Joe Mario
1 sibling, 1 reply; 14+ messages in thread
From: Andi Kleen @ 2015-12-09 17:21 UTC (permalink / raw)
To: Stephane Eranian
Cc: Andi Kleen, Peter Zijlstra, Jiri Olsa, Yunlong Song, Don Zickus,
David Ahern, Frédéric Weisbecker, Joe Mario,
Mike Galbraith, Paul Mackerras, Richard Fowles,
acme@kernel.org >> Arnaldo Carvalho de Melo,
mingo@redhat.com, Linux Kernel Mailing List, Jiri Olsa,
wangnan0@huawei.com >> Wang Nan, Richard Fowles,
Namhyung Kim
> If I recall the c2c tool is giving you more than the bouncing line. It shows you
> the offset inside the line and the participating CPUs.
On Haswell and later you could get the same with the normal address
reporting. The events above support DLA.
-Andi
--
ak@linux.intel.com -- Speaking for myself only.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Questions] perf c2c: What's the current status of perf c2c?
2015-12-09 17:21 ` Andi Kleen
@ 2015-12-09 19:48 ` Stephane Eranian
0 siblings, 0 replies; 14+ messages in thread
From: Stephane Eranian @ 2015-12-09 19:48 UTC (permalink / raw)
To: Andi Kleen
Cc: Peter Zijlstra, Jiri Olsa, Yunlong Song, Don Zickus, David Ahern,
Frédéric Weisbecker, Joe Mario, Mike Galbraith,
Paul Mackerras, Richard Fowles,
acme@kernel.org >> Arnaldo Carvalho de Melo,
mingo@redhat.com, Linux Kernel Mailing List, Jiri Olsa,
wangnan0@huawei.com >> Wang Nan, Richard Fowles,
Namhyung Kim
On Wed, Dec 9, 2015 at 9:21 AM, Andi Kleen <andi@firstfloor.org> wrote:
>> If I recall the c2c tool is giving you more than the bouncing line. It shows you
>> the offset inside the line and the participating CPUs.
>
> On Haswell and later you could get the same with the normal address
> reporting. The events above support DLA.
>
I know the events track the condition better than just with the
latency threshold.
I think what it boils down to is not so much the PMU side but rather
the tool side.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [Questions] perf c2c: What's the current status of perf c2c?
2015-12-09 17:15 ` Stephane Eranian
2015-12-09 17:21 ` Andi Kleen
@ 2015-12-09 20:41 ` Joe Mario
2015-12-10 2:36 ` Yunlong Song
1 sibling, 1 reply; 14+ messages in thread
From: Joe Mario @ 2015-12-09 20:41 UTC (permalink / raw)
To: Stephane Eranian, Andi Kleen
Cc: Peter Zijlstra, Jiri Olsa, Yunlong Song, Don Zickus, David Ahern,
Frédéric Weisbecker, Mike Galbraith, Paul Mackerras,
acme@kernel.org >> Arnaldo Carvalho de Melo,
mingo@redhat.com, Linux Kernel Mailing List, Jiri Olsa,
wangnan0@huawei.com >> Wang Nan, Richard Fowles,
Namhyung Kim
On 12/09/2015 12:15 PM, Stephane Eranian wrote:
> If I recall the c2c tool is giving you more than the bouncing line. It shows you
> the offset inside the line and the participating CPUs.
Correct. It shows much more than the bouncing line.
Appended below is the output for running "perf c2c" on a 4-node system
running a multi-thread version of linpack. I've annotated it show describe
what some of the fields mean.
Note, your screen output has to be set pretty wide to read it. For those
not wanting to read it in their mailer, grab it from:
http://people.redhat.com/jmario/perf_c2c/perf_c2c_annotated_output.txt
Let me know of any questions. My annotations begin with "// ".
Joe
--------------------------------------------------------------------
// Perf c2c output from a linpack run.
// Set screen wide to view this.
// Here's a breakdown of all loads and stores sampled.
// It shows where they hit and missed.
=================================================
Trace Event Information
=================================================
Total records : 3229269
Locked Load/Store Operations : 64420
Load Operations : 1153827
Loads - uncacheable : 11
Loads - IO : 0
Loads - Miss : 11002
Loads - no mapping : 200
Load Fill Buffer Hit : 355942
Load L1D hit : 361303
Load L2D hit : 46792
Load LLC hit : 274265
Load Local HITM : 18647
Load Remote HITM : 55225
Load Remote HIT : 8917
Load Local DRAM : 11895
Load Remote DRAM : 28275
Load MESI State Exclusive : 40170
Load MESI State Shared : 0
Load LLC Misses : 104312
LLC Misses to Local DRAM : 11.4%
LLC Misses to Remote DRAM : 27.1%
LLC Misses to Remote cache (HIT) : 8.5%
LLC Misses to Remote cache (HITM) : 52.9%
Store Operations : 2069610
Store - uncacheable : 0
Store - no mapping : 3538
Store L1D Hit : 1889168
Store L1D Miss : 176904
No Page Map Rejects : 102146
Unable to parse data source : 5832
// Table showing activity on the shared cache lines.
=================================================
Global Shared Cache Line Event Information
=================================================
Total Shared Cache Lines : 14213
Load HITs on shared lines : 426101
Fill Buffer Hits on shared lines : 159230
L1D hits on shared lines : 77377
L2D hits on shared lines : 28897
LLC hits on shared lines : 94709
Locked Access on shared lines : 45704
Store HITs on shared lines : 198050
Store L1D hits on shared lines : 188136
Total Merged records : 271778
// In the next table, for each of the 10 hottest cache lines, breakout all the activity for that line.
// The sorting is done by remote hitm percentage, defined as loads that hit in a remote node's modified cacheline.
================================================================================================================================================================================================================
Shared Data Cache Line Table
Total %All Total ---- Core Load Hit ---- -- LLC Load Hit -- ----- LLC Load Hitm ----- -- Load Dram -- LLC ---- Store Reference ----
Index Phys Adrs Records Ld Miss %hitm Loads FB L1D L2D Lcl Rmt Total Lcl Rmt Lcl Rmt Ld Miss Total L1Hit L1Miss
================================================================================================================================================================================================================
0 0x7f9d8833bf80 12402 1.87% 3.54% 11748 6582 2259 72 24 20 2788 835 1953 2 1 1976 654 648 6
1 0x85fd0f0c5c0 5593 0.35% 0.65% 3813 797 2131 153 171 32 457 97 360 20 52 464 1780 1780 0
2 0x81fd0ddfd00 2174 0.21% 0.40% 1352 318 558 50 75 13 315 94 221 9 14 257 822 822 0
3 0x85fd0f0c8c0 1854 0.21% 0.39% 1436 184 655 153 92 28 279 65 214 13 32 287 418 418 0
4 0x83fffb17280 735 0.16% 0.31% 543 3 287 12 15 2 216 44 172 2 6 182 192 192 0
5 0x7fff81db1e00 3641 0.16% 0.30% 3331 1111 849 624 384 83 228 62 166 15 37 301 310 310 0
6 0x83fff857280 755 0.15% 0.29% 563 3 316 13 24 0 201 43 158 3 3 164 192 192 0
7 0x85fff917280 801 0.15% 0.28% 648 1 409 19 19 1 198 44 154 1 0 156 153 153 0
8 0x85fffc17280 767 0.14% 0.27% 664 4 403 27 15 2 205 54 151 2 6 161 103 103 0
9 0x85fffbf7280 718 0.14% 0.27% 495 5 262 15 14 0 193 42 151 1 5 157 223 223 0
======================================================================================================================================================================================================================================================================
Shared Cache Line Distribution Pareto
---- All ---- -- Shared -- ---- HITM ---- Load Inst Execute Latency Shared Data Participants
Data Misses Data Misses Remote Local -- Store Refs --
---- cycles ---- cpu
Num %dist %cumm %dist %cumm LLCmiss LLChit L1 hit L1 Miss Data Address Pid Tid Inst Address median mean CV cnt Symbol Object Node{cpus %hitms %stores} Node{cpus %hitms %stores} ...
======================================================================================================================================================================================================================================================================
-----------------------------------------------------------------------------------------------
0 1.9% 1.9% 3.5% 3.5% 1953 835 648 6 0x7f9d8833bf80 146560
-----------------------------------------------------------------------------------------------
0.3% 0.4% 97.2% 0.0% 0x10 146560 *** 0x9969fa 2036 2265 12.4% 80 __kmp_acquire_queuing_lock_wi xlinpack_xeon64 0{20 16.7% 25.2%} 1{21 33.3% 24.0%} 2{20 33.3% 25.7%} 3{19 16.7% 25.1%}
88.9% 89.1% 0.0% 0.0% 0x10 146560 *** 0x996dfc 304 349 0.4% 79 __kmp_release_queuing_lock_wi xlinpack_xeon64 0{20 23.7% n/a} 1{21 24.5% n/a} 2{19 26.8% n/a} 3{19 25.0% n/a}
0.1% 0.0% 0.0% 0.0% 0x14 146560 146608 0x9969ac 410 410 0.0% 1 __kmp_acquire_queuing_lock_wi xlinpack_xeon64 2{ 1 100.0% n/a}
10.7% 10.5% 0.0% 0.0% 0x14 146560 *** 0x996def 303 352 1.2% 69 __kmp_release_queuing_lock_wi xlinpack_xeon64 0{18 21.5% n/a} 1{18 27.8% n/a} 2{18 31.6% n/a} 3{15 19.1% n/a}
0.0% 0.0% 2.8% 100.0% 0x14 146560 *** 0x996e54 n/a n/a n/a 19 __kmp_release_queuing_lock_wi xlinpack_xeon64 0{ 4 n/a 25.0%} 1{ 6 n/a 29.2%} 2{ 6 n/a 33.3%} 3{ 3 n/a 12.5%}
// Here's where the data gets interesting. Look at cacheline 0 above. The cacheline at data address 0x7f9d8833bf80 had the most contention.
//
// There were 1953 loads to that cacheline that hit in "remote-node modified cachelines". That's why this is the hottest cacheline.
// There were 835 loads that hit in a local modified cacheline. Also noted are the number of stores that both hit and missed the L1 cache.
// All accesses to that cacheline occurred at offsets 0x10 and 0x14.
// All accesses were done by one pid (146560), which was the Pid for linpack.
// I chose to display "***" when multiple thread ids (Tids) were involved in the same entry. Individual Tids can be displayed, but it makes for a long output.
// The instruction address of the load/store is displayed, along with the function and object name.
// Where loads are involved, the median and mean load latency cycles are displayed. Dumping the c2c raw records shows you individual worst offenders. It's not uncommon to see loads taking tens of thousands of cycles to complete when heavy contention is involved.
// The "cpu cnt" column shows how many individual cpus had samples contributing to a row of data. In the first row above, there were samples from 80 cpus for that row.
// The "Shared Data Participants" columns show the nodes the samples occured on, the number of cpus in each node that samples came from for that row, and for each node, the percentage of hitms and stores that came from that node.
//
// The above data shows how that hot cacheline is being concurrently read and written from cpus across all four nodes on the system. It's then easy to disassemble the binary (with line info) to identify the source code line numbers causing the false sharing.
-----------------------------------------------------------------------------------------------
1 0.3% 2.2% 0.7% 4.2% 360 97 1780 0 0xffff885fd0f0a5c0 ***
-----------------------------------------------------------------------------------------------
21.1% 29.9% 0.0% 0.0% 0x18 146560 *** 0xffffffff81196f40 1193 2404 13.9% 44 handle_mm_fault [kernel.kallsyms] 0{ 4 9.2% n/a} 1{12 27.6% n/a} 2{17 40.8% n/a} 3{11 22.4% n/a}
25.0% 21.6% 0.0% 0.0% 0x18 146560 *** 0xffffffff811a16d9 377 754 16.8% 43 mm_find_pmd [kernel.kallsyms] 0{11 23.3% n/a} 1{12 16.7% n/a} 2{11 41.1% n/a} 3{ 9 18.9% n/a}
17.2% 12.4% 0.0% 0.0% 0x18 0 0 0xffffffff8163accb 1994 2467 11.0% 58 __schedule [kernel.kallsyms] 0{16 24.2% n/a} 1{13 21.0% n/a} 2{13 21.0% n/a} 3{16 33.9% n/a}
3.9% 2.1% 24.1% 0.0% 0x24 *** *** 0xffffffff810b41ce 2066 3398 30.0% 136 finish_task_switch [kernel.kallsyms] 0{32 35.7% 23.5%} 1{36 14.3% 26.8%} 2{36 42.9% 26.1%} 3{32 7.1% 23.5%}
13.9% 11.3% 29.6% 0.0% 0x24 146560 *** 0xffffffff8163aa37 7230 7026 7.8% 139 __schedule [kernel.kallsyms] 0{34 26.0% 18.8%} 1{36 24.0% 28.5%} 2{36 34.0% 27.9%} 3{33 16.0% 24.7%}
0.0% 0.0% 0.1% 0.0% 0x24 146560 146665 0xffffffff8163aa3c n/a n/a n/a 1 __schedule [kernel.kallsyms] 2{ 1 n/a 100.0%}
5.3% 3.1% 0.0% 0.0% 0x38 146560 *** 0xffffffff810aa959 406 971 22.2% 20 down_read_trylock [kernel.kallsyms] 0{ 3 15.8% n/a} 1{ 9 52.6% n/a} 2{ 4 15.8% n/a} 3{ 4 15.8% n/a}
12.5% 15.5% 18.0% 0.0% 0x38 146560 *** 0xffffffff810aa965 2342 3845 13.9% 82 down_read_trylock [kernel.kallsyms] 0{21 15.6% 16.6%} 1{22 24.4% 31.6%} 2{19 35.6% 27.2%} 3{20 24.4% 24.7%}
1.1% 4.1% 28.3% 0.0% 0x38 146560 *** 0xffffffff810aa9c3 1376 1462 20.1% 87 up_read [kernel.kallsyms] 0{21 50.0% 16.7%} 1{23 50.0% 31.3%} 2{23 0.0% 28.6%} 3{20 0.0% 23.4%}
-----------------------------------------------------------------------------------------------
2 0.2% 2.4% 0.4% 4.6% 221 94 822 0 0xffff881fd0dddd00 146560
-----------------------------------------------------------------------------------------------
2.7% 2.1% 0.0% 0.0% 0x00 146560 *** 0xffffffff811a31ff 328 550 25.9% 8 page_lock_anon_vma_read [kernel.kallsyms] 0{ 2 16.7% n/a} 1{ 1 16.7% n/a} 2{ 3 33.3% n/a} 3{ 2 33.3% n/a}
9.5% 9.6% 0.0% 0.0% 0x00 146560 *** 0xffffffff811a36f5 429 814 19.2% 24 try_to_unmap_anon [kernel.kallsyms] 0{ 5 19.0% n/a} 1{ 9 42.9% n/a} 2{ 5 23.8% n/a} 3{ 5 14.3% n/a}
30.8% 27.7% 0.0% 0.0% 0x00 146560 *** 0xffffffff811a39f0 373 541 9.4% 37 rmap_walk [kernel.kallsyms] 0{12 32.4% n/a} 1{ 9 20.6% n/a} 2{ 8 25.0% n/a} 3{ 8 22.1% n/a}
24.0% 20.2% 0.0% 0.0% 0x00 146560 *** 0xffffffff811a3a6a 424 692 11.0% 35 rmap_walk [kernel.kallsyms] 0{ 9 26.4% n/a} 1{10 24.5% n/a} 2{ 9 32.1% n/a} 3{ 7 17.0% n/a}
0.9% 0.0% 0.0% 0.0% 0x08 146560 *** 0xffffffff810aa959 295 295 4.4% 2 down_read_trylock [kernel.kallsyms] 0{ 2 100.0% n/a}
2.3% 2.1% 38.3% 0.0% 0x08 146560 *** 0xffffffff810aa965 537 821 25.9% 48 down_read_trylock [kernel.kallsyms] 0{15 40.0% 25.4%} 1{12 0.0% 28.6%} 2{11 0.0% 26.7%} 3{10 60.0% 19.4%}
3.2% 4.3% 15.0% 0.0% 0x08 146560 *** 0xffffffff810aa9c3 746 834 17.1% 44 up_read [kernel.kallsyms] 0{13 14.3% 23.6%} 1{12 14.3% 23.6%} 2{10 28.6% 26.0%} 3{ 9 42.9% 26.8%}
4.1% 3.2% 22.9% 0.0% 0x08 146560 *** 0xffffffff8163a0a5 950 934 15.7% 46 down_read [kernel.kallsyms] 0{14 44.4% 31.9%} 1{13 22.2% 22.3%} 2{11 22.2% 25.5%} 3{ 8 11.1% 20.2%}
12.7% 17.0% 0.0% 0.0% 0x28 146560 *** 0xffffffff811a3140 337 827 26.1% 28 page_get_anon_vma [kernel.kallsyms] 0{ 7 21.4% n/a} 1{ 9 42.9% n/a} 2{ 6 10.7% n/a} 3{ 6 25.0% n/a}
2.3% 2.1% 17.0% 0.0% 0x28 146560 *** 0xffffffff811a3151 1288 1870 29.9% 43 page_get_anon_vma [kernel.kallsyms] 0{13 40.0% 33.6%} 1{11 20.0% 19.3%} 2{10 0.0% 27.1%} 3{ 9 40.0% 20.0%}
0.0% 0.0% 0.1% 0.0% 0x28 146560 146602 0xffffffff811a31aa n/a n/a n/a 1 page_get_anon_vma [kernel.kallsyms] 1{ 1 n/a 100.0%}
2.7% 3.2% 6.7% 0.0% 0x28 146560 *** 0xffffffff811c77b4 843 1950 43.3% 35 migrate_pages [kernel.kallsyms] 0{10 16.7% 36.4%} 1{ 6 16.7% 18.2%} 2{10 16.7% 21.8%} 3{ 9 50.0% 23.6%}
5.0% 8.5% 0.0% 0.0% 0x30 146560 *** 0xffffffff81190b15 336 561 19.6% 15 anon_vma_interval_tree_iter_f [kernel.kallsyms] 0{ 5 36.4% n/a} 1{ 4 9.1% n/a} 2{ 3 36.4% n/a} 3{ 3 18.2% n/a}
-----------------------------------------------------------------------------------------------
3 0.2% 2.6% 0.4% 5.0% 214 65 418 0 0xffff885fd0f0a8c0 ***
-----------------------------------------------------------------------------------------------
7.9% 13.8% 0.0% 0.0% 0x08 0 0 0xffffffff81065c26 430 724 17.2% 20 leave_mm [kernel.kallsyms] 0{10 58.8% n/a} 1{10 41.2% n/a}
0.5% 0.0% 12.7% 0.0% 0x08 0 0 0xffffffff81065c38 1343 1343 0.0% 25 leave_mm [kernel.kallsyms] 0{15 0.0% 54.7%} 1{10 100.0% 45.3%}
1.9% 1.5% 5.5% 0.0% 0x08 0 0 0xffffffff810b0c74 1902 3074 32.9% 18 cpumask_set_cpu [kernel.kallsyms] 0{ 9 25.0% 60.9%} 1{ 9 75.0% 39.1%}
11.2% 9.2% 0.0% 0.0% 0x08 *** *** 0xffffffff8163acb5 401 554 15.9% 20 __schedule [kernel.kallsyms] 0{13 58.3% n/a} 1{ 7 41.7% n/a}
6.5% 6.2% 0.0% 0.0% 0x0c 0 0 0xffffffff81065c26 310 540 22.2% 14 leave_mm [kernel.kallsyms] 1{ 2 14.3% n/a} 2{10 78.6% n/a} 3{ 2 7.1% n/a}
0.5% 0.0% 11.2% 0.0% 0x0c 0 0 0xffffffff81065c38 3638 3638 0.0% 25 leave_mm [kernel.kallsyms] 1{ 1 0.0% 4.3%} 2{16 100.0% 68.1%} 3{ 8 0.0% 27.7%}
2.8% 4.6% 3.8% 0.0% 0x0c 0 0 0xffffffff810b0c74 1774 2290 25.7% 20 cpumask_set_cpu [kernel.kallsyms] 1{ 3 0.0% 18.8%} 2{12 100.0% 50.0%} 3{ 5 0.0% 31.2%}
15.4% 9.2% 0.0% 0.0% 0x0c *** *** 0xffffffff8163acb5 415 768 23.1% 21 __schedule [kernel.kallsyms] 1{ 2 3.0% n/a} 2{12 63.6% n/a} 3{ 7 33.3% n/a}
10.7% 15.4% 0.0% 0.0% 0x10 0 0 0xffffffff81065c26 428 619 14.2% 18 leave_mm [kernel.kallsyms] 0{13 65.2% n/a} 1{ 3 13.0% n/a} 3{ 2 21.7% n/a}
0.0% 0.0% 14.1% 0.0% 0x10 0 0 0xffffffff81065c38 n/a n/a n/a 27 leave_mm [kernel.kallsyms] 0{15 n/a 55.9%} 1{ 5 n/a 18.6%} 3{ 7 n/a 25.4%}
0.0% 4.6% 6.9% 0.0% 0x10 0 0 0xffffffff810b0c74 n/a n/a n/a 18 cpumask_set_cpu [kernel.kallsyms] 0{10 n/a 41.4%} 1{ 5 n/a 41.4%} 3{ 3 n/a 17.2%}
9.3% 7.7% 0.0% 0.0% 0x10 *** *** 0xffffffff8163acb5 404 640 25.5% 18 __schedule [kernel.kallsyms] 0{ 6 35.0% n/a} 1{ 5 25.0% n/a} 3{ 7 40.0% n/a}
13.1% 9.2% 0.0% 0.0% 0x14 0 0 0xffffffff81065c26 340 458 9.8% 21 leave_mm [kernel.kallsyms] 1{ 7 32.1% n/a} 2{13 64.3% n/a} 3{ 1 3.6% n/a}
0.5% 0.0% 20.3% 0.0% 0x14 0 0 0xffffffff81065c38 2662 2662 0.0% 27 leave_mm [kernel.kallsyms] 1{11 0.0% 32.9%} 2{14 100.0% 57.6%} 3{ 2 0.0% 9.4%}
1.9% 0.0% 8.9% 0.0% 0x14 0 0 0xffffffff810b0c74 2892 3086 28.5% 20 cpumask_set_cpu [kernel.kallsyms] 1{ 6 0.0% 32.4%} 2{13 100.0% 64.9%} 3{ 1 0.0% 2.7%}
0.0% 0.0% 0.2% 0.0% 0x14 0 0 0xffffffff810b0c78 n/a n/a n/a 1 cpumask_set_cpu [kernel.kallsyms] 2{ 1 n/a 100.0%}
8.9% 4.6% 0.0% 0.0% 0x14 0 0 0xffffffff8163acb5 437 1041 20.5% 16 __schedule [kernel.kallsyms] 1{ 8 42.1% n/a} 2{ 7 52.6% n/a} 3{ 1 5.3% n/a}
3.3% 7.7% 0.0% 0.0% 0x18 0 0 0xffffffff81065c26 407 450 15.8% 8 leave_mm [kernel.kallsyms] 3{ 8 100.0% n/a}
0.0% 0.0% 12.4% 0.0% 0x18 0 0 0xffffffff81065c38 n/a n/a n/a 16 leave_mm [kernel.kallsyms] 3{16 n/a 100.0%}
0.9% 1.5% 3.8% 0.0% 0x18 0 0 0xffffffff810b0c74 1007 1230 18.2% 12 cpumask_set_cpu [kernel.kallsyms] 3{12 100.0% 100.0%}
4.7% 4.6% 0.0% 0.0% 0x18 *** *** 0xffffffff8163acb5 342 629 25.1% 10 __schedule [kernel.kallsyms] 3{10 100.0% n/a}
-----------------------------------------------------------------------------------------------
4 0.2% 2.8% 0.3% 5.3% 172 44 192 0 0xffff883fffb15280 146560
-----------------------------------------------------------------------------------------------
0.6% 0.0% 85.4% 0.0% 0x00 146560 *** 0xffffffff810e6e13 298 298 0.0% 1 flush_smp_call_function_queue [kernel.kallsyms] 1{ 1 100.0% 100.0%}
98.8% 97.7% 0.0% 0.0% 0x00 146560 *** 0xffffffff813067d8 291 363 4.9% 44 llist_add_batch [kernel.kallsyms] 0{13 27.1% n/a} 1{10 13.5% n/a} 2{12 29.4% n/a} 3{ 9 30.0% n/a}
0.6% 2.3% 14.6% 0.0% 0x00 146560 *** 0xffffffff813067e1 892 1255 0.0% 21 llist_add_batch [kernel.kallsyms] 0{ 7 0.0% 35.7%} 1{ 2 0.0% 7.1%} 2{ 6 100.0% 28.6%} 3{ 6 0.0% 28.6%}
-----------------------------------------------------------------------------------------------
5 0.2% 3.0% 0.3% 5.6% 166 62 310 0 0xffffffff81dafe00 ***
-----------------------------------------------------------------------------------------------
0.6% 0.0% 0.0% 0.0% 0x00 146560 146696 0xffffffff810bcc6c 1363 1363 0.0% 1 update_cfs_rq_h_load [kernel.kallsyms] 1{ 1 100.0% n/a}
22.3% 21.0% 0.0% 0.0% 0x00 146560 *** 0xffffffff810bdf56 1421 2209 15.8% 38 select_task_rq_fair [kernel.kallsyms] 0{13 32.4% n/a} 1{ 7 16.2% n/a} 2{12 35.1% n/a} 3{ 6 16.2% n/a}
4.2% 6.5% 0.0% 0.0% 0x00 146560 *** 0xffffffff810bdfaf 1500 5357 64.4% 10 select_task_rq_fair [kernel.kallsyms] 0{ 1 0.0% n/a} 1{ 4 71.4% n/a} 2{ 2 0.0% n/a} 3{ 3 28.6% n/a}
6.0% 11.3% 0.0% 0.0% 0x00 *** *** 0xffffffff810befcf 3172 2506 25.8% 15 update_blocked_averages [kernel.kallsyms] 0{ 2 10.0% n/a} 1{ 4 20.0% n/a} 2{ 4 30.0% n/a} 3{ 5 40.0% n/a}
54.8% 54.8% 0.0% 0.0% 0x00 *** *** 0xffffffff810c1963 1884 2815 13.1% 81 update_cfs_shares [kernel.kallsyms] 0{16 23.1% n/a} 1{21 24.2% n/a} 2{23 27.5% n/a} 3{21 25.3% n/a}
1.2% 0.0% 0.0% 0.0% 0x08 146560 *** 0xffffffff810b5fb7 1365 1365 51.4% 2 set_task_cpu [kernel.kallsyms] 0{ 1 50.0% n/a} 1{ 1 50.0% n/a}
3.0% 3.2% 0.0% 0.0% 0x08 146560 *** 0xffffffff810bf901 815 1186 45.9% 6 can_migrate_task [kernel.kallsyms] 0{ 2 20.0% n/a} 1{ 2 40.0% n/a} 2{ 2 40.0% n/a}
3.6% 0.0% 44.8% 0.0% 0x20 *** *** 0xffffffff810bcf6f 5364 6967 41.5% 94 update_cfs_rq_blocked_load [kernel.kallsyms] 0{22 33.3% 17.3%} 1{27 0.0% 36.0%} 2{23 16.7% 23.0%} 3{22 50.0% 23.7%}
0.0% 0.0% 1.0% 0.0% 0x28 *** *** 0xffffffff810bf3a9 n/a n/a n/a 3 update_blocked_averages [kernel.kallsyms] 0{ 1 n/a 33.3%} 3{ 2 n/a 66.7%}
0.0% 0.0% 0.3% 0.0% 0x28 0 0 0xffffffff810c102a n/a n/a n/a 1 idle_enter_fair [kernel.kallsyms] 1{ 1 n/a 100.0%}
0.6% 3.2% 29.7% 0.0% 0x28 146560 *** 0xffffffff810c2455 1647 1647 0.0% 68 dequeue_task_fair [kernel.kallsyms] 0{23 100.0% 35.9%} 1{18 0.0% 26.1%} 2{19 0.0% 25.0%} 3{ 8 0.0% 13.0%}
0.0% 0.0% 0.6% 0.0% 0x28 146560 *** 0xffffffff810c2ba0 n/a n/a n/a 2 task_tick_fair [kernel.kallsyms] 2{ 1 n/a 50.0%} 3{ 1 n/a 50.0%}
3.0% 0.0% 23.5% 0.0% 0x28 *** *** 0xffffffff810c4203 2085 2958 31.8% 61 enqueue_task_fair [kernel.kallsyms] 0{14 20.0% 30.1%} 1{14 0.0% 21.9%} 2{13 20.0% 21.9%} 3{20 60.0% 26.0%}
0.6% 0.0% 0.0% 0.0% 0x30 146560 146593 0xffffffff810b5fe7 3070 3070 0.0% 1 set_task_cpu [kernel.kallsyms] 1{ 1 100.0% n/a}
-----------------------------------------------------------------------------------------------
6 0.2% 3.1% 0.3% 5.9% 158 43 192 0 0xffff883fff855280 146560
-----------------------------------------------------------------------------------------------
0.0% 0.0% 87.5% 0.0% 0x00 146560 146569 0xffffffff810e6e13 n/a n/a n/a 1 flush_smp_call_function_queue [kernel.kallsyms] 1{ 1 n/a 100.0%}
98.7% 100.0% 0.0% 0.0% 0x00 146560 *** 0xffffffff813067d8 317 406 5.9% 44 llist_add_batch [kernel.kallsyms] 0{13 32.1% n/a} 1{10 11.5% n/a} 2{11 30.1% n/a} 3{10 26.3% n/a}
1.3% 0.0% 12.5% 0.0% 0x00 146560 *** 0xffffffff813067e1 1040 1040 23.1% 20 llist_add_batch [kernel.kallsyms] 0{ 6 50.0% 29.2%} 1{ 5 0.0% 20.8%} 2{ 6 50.0% 25.0%} 3{ 3 0.0% 25.0%}
-----------------------------------------------------------------------------------------------
7 0.1% 3.3% 0.3% 6.2% 154 44 153 0 0xffff885fff915280 146560
-----------------------------------------------------------------------------------------------
0.6% 0.0% 86.9% 0.0% 0x00 146560 146624 0xffffffff810e6e13 1341 1341 0.0% 1 flush_smp_call_function_queue [kernel.kallsyms] 2{ 1 100.0% 100.0%}
98.7% 100.0% 0.0% 0.0% 0x00 146560 *** 0xffffffff813067d8 287 377 9.2% 43 llist_add_batch [kernel.kallsyms] 0{12 36.8% n/a} 1{11 24.3% n/a} 2{10 11.8% n/a} 3{10 27.0% n/a}
0.6% 0.0% 13.1% 0.0% 0x00 146560 *** 0xffffffff813067e1 539 539 0.0% 17 llist_add_batch [kernel.kallsyms] 0{ 6 0.0% 30.0%} 1{ 6 0.0% 40.0%} 2{ 1 0.0% 5.0%} 3{ 4 100.0% 25.0%}
-----------------------------------------------------------------------------------------------
8 0.1% 3.4% 0.3% 6.4% 151 54 103 0 0xffff885fffc15280 146560
-----------------------------------------------------------------------------------------------
0.7% 0.0% 69.9% 0.0% 0x00 146560 146607 0xffffffff810e6e13 902 902 0.0% 1 flush_smp_call_function_queue [kernel.kallsyms] 2{ 1 100.0% 100.0%}
99.3% 98.1% 0.0% 0.0% 0x00 146560 *** 0xffffffff813067d8 278 377 5.3% 45 llist_add_batch [kernel.kallsyms] 0{13 34.7% n/a} 1{11 24.7% n/a} 2{11 10.7% n/a} 3{10 30.0% n/a}
0.0% 1.9% 30.1% 0.0% 0x00 146560 *** 0xffffffff813067e1 n/a n/a n/a 23 llist_add_batch [kernel.kallsyms] 0{ 6 n/a 22.6%} 1{ 7 n/a 32.3%} 2{ 6 n/a 25.8%} 3{ 4 n/a 19.4%}
-----------------------------------------------------------------------------------------------
9 0.1% 3.5% 0.3% 6.7% 151 42 223 0 0xffff885fffbf5280 146560
-----------------------------------------------------------------------------------------------
0.0% 0.0% 87.4% 0.0% 0x00 146560 *** 0xffffffff810e6e13 n/a n/a n/a 1 flush_smp_call_function_queue [kernel.kallsyms] 2{ 1 n/a 100.0%}
99.3% 97.6% 0.0% 0.0% 0x00 146560 *** 0xffffffff813067d8 309 346 2.5% 44 llist_add_batch [kernel.kallsyms] 0{11 28.0% n/a} 1{11 23.3% n/a} 2{11 13.3% n/a} 3{11 35.3% n/a}
0.7% 2.4% 12.6% 0.0% 0x00 146560 *** 0xffffffff813067e1 1028 1696 0.0% 20 llist_add_batch [kernel.kallsyms] 0{ 5 0.0% 25.0%} 1{ 2 0.0% 7.1%} 2{ 6 100.0% 35.7%} 3{ 7 0.0% 32.1%}
=====================================================================================================================================
Object Name, Path & Reference Counts
Index Records Object Name Object Path
=====================================================================================================================================
0 2032703 xlinpack_xeon64 /home/joe/linpack/xlinpack_xeon64
1 1059580 [kernel.kallsyms] /proc/kcore
2 21352 libpthread-2.17.so /usr/lib64/libpthread-2.17.so
3 3882 perf /home/root/git/rhel7/tools/perf/perf
4 26 libc-2.17.so /usr/lib64/libc-2.17.so
5 5 libpython2.7.so.1.0 /usr/lib64/libpython2.7.so.1.0
6 3 ld-2.17.so /usr/lib64/ld-2.17.so
7 1 sendmail.sendmail /usr/sbin/sendmail.sendmail
8 1 irqbalance /usr/sbin/irqbalance
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [Questions] perf c2c: What's the current status of perf c2c?
2015-12-09 20:41 ` Joe Mario
@ 2015-12-10 2:36 ` Yunlong Song
0 siblings, 0 replies; 14+ messages in thread
From: Yunlong Song @ 2015-12-10 2:36 UTC (permalink / raw)
To: Joe Mario, Stephane Eranian, Andi Kleen
Cc: Peter Zijlstra, Jiri Olsa, Don Zickus, David Ahern,
Frédéric Weisbecker, Mike Galbraith, Paul Mackerras,
acme@kernel.org >> Arnaldo Carvalho de Melo,
mingo@redhat.com, Linux Kernel Mailing List, Jiri Olsa,
wangnan0@huawei.com >> Wang Nan, Richard Fowles,
Namhyung Kim
On 2015/12/10 4:41, Joe Mario wrote:
> Appended below is the output for running "perf c2c" on a 4-node system
> running a multi-thread version of linpack. I've annotated it show describe
> what some of the fields mean.
>
> Note, your screen output has to be set pretty wide to read it. For those
> not wanting to read it in their mailer, grab it from:
> http://people.redhat.com/jmario/perf_c2c/perf_c2c_annotated_output.txt
>
> Let me know of any questions. My annotations begin with "// ".
>
> Joe
> --------------------------------------------------------------------
> // Perf c2c output from a linpack run.
Hi, Joe,
Got these details, thanks a lot for your interpretation. -_-
--
Thanks,
Yunlong Song
^ permalink raw reply [flat|nested] 14+ messages in thread