Re: BUG: perf top enters loop synthesizing events for existing threads

[Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com>]

Em Wed, Apr 08, 2015 at 11:14:27AM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Wed, Apr 08, 2015 at 07:48:23AM -0600, David Ahern escreveu:
> > On 4/8/15 7:45 AM, Arnaldo Carvalho de Melo wrote:
> > >>I also noticed that if I set /proc/sys/kernel/perf_event_paranoid to -1
> > >>to run it as !root, then the problem "goes away", which I think probably
> > >>is explained by, as !root, not being able to parse some of the /proc
> > >>files for existing threads and thus not triggering the bug, still
> > >>investigating...
> > 
> > odd. because this has nothing to do with perf_events; it is just walking
> > /proc and for the ppid adds a strstr and atoi(str). Something else is at
> > play. I'll take a look.
> 
> You are correcly setting the pid values, that will have effects when
> using findnew, i.e. threads will be added to the rbtree, which causes
> allocations, etc I.e. it is probably triggering a dormant bug

Sitting there, will stop chrome to see if the problem is triggered by it...

Program received signal SIGTERM, Terminated.
0x00000000004abc39 in rb_next (node=0x16202d20) at ../../lib/rbtree.c:457
457			while (node->rb_left)
(gdb) bt
#0  0x00000000004abc39 in rb_next (node=0x16202d20) at ../../lib/rbtree.c:457
#1  0x00000000004ccd27 in map_groups__fixup_overlappings (mg=0x12745740, map=0x16676210, fp=0x7ffff5a711c0 <_IO_2_1_stderr_>) at util/map.c:643
#2  0x00000000004d457d in thread__insert_map (thread=0x12745680, map=0x16676210) at util/thread.c:177
#3  0x00000000004c8de3 in machine__process_mmap2_event (machine=0xa28570, event=0xa59410, sample=0x870160 <synth_sample>) at util/machine.c:1204
#4  0x00000000004c92c3 in machine__process_event (machine=0xa28570, event=0xa59410, sample=0x870160 <synth_sample>) at util/machine.c:1329
#5  0x0000000000494a81 in perf_event__process (tool=0x0, event=0xa59410, sample=0x870160 <synth_sample>, machine=0xa28570) at util/event.c:778
#6  0x0000000000493606 in perf_event__synthesize_mmap_events (tool=0x0, event=0xa59410, pid=2513, tgid=2513, process=0x494a42 <perf_event__process>, machine=0xa28570, mmap_data=false) at util/event.c:302
#7  0x0000000000493b7b in __event__synthesize_thread (comm_event=0xa2a820, mmap_event=0xa59410, fork_event=0xa2a6a0, pid=2513, full=1, process=0x494a42 <perf_event__process>, tool=0x0, machine=0xa28570, mmap_data=false)
    at util/event.c:429
#8  0x0000000000493fe8 in perf_event__synthesize_threads (tool=0x0, process=0x494a42 <perf_event__process>, machine=0xa28570, mmap_data=false) at util/event.c:550
#9  0x00000000004ca82e in __machine__synthesize_threads (machine=0xa28570, tool=0x0, target=0x7fffffffb008, threads=0x94e490, process=0x494a42 <perf_event__process>, data_mmap=false) at util/machine.c:1781
#10 0x000000000043ae4c in machine__synthesize_threads (machine=0xa28570, target=0x7fffffffb008, threads=0x94e490, data_mmap=false) at util/machine.h:218
#11 0x000000000043d744 in __cmd_top (top=0x7fffffffaf80) at builtin-top.c:952
#12 0x000000000043ea4f in cmd_top (argc=0, argv=0x7fffffffe3d0, prefix=0x0) at builtin-top.c:1267
#13 0x0000000000489132 in run_builtin (p=0x86f128 <commands+264>, argc=1, argv=0x7fffffffe3d0) at perf.c:370
#14 0x0000000000489391 in handle_internal_command (argc=1, argv=0x7fffffffe3d0) at perf.c:429
#15 0x00000000004894dd in run_argv (argcp=0x7fffffffe22c, argv=0x7fffffffe220) at perf.c:473
#16 0x000000000048983e in main (argc=1, argv=0x7fffffffe3d0) at perf.c:588
(gdb) 

(gdb) fr 3
#3  0x00000000004c8de3 in machine__process_mmap2_event (machine=0xa28570, event=0xa59410, sample=0x870160 <synth_sample>) at util/machine.c:1204
1204		thread__insert_map(thread, map);
(gdb) print *thread
$1 = {{rb_node = {__rb_parent_color = 37177040, rb_right = 0x0, rb_left = 0x0}, node = {next = 0x23746d0, prev = 0x0}}, mg = 0x12745740, pid_ = 2513, tid = 2513, ppid = 2480, cpu = -1, refcnt = 2, shortname = "\000\000", 
  comm_set = true, dead = false, comm_list = {next = 0x16f948b0, prev = 0x12745710}, comm_len = 0, db_id = 0, priv = 0x0, ts = 0x0}
(gdb) print event->header
$4 = {type = 10, misc = 2, size = 104}
(gdb)
[acme@zoo linux]$ ps ax|grep 2513
 2513 ?        S      0:00 /opt/google/chrome/chrome --type=gpu-broker

[acme@zoo linux]$ grep xp /proc/2513/maps 
7fac6ac94000-7fac6ac9c000 r-xp 00000000 fd:01 1187628    /usr/lib64/libpciaccess.so.0.11.1
7fac6ae9e000-7fac6aeaa000 r-xp 00000000 fd:01 1184221    /usr/lib64/libdrm_radeon.so.1.0.1
7fac6b0ab000-7fac6b0b1000 r-xp 00000000 fd:01 1184218    /usr/lib64/libdrm_nouveau.so.2.0.0
7fac6b2b2000-7fac6b2d2000 r-xp 00000000 fd:01 1184215    /usr/lib64/libdrm_intel.so.1.0.0
7fac6b4d3000-7fac6b9ef000 r-xp 00000000 fd:01 1442035    /usr/lib64/dri/i965_dri.so
7fac6bc1d000-7fac6bc2d000 r-xp 00000000 fd:01 1192079    /usr/lib64/libudev.so.1.4.0
7fac73553000-7fac735ca000 r-xp 00000000 fd:01 1185534    /usr/lib64/libfreebl3.so
7fac737d0000-7fac737d3000 r-xp 00000000 fd:01 1187499    /usr/lib64/libkeyutils.so.1.5
7fac739d4000-7fac739e1000 r-xp 00000000 fd:01 1189618    /usr/lib64/libkrb5support.so.0.1
7fac73be2000-7fac73be7000 r-xp 00000000 fd:01 1188148    /usr/lib64/libXxf86vm.so.1.0.0
7fac73de8000-7fac73dff000 r-xp 00000000 fd:01 1188964    /usr/lib64/libxcb-glx.so.0.0.0
7fac74002000-7fac74004000 r-xp 00000000 fd:01 1188124    /usr/lib64/libXau.so.6.0.0
7fac74206000-7fac74211000 r-xp 00000000 fd:01 1179773    /usr/lib64/libdrm.so.2.4.0
7fac74412000-7fac74437000 r-xp 00000000 fd:01 1190780    /usr/lib64/libglapi.so.0.0.0
7fac7463c000-7fac7464a000 r-xp 00000000 fd:01 1188942    /usr/lib64/libwayland-server.so.0.1.0
7fac7484c000-7fac74857000 r-xp 00000000 fd:01 1188939    /usr/lib64/libwayland-client.so.0.1.0
7fac74a59000-7fac74a66000 r-xp 00000000 fd:01 1187278    /usr/lib64/libgbm.so.1.0.0
7fac74c67000-7fac74c6a000 r-xp 00000000 fd:01 1188970    /usr/lib64/libxcb-shape.so.0.0.0
7fac74e6b000-7fac74e71000 r-xp 00000000 fd:01 1188976    /usr/lib64/libxcb-xfixes.so.0.0.0
7fac75072000-7fac75076000 r-xp 00000000 fd:01 1188963    /usr/lib64/libxcb-dri2.so.0.0.0
7fac75277000-7fac75278000 r-xp 00000000 fd:01 1188121    /usr/lib64/libX11-xcb.so.1.0.0
7fac75479000-7fac75493000 r-xp 00000000 fd:01 1188433    /usr/lib64/libgraphite2.so.3.0.1
7fac75695000-7fac756b9000 r-xp 00000000 fd:01 1184295    /usr/lib64/liblzma.so.5.0.99
7fac758ba000-7fac7591f000 r-xp 00000000 fd:01 1190568    /usr/lib64/libpcre.so.1.2.1
7fac75b20000-7fac75b22000 r-xp 00000000 fd:01 1188135    /usr/lib64/libXinerama.so.1.0.0
7fac75d23000-7fac75d43000 r-xp 00000000 fd:01 1188169    /usr/lib64/libatk-1.0.so.0.21009.1
7fac75f46000-7fac75f4e000 r-xp 00000000 fd:01 1188257    /usr/lib64/libcrypt-2.18.so
7fac7617d000-7fac7618d000 r-xp 00000000 fd:01 1188177    /usr/lib64/libavahi-client.so.3.2.9
7fac7638e000-7fac7639a000 r-xp 00000000 fd:01 1188178    /usr/lib64/libavahi-common.so.3.5.3
7fac7659b000-7fac76758000 r-xp 00000000 fd:01 1180383    /usr/lib64/libcrypto.so.1.0.1e
7fac76983000-7fac769e5000 r-xp 00000000 fd:01 1185907    /usr/lib64/libssl.so.1.0.1e
7fac76bef000-7fac76bf2000 r-xp 00000000 fd:01 1185817    /usr/lib64/libcom_err.so.2.1
7fac76df3000-7fac76e25000 r-xp 00000000 fd:01 1187372    /usr/lib64/libk5crypto.so.3.1
7fac77028000-7fac770f8000 r-xp 00000000 fd:01 1187503    /usr/lib64/libkrb5.so.3.3
7fac77308000-7fac7734f000 r-xp 00000000 fd:01 1189531    /usr/lib64/libgssapi_krb5.so.2.2
7fac77552000-7fac775e2000 r-xp 00000000 fd:01 1190785    /usr/lib64/libGL.so.1.2.0
7fac777e6000-7fac77806000 r-xp 00000000 fd:01 1188983    /usr/lib64/libxcb.so.1.1.0
7fac77a07000-7fac77a10000 r-xp 00000000 fd:01 1188967    /usr/lib64/libxcb-render.so.0.0.0
7fac77c11000-7fac77c13000 r-xp 00000000 fd:01 1188971    /usr/lib64/libxcb-shm.so.0.0.0
7fac77e14000-7fac77e3c000 r-xp 00000000 fd:01 1190790    /usr/lib64/libEGL.so.1.0.0
7fac7803e000-7fac780e0000 r-xp 00000000 fd:01 1187639    /usr/lib64/libpixman-1.so.0.30.0
7fac782e8000-7fac7833c000 r-xp 00000000 fd:01 1187448    /usr/lib64/libharfbuzz.so.0.927.0
7fac7853e000-7fac7853f000 r-xp 00000000 fd:01 1188489    /usr/lib64/libgthread-2.0.so.0.3800.2
7fac78740000-7fac78754000 r-xp 00000000 fd:01 1187618    /usr/lib64/libpangoft2-1.0.so.0.3600.1
7fac78956000-7fac78988000 r-xp 00000000 fd:01 1184251    /usr/lib64/libpng16.so.16.6.0
7fac78b89000-7fac78b9f000 r-xp 00000000 fd:01 1189224    /usr/lib64/libresolv-2.18.so
7fac78da3000-7fac78dc4000 r-xp 00000000 fd:01 1187969    /usr/lib64/libselinux.so.1
7fac78fc7000-7fac78fdc000 r-xp 00000000 fd:01 1188999    /usr/lib64/libz.so.1.2.8
7fac791dd000-7fac79203000 r-xp 00000000 fd:01 1188266    /usr/lib64/libdbus-glib-1.so.2.2.2
7fac79405000-7fac79408000 r-xp 00000000 fd:01 1188412    /usr/lib64/libgmodule-2.0.so.0.3800.2
7fac79609000-7fac7960c000 r-xp 00000000 fd:01 1189435    /usr/lib64/libplds4.so
7fac7980d000-7fac79814000 r-xp 00000000 fd:01 1188331    /usr/lib64/libffi.so.6.0.1
7fac79a15000-7fac79bc9000 r-xp 00000000 fd:01 1187054    /usr/lib64/libc-2.18.so
7fac79dd3000-7fac79de8000 r-xp 00000000 fd:01 1183174    /usr/lib64/libgcc_s-4.8.3-20140911.so.1
7fac79fe9000-7fac7a0d2000 r-xp 00000000 fd:01 1183177    /usr/lib64/libstdc++.so.6.0.19
7fac7a2f1000-7fac7a2f3000 r-xp 00000000 fd:01 1190263    /usr/lib64/libXss.so.1.0.0
7fac7a4f5000-7fac7a518000 r-xp 00000000 fd:01 1187313    /usr/lib64/libgdk_pixbuf-2.0.so.0.3000.3
7fac7a719000-7fac7a7d5000 r-xp 00000000 fd:01 1185350    /usr/lib64/libgdk-x11-2.0.so.0.2400.27
7fac7a9da000-7fac7ae52000 r-xp 00000000 fd:01 1185352    /usr/lib64/libgtk-x11-2.0.so.0.2400.27
7fac7b060000-7fac7b0a6000 r-xp 00000000 fd:01 1185836    /usr/lib64/libdbus-1.so.3.7.12
7fac7b2a8000-7fac7b2c0000 r-xp 00000000 fd:01 1188678    /usr/lib64/libpthread-2.18.so
7fac7b4c5000-7fac7b539000 r-xp 00000000 fd:01 1185342    /usr/lib64/libcups.so.2
7fac7b742000-7fac7b769000 r-xp 00000000 fd:01 1188325    /usr/lib64/libexpat.so.1.6.0
7fac7b769000-7fac7b969000 ---p 00027000 fd:01 1188325    /usr/lib64/libexpat.so.1.6.0
7fac7b969000-7fac7b96b000 r--p 00027000 fd:01 1188325    /usr/lib64/libexpat.so.1.6.0
7fac7b96b000-7fac7b96c000 rw-p 00029000 fd:01 1188325    /usr/lib64/libexpat.so.1.6.0
7fac7b96c000-7fac7b975000 r-xp 00000000 fd:01 1188139    /usr/lib64/libXrandr.so.2.2.0
7fac7bb76000-7fac7bb7b000 r-xp 00000000 fd:01 1188142    /usr/lib64/libXtst.so.6.1.0
7fac7bd7c000-7fac7bd7e000 r-xp 00000000 fd:01 1188127    /usr/lib64/libXdamage.so.1.1.0
7fac7bf7f000-7fac7c084000 r-xp 00000000 fd:01 1188374    /usr/lib64/libm-2.18.so
7fac7c286000-7fac7c367000 r-xp 00000000 fd:01 1188164    /usr/lib64/libasound.so.2.0.0
7fac7c56f000-7fac7c571000 r-xp 00000000 fd:01 1188125    /usr/lib64/libXcomposite.so.1.0.0
7fac7c772000-7fac7c77b000 r-xp 00000000 fd:01 1188140    /usr/lib64/libXrender.so.1.3.0
7fac7c97c000-7fac7c981000 r-xp 00000000 fd:01 1188131    /usr/lib64/libXfixes.so.3.1.0
7fac7cb82000-7fac7cb93000 r-xp 00000000 fd:01 1188130    /usr/lib64/libXext.so.6.4.0
7fac7cd94000-7fac7cd9e000 r-xp 00000000 fd:01 1188126    /usr/lib64/libXcursor.so.1.0.2
7fac7cf9f000-7fac7cfae000 r-xp 00000000 fd:01 1183298    /usr/lib64/libXi.so.6.1.0
7fac7d1af000-7fac7d2e7000 r-xp 00000000 fd:01 1188122    /usr/lib64/libX11.so.6.3.0
7fac7d4ed000-7fac7d535000 r-xp 00000000 fd:01 1187616    /usr/lib64/libpango-1.0.so.0.3600.1
7fac7d738000-7fac7d859000 r-xp 00000000 fd:01 1187140    /usr/lib64/libcairo.so.2.11400.0
7fac7da5f000-7fac7da6b000 r-xp 00000000 fd:01 1187617    /usr/lib64/libpangocairo-1.0.so.0.3600.1
7fac7dc6c000-7fac7dd0b000 r-xp 00000000 fd:01 1187649    /usr/lib64/libfreetype.so.6.10.2
7fac7df11000-7fac7df4c000 r-xp 00000000 fd:01 1187277    /usr/lib64/libfontconfig.so.1.8.0
7fac7e14e000-7fac7e2ac000 r-xp 00000000 fd:01 1188399    /usr/lib64/libgio-2.0.so.0.3800.2
7fac7e4b3000-7fac7e4e3000 r-xp 00000000 fd:01 1188370    /usr/lib64/libgconf-2.so.4.1.5
7fac7e6e5000-7fac7e71f000 r-xp 00000000 fd:01 1189378    /usr/lib64/libnspr4.so
7fac7e923000-7fac7e927000 r-xp 00000000 fd:01 1189431    /usr/lib64/libplc4.so
7fac7eb28000-7fac7eb4c000 r-xp 00000000 fd:01 1192057    /usr/lib64/libsmime3.so
7fac7ed4f000-7fac7ed74000 r-xp 00000000 fd:01 1189436    /usr/lib64/libnssutil3.so
7fac7ef7b000-7fac7f099000 r-xp 00000000 fd:01 1191883    /usr/lib64/libnss3.so
7fac7f2a2000-7fac7f3cb000 r-xp 00000000 fd:01 1188407    /usr/lib64/libglib-2.0.so.0.3800.2
7fac7f5ce000-7fac7f61d000 r-xp 00000000 fd:01 1188425    /usr/lib64/libgobject-2.0.so.0.3800.2
7fac7f81f000-7fac7f822000 r-xp 00000000 fd:01 1188290    /usr/lib64/libdl-2.18.so
7fac7fa23000-7fac7fa2a000 r-xp 00000000 fd:01 1189264    /usr/lib64/librt-2.18.so
7fac7fc2b000-7fac7fc4b000 r-xp 00000000 fd:01 1187898    /usr/lib64/ld-2.18.so
7fac7fe48000-7fac7fe49000 rwxp 00000000 00:00 0 
7fac7fe4d000-7fac84fa0000 r-xp 00000000 fd:01 917553     /opt/google/chrome/chrome
7ffdd8dea000-7ffdd8dec000 r-xp 00000000 00:00 0          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0  [vsyscall]
[acme@zoo linux]$