* 4.19 dwarf unwinding broken
@ 2019-10-03 7:54 Josh Hunt
2019-10-03 10:03 ` Jiri Olsa
0 siblings, 1 reply; 4+ messages in thread
From: Josh Hunt @ 2019-10-03 7:54 UTC (permalink / raw)
To: john, jolsa, alexander.shishkin, khlebnikov, namhyung, peterz,
acme
Cc: linux-kernel
The following commit is breaking dwarf unwinding on 4.19 kernels:
commit e5adfc3e7e774ba86f7bb725c6eef5f32df8630e
Author: Konstantin Khlebnikov <khlebnikov@yandex-team.ru>
Date: Tue Aug 7 12:09:01 2018 +0300
perf map: Synthesize maps only for thread group leader
It looks like this was fixed later by:
commit e8ba2906f6b9054102ad035ac9cafad9d4168589
Author: John Keeping <john@metanate.com>
Date: Thu Aug 15 11:01:45 2019 +0100
perf unwind: Fix libunwind when tid != pid
but was not selected as a backport to 4.19. Are there plans to backport
the fix? If not, could we have the breaking commit reverted in 4.19 LTS?
Thanks
Josh
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: 4.19 dwarf unwinding broken 2019-10-03 7:54 4.19 dwarf unwinding broken Josh Hunt @ 2019-10-03 10:03 ` Jiri Olsa 2019-10-03 15:17 ` Josh Hunt 0 siblings, 1 reply; 4+ messages in thread From: Jiri Olsa @ 2019-10-03 10:03 UTC (permalink / raw) To: Josh Hunt Cc: john, jolsa, alexander.shishkin, khlebnikov, namhyung, peterz, acme, linux-kernel On Thu, Oct 03, 2019 at 12:54:09AM -0700, Josh Hunt wrote: > The following commit is breaking dwarf unwinding on 4.19 kernels: how? jirka > > commit e5adfc3e7e774ba86f7bb725c6eef5f32df8630e > Author: Konstantin Khlebnikov <khlebnikov@yandex-team.ru> > Date: Tue Aug 7 12:09:01 2018 +0300 > > perf map: Synthesize maps only for thread group leader > > It looks like this was fixed later by: > > commit e8ba2906f6b9054102ad035ac9cafad9d4168589 > Author: John Keeping <john@metanate.com> > Date: Thu Aug 15 11:01:45 2019 +0100 > > perf unwind: Fix libunwind when tid != pid > > but was not selected as a backport to 4.19. Are there plans to backport the > fix? If not, could we have the breaking commit reverted in 4.19 LTS? > > Thanks > Josh ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 4.19 dwarf unwinding broken 2019-10-03 10:03 ` Jiri Olsa @ 2019-10-03 15:17 ` Josh Hunt 2019-10-03 16:04 ` Arnaldo Carvalho de Melo 0 siblings, 1 reply; 4+ messages in thread From: Josh Hunt @ 2019-10-03 15:17 UTC (permalink / raw) To: Jiri Olsa Cc: john, jolsa, alexander.shishkin, khlebnikov, namhyung, peterz, acme, linux-kernel On 10/3/19 3:03 AM, Jiri Olsa wrote: > On Thu, Oct 03, 2019 at 12:54:09AM -0700, Josh Hunt wrote: >> The following commit is breaking dwarf unwinding on 4.19 kernels: > > how? When doing something like: perf record -p $(cat /var/run/app.pid) -g --call-graph dwarf -F 999 -- sleep 3 with 4.19.75 perf I see things like: app_Thr00 26247 1810131.375329: 168288 cycles:ppp: app_Thr01 26767 1810131.377449: 344415 cycles:ppp: uvm:WorkerThread 26746 1810131.383052: 504 cycles:ppp: ffffffff9f77cce0 _raw_spin_lock+0x10 (/boot/vmlinux-4.19.46) ffffffff9f181527 __perf_event_task_sched_in+0xf7 (/boot/vmlinux-4.19.46) ffffffff9f09a7b8 finish_task_switch+0x158 (/boot/vmlinux-4.19.46) ffffffff9f778276 __schedule+0x2f6 (/boot/vmlinux-4.19.46) ffffffff9f7787f2 schedule+0x32 (/boot/vmlinux-4.19.46) ffffffff9f77bb0a schedule_hrtimeout_range_clock+0x8a (/boot/vmlinux-4.19.46) ffffffff9f22ea12 poll_schedule_timeout.constprop.6+0x42 (/boot/vmlinux-4.19.46) ffffffff9f22eeeb do_sys_poll+0x4ab (/boot/vmlinux-4.19.46) ffffffff9f22fb7b __se_sys_poll+0x5b (/boot/vmlinux-4.19.46) ffffffff9f0023de do_syscall_64+0x4e (/boot/vmlinux-4.19.46) ffffffff9f800088 entry_SYSCALL_64+0x68 (/boot/vmlinux-4.19.46) --- and with 4.19.75 perf with e5adfc3e7e77 reverted those empty call stacks go away and also other call stacks show more thread details: uvm:WorkerThread 26746 1810207.336391: 1 cycles:ppp: ffffffff9f181505 __perf_event_task_sched_in+0xd5 (/boot/vmlinux-4.19.46) ffffffff9f09a7b8 finish_task_switch+0x158 (/boot/vmlinux-4.19.46) ffffffff9f778276 __schedule+0x2f6 (/boot/vmlinux-4.19.46) ffffffff9f7787f2 schedule+0x32 (/boot/vmlinux-4.19.46) ffffffff9f77bb0a schedule_hrtimeout_range_clock+0x8a (/boot/vmlinux-4.19.46) ffffffff9f22ea12 poll_schedule_timeout.constprop.6+0x42 (/boot/vmlinux-4.19.46) ffffffff9f22eeeb do_sys_poll+0x4ab (/boot/vmlinux-4.19.46) ffffffff9f22fb7b __se_sys_poll+0x5b (/boot/vmlinux-4.19.46) ffffffff9f0023de do_syscall_64+0x4e (/boot/vmlinux-4.19.46) ffffffff9f800088 entry_SYSCALL_64+0x68 (/boot/vmlinux-4.19.46) 7f7ef3f5c90d [unknown] (/lib/x86_64-linux-gnu/libc-2.23.so) 3eb5c99 poll+0xc9 (inlined) 3eb5c99 colib::ipc::EventFd::wait+0xc9 (/usr/local/bin/app) 3296779 uvm::WorkerThread::run+0x129 (/usr/local/bin/app) ffffffffffffffff [unknown] ([unknown]) They also look the same as earlier kernel versions we have running. In addition reading e8ba2906f6b's changelog sounded very similar to what I was seeing. This application launches a # of threads and is definitely already running before the invocation of perf. Thanks for looking at this. Josh > > jirka > >> >> commit e5adfc3e7e774ba86f7bb725c6eef5f32df8630e >> Author: Konstantin Khlebnikov <khlebnikov@yandex-team.ru> >> Date: Tue Aug 7 12:09:01 2018 +0300 >> >> perf map: Synthesize maps only for thread group leader >> >> It looks like this was fixed later by: >> >> commit e8ba2906f6b9054102ad035ac9cafad9d4168589 >> Author: John Keeping <john@metanate.com> >> Date: Thu Aug 15 11:01:45 2019 +0100 >> >> perf unwind: Fix libunwind when tid != pid >> >> but was not selected as a backport to 4.19. Are there plans to backport the >> fix? If not, could we have the breaking commit reverted in 4.19 LTS? >> >> Thanks >> Josh ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 4.19 dwarf unwinding broken 2019-10-03 15:17 ` Josh Hunt @ 2019-10-03 16:04 ` Arnaldo Carvalho de Melo 0 siblings, 0 replies; 4+ messages in thread From: Arnaldo Carvalho de Melo @ 2019-10-03 16:04 UTC (permalink / raw) To: Josh Hunt Cc: Jiri Olsa, john, jolsa, alexander.shishkin, khlebnikov, namhyung, peterz, linux-kernel Em Thu, Oct 03, 2019 at 08:17:38AM -0700, Josh Hunt escreveu: > On 10/3/19 3:03 AM, Jiri Olsa wrote: > >On Thu, Oct 03, 2019 at 12:54:09AM -0700, Josh Hunt wrote: > >>The following commit is breaking dwarf unwinding on 4.19 kernels: > > > >how? > > When doing something like: > perf record -p $(cat /var/run/app.pid) -g --call-graph dwarf -F 999 > -- sleep 3 > > with 4.19.75 perf I see things like: > > app_Thr00 26247 1810131.375329: 168288 cycles:ppp: > > app_Thr01 26767 1810131.377449: 344415 cycles:ppp: > > uvm:WorkerThread 26746 1810131.383052: 504 cycles:ppp: > ffffffff9f77cce0 _raw_spin_lock+0x10 (/boot/vmlinux-4.19.46) > ffffffff9f181527 __perf_event_task_sched_in+0xf7 > (/boot/vmlinux-4.19.46) > ffffffff9f09a7b8 finish_task_switch+0x158 (/boot/vmlinux-4.19.46) > ffffffff9f778276 __schedule+0x2f6 (/boot/vmlinux-4.19.46) > ffffffff9f7787f2 schedule+0x32 (/boot/vmlinux-4.19.46) > ffffffff9f77bb0a schedule_hrtimeout_range_clock+0x8a > (/boot/vmlinux-4.19.46) > ffffffff9f22ea12 poll_schedule_timeout.constprop.6+0x42 > (/boot/vmlinux-4.19.46) > ffffffff9f22eeeb do_sys_poll+0x4ab (/boot/vmlinux-4.19.46) > ffffffff9f22fb7b __se_sys_poll+0x5b (/boot/vmlinux-4.19.46) > ffffffff9f0023de do_syscall_64+0x4e (/boot/vmlinux-4.19.46) > ffffffff9f800088 entry_SYSCALL_64+0x68 (/boot/vmlinux-4.19.46) > --- > > and with 4.19.75 perf with e5adfc3e7e77 reverted those empty call > stacks go away and also other call stacks show more thread details: > > uvm:WorkerThread 26746 1810207.336391: 1 cycles:ppp: > ffffffff9f181505 __perf_event_task_sched_in+0xd5 > (/boot/vmlinux-4.19.46) > ffffffff9f09a7b8 finish_task_switch+0x158 (/boot/vmlinux-4.19.46) > ffffffff9f778276 __schedule+0x2f6 (/boot/vmlinux-4.19.46) > ffffffff9f7787f2 schedule+0x32 (/boot/vmlinux-4.19.46) > ffffffff9f77bb0a schedule_hrtimeout_range_clock+0x8a > (/boot/vmlinux-4.19.46) > ffffffff9f22ea12 poll_schedule_timeout.constprop.6+0x42 > (/boot/vmlinux-4.19.46) > ffffffff9f22eeeb do_sys_poll+0x4ab (/boot/vmlinux-4.19.46) > ffffffff9f22fb7b __se_sys_poll+0x5b (/boot/vmlinux-4.19.46) > ffffffff9f0023de do_syscall_64+0x4e (/boot/vmlinux-4.19.46) > ffffffff9f800088 entry_SYSCALL_64+0x68 (/boot/vmlinux-4.19.46) > 7f7ef3f5c90d [unknown] (/lib/x86_64-linux-gnu/libc-2.23.so) > 3eb5c99 poll+0xc9 (inlined) > 3eb5c99 colib::ipc::EventFd::wait+0xc9 > (/usr/local/bin/app) > 3296779 uvm::WorkerThread::run+0x129 (/usr/local/bin/app) > ffffffffffffffff [unknown] ([unknown]) > > They also look the same as earlier kernel versions we have running. > > In addition reading e8ba2906f6b's changelog sounded very similar to > what I was seeing. This application launches a # of threads and is > definitely already running before the invocation of perf. > > Thanks for looking at this. So, if at __event__synthesize_thread() if (pid == tgid && perf_event__synthesize_mmap_events(tool, mmap_event, pid, tgid, process, machine, mmap_data)) We did something like: if (pid != tgid && machine__find_thread(tgid, tgid) == NULL) { struct thread *t = thread__new(tgid, tgid); then use the info for pid to get synthesize the thread group leader so that the get the sharing we need? } - Arnaldo ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2019-10-03 16:04 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2019-10-03 7:54 4.19 dwarf unwinding broken Josh Hunt 2019-10-03 10:03 ` Jiri Olsa 2019-10-03 15:17 ` Josh Hunt 2019-10-03 16:04 ` Arnaldo Carvalho de Melo
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.