From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1161638AbcHEPzN (ORCPT ); Fri, 5 Aug 2016 11:55:13 -0400 Received: from mail.kernel.org ([198.145.29.136]:59076 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1759999AbcHEPzL (ORCPT ); Fri, 5 Aug 2016 11:55:11 -0400 Date: Fri, 5 Aug 2016 12:55:04 -0300 From: Arnaldo Carvalho de Melo To: Masami Hiramatsu Cc: Arnaldo Carvalho de Melo , "Wangnan (F)" , Alexei Starovoitov , Linux Kernel Mailing List Subject: Re: perf test BPF failing on f24: fix Message-ID: <20160805155504.GA12595@kernel.org> References: <20160802195102.GD14639@kernel.org> <57A1A913.6000307@huawei.com> <20160803234557.29f43f755b7e14c634a54a9a@kernel.org> <20160803200415.GG14639@kernel.org> <20160804153221.8aa29dfc5533b14089bd7470@kernel.org> <20160804124857.GH14639@kernel.org> <20160804193656.GK14639@kernel.org> <20160804214724.GL14639@kernel.org> <20160805184550.33ffa7a6989fad3940187d7e@kernel.org> <20160805143532.GO14639@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160805143532.GO14639@kernel.org> X-Url: http://acmel.wordpress.com User-Agent: Mutt/1.6.2 (2016-07-01) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Em Fri, Aug 05, 2016 at 11:35:32AM -0300, Arnaldo Carvalho de Melo escreveu: > Em Fri, Aug 05, 2016 at 06:45:50PM +0900, Masami Hiramatsu escreveu: > > See the symbol address calcurated from symbol map, in successful case > > the address exactly same address of SyS_epoll_wait. This indicates > > something might wrong in the symbol map. (maybe KASLR?) > > Could you check what happen if nokaslr is passed to your kernel? > [acme@jouet linux]$ grep RANDOMIZE_BASE ../build/v4.7.0+/.config > CONFIG_RANDOMIZE_BASE=y > Yes, it is in place, I'll check rebooting with nokaslr. > Can you think about how to properly warn the user about this > possibility, if it is confirmed to cause this issue? > I wonder why this doesn't seem to affect annotation, that also uses vmlinux, > for instance: > [root@jouet ~]# perf record --all-kernel -F 10000 ping -f -c 100000 localhost > PING localhost.localdomain (127.0.0.1) 56(84) bytes of data. > --- localhost.localdomain ping statistics --- > 100000 packets transmitted, 100000 received, 0% packet loss, time 732ms > rtt min/avg/max/mdev = 0.003/0.003/0.048/0.002 ms, ipg/ewma 0.007/0.003 ms > [ perf record: Woken up 2 times to write data ] > [ perf record: Captured and wrote 0.298 MB perf.data (7340 samples) ] > [root@jouet ~]# perf report -v | grep '\[' | head -10 > Using /lib/modules/4.7.0+/build/vmlinux for symbols > 20.42% ping /lib/modules/4.7.0+/build/vmlinux 0x960374 v [k] ipt_do_table > 2.55% ping /lib/modules/4.7.0+/build/vmlinux 0x900b8a v [k] nf_iterate > 1.94% ping /lib/modules/4.7.0+/build/vmlinux 0x95190b v [k] fib_table_lookup > 1.68% ping /lib/modules/4.7.0+/build/vmlinux 0x2accfa v [k] __local_bh_enable_ip > 1.68% ping /lib/modules/4.7.0+/build/vmlinux 0x9e41ab v [k] _raw_spin_lock_irqsave > 1.66% ping /lib/modules/4.7.0+/build/vmlinux 0x55fd94 v [k] avc_has_perm > 1.65% ping /lib/modules/4.7.0+/build/vmlinux 0x90e3e2 v [k] __ip_append_data.isra.44 > 1.31% ping /lib/modules/4.7.0+/build/vmlinux 0x5ead59 v [k] copy_user_enhanced_fast_string > 1.21% ping /lib/modules/4.7.0+/build/vmlinux 0x938820 v [k] raw_local_deliver > 1.14% ping /lib/modules/4.7.0+/build/vmlinux 0x6c6cea v [k] n_tty_write > [root@jouet ~]# > [root@jouet ~]# perf annotate -v --stdio ipt_do_table > Looking at the vmlinux_path (8 entries long) > Using /lib/modules/4.7.0+/build/vmlinux for symbols > symbol__disassemble: filename=/lib/modules/4.7.0+/build/vmlinux, sym=ipt_do_table, start=0xffffffffbd7601c0, end=0xffffffffbd7607b0 > annotating [0x3cef9b0] /lib/modules/4.7.0+/build/vmlinux : [0x4159750] ipt_do_table > Executing: objdump --start-address=0xffffffff817601c0 --stop-address=0xffffffff817607b0 -l -d --no-show-raw -S -C /lib/modules/4.7.0+/build/vmlinux 2>/dev/nu > Percent | Source code & Disassembly of vmlinux for cycles:ppp (1498 samples) > ---------------------------------------------------------------------------------- > > [root@jouet ~]# grep ffffffffbd7601c0 /proc/kallsyms > ffffffffbd7601c0 T ipt_do_table > [root@jouet ~]# > Anyway, will reboot this machine with nokaslr to check that possibility. So, more info, but executive summary: distro kernels don't exhibit this problem, but they have CONFIG_RANDOMIZE_BASE=y, the kernel where I noticed this problem (4.7.0+) doesn't exhibit this problem when used with 'nokaslr', but I think more investigation is needed to get to the bottom of this, i.e. to emit meaningful messages when kaslr is in place (or is suspected to be): Using a distro kernel: [root@jouet ~]# uname -r 4.6.4-301.fc24.x86_64 [root@jouet ~]# perf probe sys_epoll_wait Added new event: probe:sys_epoll_wait (on sys_epoll_wait) You can now use it in all perf tools, such as: perf record -e probe:sys_epoll_wait -aR sleep 1 [root@jouet ~]# rpm -q kernel-debuginfo kernel-debuginfo-4.6.3-300.fc24.x86_64 [root@jouet ~]# uname -r 4.6.4-301.fc24.x86_64 [root@jouet ~]# So it must be using just /proc/kallsyms, since there is no matching vmlinux: [root@jouet ~]# perf probe -d *:* Removed event: probe:sys_epoll_wait [root@jouet ~]# perf probe -v sys_epoll_wait probe-definition(0): sys_epoll_wait symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments Looking at the vmlinux_path (8 entries long) symsrc__init: cannot get elf header. Could not open debuginfo. Try to use symbols. Looking at the vmlinux_path (8 entries long) symsrc__init: cannot get elf header. Using /proc/kcore for kernel object code Using /proc/kallsyms for symbols Opening /sys/kernel/debug/tracing//kprobe_events write=1 Writing event: p:probe/sys_epoll_wait _text+2698592 Added new event: probe:sys_epoll_wait (on sys_epoll_wait) You can now use it in all perf tools, such as: perf record -e probe:sys_epoll_wait -aR sleep 1 [root@jouet ~]# Ok, installing a matching kernel vmlinux: [root@jouet ~]# perf probe -d *:* Removed event: probe:sys_epoll_wait [root@jouet ~]# dnf debuginfo-install kernel Trying again: [root@jouet ~]# perf probe -v sys_epoll_wait probe-definition(0): sys_epoll_wait symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments Looking at the vmlinux_path (8 entries long) Using /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux for symbols Open Debuginfo file: /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux Try to find probe point from debuginfo. Symbol sys_epoll_wait address found : ffffffff81292d60 Matched function: SyS_epoll_wait found inline addr: 0xffffffff812930f3 Probe point found: compat_SyS_epoll_pwait+147 found inline addr: 0xffffffff81292ed6 Probe point found: SyS_epoll_pwait+134 found inline addr: 0xffffffff81292d60 Probe point found: SyS_epoll_wait+0 Found 3 probe_trace_events. Opening /sys/kernel/debug/tracing//kprobe_events write=1 Writing event: p:probe/sys_epoll_wait _text+2699507 Writing event: p:probe/sys_epoll_wait_1 _text+2698966 Writing event: p:probe/sys_epoll_wait_2 _text+2698592 Added new events: probe:sys_epoll_wait (on sys_epoll_wait) probe:sys_epoll_wait_1 (on sys_epoll_wait) probe:sys_epoll_wait_2 (on sys_epoll_wait) You can now use it in all perf tools, such as: perf record -e probe:sys_epoll_wait_2 -aR sleep 1 [root@jouet ~]# So, a distro kernel works with or without vmlinux, but then... [root@jouet ~]# grep RANDOMIZE_BASE /boot/config-4.6.4-301.fc24.x86_64 CONFIG_RANDOMIZE_BASE=y CONFIG_RANDOMIZE_BASE_MAX_OFFSET=0x40000000 [root@jouet ~]# It also has kaslr :-\ Rebooting with the kernel where the problem happened, this time with nokaslr: [root@jouet ~]# uname -r 4.7.0+ [root@jouet ~]# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-4.7.0+ root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet LANG=en_US.UTF-8 nokaslr [root@jouet ~]# perf probe -v sys_epoll_wait probe-definition(0): sys_epoll_wait symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments Looking at the vmlinux_path (8 entries long) Using /lib/modules/4.7.0+/build/vmlinux for symbols Open Debuginfo file: /lib/modules/4.7.0+/build/vmlinux Try to find probe point from debuginfo. Symbol sys_epoll_wait address found : ffffffff81295b50 Matched function: SyS_epoll_wait found inline addr: 0xffffffff81295ee7 Probe point found: compat_SyS_epoll_pwait+151 found inline addr: 0xffffffff81295cca Probe point found: SyS_epoll_pwait+138 found inline addr: 0xffffffff81295b50 Probe point found: SyS_epoll_wait+0 Found 3 probe_trace_events. Opening /sys/kernel/debug/tracing//kprobe_events write=1 Writing event: p:probe/sys_epoll_wait _text+2711271 Writing event: p:probe/sys_epoll_wait_1 _text+2710730 Writing event: p:probe/sys_epoll_wait_2 _text+2710352 Added new events: probe:sys_epoll_wait (on sys_epoll_wait) probe:sys_epoll_wait_1 (on sys_epoll_wait) probe:sys_epoll_wait_2 (on sys_epoll_wait) You can now use it in all perf tools, such as: perf record -e probe:sys_epoll_wait_2 -aR sleep 1 [root@jouet ~]# It works, but why was annotation worked when "nokaslr" wasn't used? - Arnaldo