From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S933727AbcHDMtK (ORCPT ); Thu, 4 Aug 2016 08:49:10 -0400 Received: from mail.kernel.org ([198.145.29.136]:46674 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S933323AbcHDMtH (ORCPT ); Thu, 4 Aug 2016 08:49:07 -0400 Date: Thu, 4 Aug 2016 09:48:57 -0300 From: Arnaldo Carvalho de Melo To: Masami Hiramatsu Cc: "Wangnan (F)" , Linux Kernel Mailing List Subject: Re: perf test BPF failing on f24: fix Message-ID: <20160804124857.GH14639@kernel.org> References: <20160802195102.GD14639@kernel.org> <57A1A913.6000307@huawei.com> <20160803234557.29f43f755b7e14c634a54a9a@kernel.org> <20160803200415.GG14639@kernel.org> <20160804153221.8aa29dfc5533b14089bd7470@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160804153221.8aa29dfc5533b14089bd7470@kernel.org> X-Url: http://acmel.wordpress.com User-Agent: Mutt/1.6.2 (2016-07-01) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Em Thu, Aug 04, 2016 at 03:32:21PM +0900, Masami Hiramatsu escreveu: > On Wed, 3 Aug 2016 17:04:15 -0300 > Arnaldo Carvalho de Melo wrote: > > Em Wed, Aug 03, 2016 at 11:45:57PM +0900, Masami Hiramatsu escreveu: > > > > If we remove vmlinux, perf should use /proc/kallsyms. I think > > I am not removing vmlinux, it is being used, its just that the function > > chosen by the 'perf test BPF' testcase isn't there. > > So lets go again trying to chase this without missing a single step of > > the way: > > We start with: > > > > [root@jouet ~]# perf test bpf > > 37: Test BPF filter : > > 37.1: Test basic BPF filtering : FAILED! > > 37.2: Test BPF prologue generation : Skip > > 37.3: Test BPF relocation checker : Skip > > [root@jouet ~]# > > > > Ok, so we add -v to get more information: > > > > [root@jouet ~]# perf test -v bpf > > > > bpf: config program 'func=sys_epoll_wait' > > symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null) > > bpf: config 'func=sys_epoll_wait' is ok > > Looking at the vmlinux_path (8 entries long) > > Using /lib/modules/4.7.0+/build/vmlinux for symbols > > Open Debuginfo file: /lib/modules/4.7.0+/build/vmlinux > > Try to find probe point from debuginfo. > > Symbol sys_epoll_wait address found : ffffffffbd295b50 > > Failed to find debug information for address ffffffffbd295b50 > > Probe point 'sys_epoll_wait' not found. > > bpf_probe: failed to convert perf probe eventsFailed to add events > > selected by BPF > > test child finished with -1 > > ---- end ---- > > Test BPF filter subtest 0: FAILED! > > > > -------------- > > > > See? It _is_ using /lib/modules/4.7.0+/build/vmlinux, and it should > > because: > > > > [acme@jouet linux]$ file /lib/modules/4.7.0+/build/vmlinux > > /lib/modules/4.7.0+/build/vmlinux: ELF 64-bit LSB executable, x86-64, > > version 1 (SYSV), statically linked, > > BuildID[sha1]=a08d121dcee2a0ea0cfa5d84363de0c1cfdc729a, not stripped > > [acme@jouet linux]$ > > > > Its the kernel that is in use: > > > > [acme@jouet linux]$ perf buildid-list --kernel > > a08d121dcee2a0ea0cfa5d84363de0c1cfdc729a > > > [acme@jouet linux]$ perf buildid-list -h --kernel > > > > Usage: perf buildid-list [] > > > > -k, --kernel Show current kernel build id > > > > [acme@jouet linux]$ > > > > And, in this vmlinux file, there is _no_ such function: > > > > [acme@jouet linux]$ readelf -wi /lib/modules/4.7.0+/build/vmlinux | grep -w sys_epoll_wait > > [acme@jouet linux]$ > > > > Exactly like the 'perf probe -v bpf' says: > > > > Symbol sys_epoll_wait address found : ffffffffbd295b50 > > Failed to find debug information for address ffffffffbd295b50 > > > > ----- > > > > It mapped it to an address, sure, it found it in /proc/kallsyms, but > > then it didn't find it in the matching vmlinux file. > > > > Since the test was working before, when did it stop to be available on > > vmlinux? > > Would you remember the previous test was done with this kernel or other one? > > Actually, I could put probes on sys_epoll_wait on F24 standard kernel. No, you did not, you put probes in SyS_epoll_wait, because the part that is failing for me, i.e. mapping sys_epoll_wait kallsyms's addr to SyS_epoll_wait's addr is working for you. I will test with the standard kernel for f24, this is my hunch as well, that something on the 4.7.0 kernel series (or one after the f24/ubuntu-you-use ones) is causing this, will diff the configs more below > ----- > [mhiramat@devbox perf]$ sudo ./perf probe -vnf sys_epoll_wait > probe-definition(0): sys_epoll_wait > symbol:sys_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null) > 0 arguments > Looking at the vmlinux_path (8 entries long) > Using /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux for symbols > Open Debuginfo file: /usr/lib/debug/lib/modules/4.6.4-301.fc24.x86_64/vmlinux > Try to find probe point from debuginfo. > Symbol sys_epoll_wait address found : ffffffff81292d60 > Matched function: SyS_epoll_wait > found inline addr: 0xffffffff812930f3 > Probe point found: compat_SyS_epoll_pwait+147 > found inline addr: 0xffffffff81292ed6 > Probe point found: SyS_epoll_pwait+134 > found inline addr: 0xffffffff81292d60 > Probe point found: SyS_epoll_wait+0 > Found 3 probe_trace_events. > Opening /sys/kernel/debug/tracing//kprobe_events write=1 > Writing event: p:probe/sys_epoll_wait _text+2699507 > Writing event: p:probe/sys_epoll_wait_1 _text+2698966 > Writing event: p:probe/sys_epoll_wait_2 _text+2698592 > Added new events: > probe:sys_epoll_wait (on sys_epoll_wait) > probe:sys_epoll_wait_1 (on sys_epoll_wait) > probe:sys_epoll_wait_2 (on sys_epoll_wait) > > You can now use it in all perf tools, such as: > > perf record -e probe:sys_epoll_wait_2 -aR sleep 1 > > ----- > FYI, if perf-probe failed to find the symbol in debuginfo, it tries to find its address > from symbol table(kallsyms, here), and then tries to convert the address to the symbol > in debuginfo again. It seems that in your case this convert process is failed. Yeah, that is what is failing for me. > Then, could you grep DEBUG_INFO in .config? I guess your kernel enables some > reduced debuginfo related config enabled... If that is the case, then we better add a proper warning because this is very subtle :-) Checking... [acme@jouet linux]$ grep ^CONFIG_DEBUG_ ../build/v4.7.0+/.config | grep 'INFO\|REDUCED' CONFIG_DEBUG_INFO=y [acme@jouet linux]$ Nope. - Arnaldo