From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Cyrus-Session-Id: sloti22d1t05-1211982-1517542056-2-2769553060271730382 X-Sieve: CMU Sieve 3.0 X-Spam-known-sender: no X-Spam-score: 0.0 X-Spam-hits: BAYES_00 -1.9, HEADER_FROM_DIFFERENT_DOMAINS 0.001, RCVD_IN_DNSWL_NONE -0.0001, LANGUAGES en, BAYES_USED global, SA_VERSION 3.4.0 X-Spam-source: IP='96.67.55.147', Host='shelob.surriel.com', Country='US', FromHeader='org', MailFrom='org' X-Spam-charsets: plain='us-ascii' X-Resolved-to: greg@kroah.com X-Delivered-to: greg@kroah.com X-Mail-from: kernelnewbies-bounces@kernelnewbies.org ARC-Seal: i=1; a=rsa-sha256; cv=none; d=messagingengine.com; s=arctest; t=1517542055; b=A2Fq+ljTgEiLzPQhxbXk8pFpA2KIyUGgVOqNU6JHH00hsRk eJaiPoxWkUBisiicOy64tpMHZ/WMiMHpz9n1ZWrGf5J62spPafsNTqhC6SfK1kpQ fjSn35k0ycGY33IXk+TzzY4ItoB+wfeK9WkYrqsKzGghowWLSLanGkONuODvvBEC pPXsRqwrDOMxlZQqvja2fMXhcOB7Qic3f8Q2CJ3njoyituSLM2b3BJCegG8btqiM 5nLFcnhEDGPYGUchK7xFtrtbM2DDpz+ivO+NAP0EEjDChcrul3TxDN4mRlQKMi1/ zZ3d9ERg2vnJ+ZQ/qZWHp7Bjae5ZvUD2lJetlKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=date:from:to:subject:message-id :in-reply-to:references:mime-version:cc:list-id:list-unsubscribe :list-archive:list-post:list-help:list-subscribe:content-type :content-transfer-encoding; s=arctest; t=1517542055; bh=aCfRhFQl D0Tq9fSqrbdRaQxfNVn4SRoCx5++KDj+6ag=; b=a/oR9+k4AAG3nrEPcYpRT63r bqlWFFtR+DTnkSTy8co0hXIxcPiFIbV8KRTse2Tjxuvq4VTrSRfqc/uGkdz1E5Dy v7rj/HAirKlpToK2mNqMxSLFjMSxGmw1cqVU3CLOFkvIv96yf2+jgqAa0UY8Uo+0 6pNu/vlT1nzqz9fnFV0IIouMadAy15UU+b1CAmsLqD7abUALkSxD/MgDYTX9Ms/b Xf7c4RLJBCyt3saAC4ycsxUPmwZHAHdczpyjXHmM3oV8NQAUQhA4fFATX8OtEbGv e4BSc7dk/9yM82AVW6ZoNpaqWlWI9vXZEkGcPgMh/5T1nT2JG/nTOz38lf48rA== ARC-Authentication-Results: i=1; mx3.messagingengine.com; arc=none (no signatures found); dkim=none (no signatures found); dmarc=none (p=none,has-list-id=yes,d=none) header.from=kernel.org; iprev=pass policy.iprev=96.67.55.147 (shelob.surriel.com); spf=permerror smtp.mailfrom=kernelnewbies-bounces@kernelnewbies.org smtp.helo=shelob.surriel.com; x-aligned-from=fail; x-ptr=pass x-ptr-helo=shelob.surriel.com x-ptr-lookup=shelob.surriel.com; x-return-mx=pass smtp.domain=kernelnewbies.org smtp.result=pass smtp_is_org_domain=yes header.domain=kernel.org header.result=pass header_is_org_domain=yes; x-tls=pass version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128 Authentication-Results: mx3.messagingengine.com; arc=none (no signatures found); dkim=none (no signatures found); dmarc=none (p=none,has-list-id=yes,d=none) header.from=kernel.org; iprev=pass policy.iprev=96.67.55.147 (shelob.surriel.com); spf=permerror smtp.mailfrom=kernelnewbies-bounces@kernelnewbies.org smtp.helo=shelob.surriel.com; x-aligned-from=fail; x-ptr=pass x-ptr-helo=shelob.surriel.com x-ptr-lookup=shelob.surriel.com; x-return-mx=pass smtp.domain=kernelnewbies.org smtp.result=pass smtp_is_org_domain=yes header.domain=kernel.org header.result=pass header_is_org_domain=yes; x-tls=pass version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128 DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 56C742173F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=mhiramat@kernel.org Date: Wed, 31 Jan 2018 12:31:06 +0900 From: Masami Hiramatsu To: Pintu Kumar Subject: Re: [perf] perf probe fails sometimes on 4.9 Message-Id: <20180131123106.e48e89ab21cceee95f88f405@kernel.org> In-Reply-To: References: <20180129231723.524a8c32e1456358f7cb88ce@kernel.org> <20180130144346.86307346478928ca872473b5@kernel.org> X-Mailer: Sylpheed 3.5.0 (GTK+ 2.24.30; x86_64-pc-linux-gnu) Mime-Version: 1.0 X-Mailman-Approved-At: Thu, 01 Feb 2018 22:23:39 -0500 Cc: "Wangnan \(F\)" , Alexis Berlemont , daniel@iogearbox.net, kernelnewbies@kernelnewbies.org, alexander.shishkin@linux.intel.com, linux-kernel@vger.kernel.org, acme@redhat.com, peterz@infradead.org, mingo@redhat.com, huawei.libin@huawei.com, naveen.n.rao@linux.vnet.ibm.com, quentin.monnet@netronome.com, maddy@linux.vnet.ibm.com, davem@davemloft.net X-BeenThere: kernelnewbies@kernelnewbies.org X-Mailman-Version: 2.1.15 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kernelnewbies-bounces@kernelnewbies.org X-getmail-retrieved-from-mailbox: INBOX X-Mailing-List: linux-kernel@vger.kernel.org List-ID: On Tue, 30 Jan 2018 19:20:36 +0530 Pintu Kumar wrote: > On Tue, Jan 30, 2018 at 11:13 AM, Masami Hiramatsu wrote: > > > > On Mon, 29 Jan 2018 22:00:52 +0530 > > Pintu Kumar wrote: > > > > > Dear Masami, > > > > > > Thank you so much for your reply. > > > Please find some of my answers inline. > > > > > > > > > On Mon, Jan 29, 2018 at 7:47 PM, Masami Hiramatsu wrote: > > > > On Mon, 29 Jan 2018 13:40:34 +0530 > > > > Pintu Kumar wrote: > > > > > > > >> Hi All, > > > >> > > > >> 'perf probe' is failing sometimes on 4.9.20 with AMD-64. > > > >> # perf probe --add schedule > > > >> schedule is out of .text, skip it. > > > >> Error: Failed to add events. > > > >> > > > >> If any one have come across this problem please let me know the cause. > > > > > > > > Hi Pintu, > > > > > > > > Could you run it with --vv? > > > > > > > Ok, I will send verbose output by tomorrow. > > > > > > >> > > > >> Note: I don't have CONFIG_DEBUG_INFO enabled in kernel. Is this the problem? > > > > > > > > Without it, you can not probe source-level probe nor trace local variable. > > > > > > > > > > Currently I am facing problem in enabling DEBUG_INFO in our kernel 4.9.20 > > > However, I will try to manually include "-g" option during compilation. > > > > > > >> However, I manually copied the vmlinux file to /boot/ directory, but > > > >> still it does not work. > > > > > > > > That doesn't work. > > > > CONFIG_DEBUG_INFO option enables gcc to compile kernel with extra debuginfo. > > > > Without that option, debuginfo is not generated with vmlinux. > > > > > > > >> > > > >> I checked upstream patches until 4.15 but could not find any clue. > > > >> Please let me know if there is any fixes available for this. > > > > > > > > Could you also ensure that you run perf by root user? > > > > > > > > > > Yes I am running with root user. > > > > > > My concern is sometimes it works but sometimes it fails. > > > > What I thought was that your kernel enables kptr_strict(but maybe not.) > > Can you also try to find "schedule" and "_etext" functions in > > /proc/kallsyms? > > > > Yes these functions are present in kallsyms. > > # cat /proc/kallsyms | grep -w schedule ; cat /proc/kallsyms | grep -w _etext > ffffffff829328e0 T schedule > ffffffff8293e751 T _etext > > > > > I still needs to figure out, in which condition it works and which > > > condition it fails. > > > > Yeah, it is important. > > > > > Usually I noticed that in fresh reboot case it works. > > > > So, after a while, it doesn't work again? If so, it sounds like a daemon > > process changes settings in background. > > > > Fortunately or unfortunately, now it is passing every time. > Still not sure about the cause. > If it fails again, I will collect info with -v and report again. > Any specific info that I need to check when it fails ? OK, if you hit it again, please run perf-probe with -vvv option (-v is enough usually, if it is unknown bug, -vvv will be better) Thank you, > > Thank You, > Pintu > > > > Thank you, > > > > > > > > > > > > Thank you, > > > > > > > > > > > >> > > > >> > > > >> Thank You! > > > >> Regards, > > > >> Pintu > > > >> > > > >> > > > >> On Thu, Jan 25, 2018 at 7:09 PM, Pintu Kumar wrote: > > > >> > Hi, > > > >> > > > > >> > ** Changed the subject now, since these issues are related to general > > > >> > perf commands. > > > >> > > > > >> > Following are the issues: > > > >> > > > > >> > 1) perf probe --add schedule - FAILED > > > >> > output: > > > >> > schedule is out of .text, skip it. > > > >> > Error: Failed to add events. > > > >> > > > > >> > what is the issue here? > > > >> > Sometimes it pass and sometimes it fails... > > > >> > Similar is the case of 'perf inject' as well. > > > >> > > > > >> > 2) perf test - 1 FAILURE > > > >> > 37.1: Test basic BPF filtering : FAILED! > > > >> > 37.2: Test BPF prologue generation : Skip > > > >> > 37.3: Test BPF relocation checker : Skip > > > >> > > > > >> > bpf: config program 'func=SyS_epoll_wait' > > > >> > symbol:SyS_epoll_wait file:(null) line:0 offset:0 return:0 lazy:(null) > > > >> > bpf: config 'func=SyS_epoll_wait' is ok > > > >> > Looking at the vmlinux_path (8 entries long) > > > >> > Using /boot/vmlinux for symbols > > > >> > Could not open debuginfo. Try to use symbols. > > > >> > SyS_epoll_wait is out of .text, skip it. > > > >> > bpf_probe: failed to convert perf probe eventsFailed to add events > > > >> > selected by BPF > > > >> > test child finished with -1 > > > >> > ---- end ---- > > > >> > Test BPF filter subtest 0: FAILED! > > > >> > > > > >> > Looks like both 1,2 are related. > > > >> > Since, CONFIG_DEBUG_INFO is not enabled, I manually copied the vmlinux > > > >> > to /boot/ folder. > > > >> > > > > >> > ------- > > > >> > Some more info: > > > >> > > > > >> > Kernel build dir is set to /lib/modules/4.9.20-sc-amd-x86-64/build > > > >> > set env: KBUILD_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build > > > >> > unset env: KBUILD_OPTS > > > >> > include option is set to -nostdinc -isystem > > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include > > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated > > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi > > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h > > > >> > set env: NR_CPUS=8 > > > >> > set env: LINUX_VERSION_CODE=0x40914 > > > >> > set env: CLANG_EXEC=/usr/bin/clang > > > >> > set env: CLANG_OPTIONS=-xc > > > >> > set env: KERNEL_INC_OPTIONS= -nostdinc -isystem > > > >> > /usr/lib/gcc/x86_64-linux-gnu/5/include -I./arch/x86/include > > > >> > -I./arch/x86/include/generated/uapi -I./arch/x86/include/generated > > > >> > -I./include -I./arch/x86/include/uapi -I./include/uapi > > > >> > -I./include/generated/uapi -include ./include/linux/kconfig.h > > > >> > set env: WORKING_DIR=/lib/modules/4.9.20-sc-amd-x86-64/build > > > >> > > > > >> > > > > >> > If you have any clue about these failure please help me. > > > >> > > > > >> > > > > >> > Thanks, > > > >> > Pintu > > > >> > > > > >> > > > > >> > On Wed, Jan 24, 2018 at 8:23 PM, Pintu Kumar wrote: > > > >> >> Hi, > > > >> >> > > > >> >> Thanks for your help. > > > >> >> Yes it was a sub version issue. > > > >> >> > > > >> >> Earlier, while building the kernel I left the SUBLEVEL as blank. > > > >> >> > > > >> >> make -j8 bindeb-pkg SUBLEVEL=20 > > > >> >> > > > >> >> After passing the correct sublevel now the test is working. > > > >> >> > > > >> >> But still following are failing: > > > >> >> > > > >> >> 16: Try 'import perf' in python, checking link problems : FAILED! > > > >> >> 37.2: Test BPF prologue generation : FAILED! > > > >> >> > > > >> >> > > > >> >> This is the error I get: > > > >> >> > > > >> >> 16: Try 'import perf' in python, checking link problems : > > > >> >> --- start --- > > > >> >> test child forked, pid 7637 > > > >> >> Traceback (most recent call last): > > > >> >> File "", line 1, in > > > >> >> ImportError: No module named perf > > > >> >> test child finished with -1 > > > >> >> ---- end ---- > > > >> >> Try 'import perf' in python, checking link problems: FAILED! > > > >> >> > > > >> >> > > > >> >> Looking at the vmlinux_path (8 entries long) > > > >> >> symsrc__init: cannot get elf header. > > > >> >> Failed to find the path for kernel: Invalid ELF file > > > >> >> bpf_probe: failed to convert perf probe eventsFailed to add events > > > >> >> selected by BPF > > > >> >> test child finished with -1 > > > >> >> ---- end ---- > > > >> >> Test BPF filter subtest 1: FAILED! > > > >> >> > > > >> >> > > > >> >> > > > >> >> Thanks, > > > >> >> Pintu > > > >> >> > > > >> >> > > > >> >> On Wed, Jan 24, 2018 at 6:39 AM, Wangnan (F) wrote: > > > >> >>> > > > >> >>> > > > >> >>> On 2018/1/23 20:37, Pintu Kumar wrote: > > > >> >>>> > > > >> >>>> Hi All, > > > >> >>>> > > > >> >>>> I am verifying all perf tests on Ubuntu-16 x86-64 platform using the > > > >> >>>> kernel version 4.9.20. > > > >> >>>> I have installed several others packages including: clang, llvm > > > >> >>>> > > > >> >>>> But, when I run 'perf test' I get some FAILURE. > > > >> >>>> Specially, 'perf test LLVM' is failing. > > > >> >>>> > > > >> >>>> Please check the below error logs: > > > >> >>>> > > > >> >>>> ---------------------------------------------------- > > > >> >>>> # perf test LLVM > > > >> >>>> 35: Test LLVM searching and compiling : > > > >> >>>> 35.1: Basic BPF llvm compiling test : FAILED! > > > >> >>>> 35.2: Test kbuild searching : Skip > > > >> >>>> 35.3: Compile source for BPF prologue generation test : Skip > > > >> >>>> 35.4: Compile source for BPF relocation test : Skip > > > >> >>>> ---------------------------------------------------- > > > >> >>>> > > > >> >>>> When I run with -v I get this: > > > >> >>>> > > > >> >>>> ----------------------------------------------------- > > > >> >>>> # perf test -v LLVM > > > >> >>>> 35: Test LLVM searching and compiling : > > > >> >>>> 35.1: Basic BPF llvm compiling test : > > > >> >>>> --- start --- > > > >> >>>> test child forked, pid 3304 > > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64' > > > >> >>>> WARNING: unable to get correct kernel building directory. > > > >> >>>> Hint: Set correct kbuild directory using 'kbuild-dir' option in [llvm] > > > >> >>>> section of ~/.perfconfig or set it to "" to suppress kbuild > > > >> >>>> detection. > > > >> >>>> > > > >> >>>> Unablt to get kernel version from uname '4.9-xxxx-amd-x86-64' > > > >> >>>> > > > >> >>>> > > > >> >>>> int _version SEC("version") = LINUX_VERSION_CODE; > > > >> >>>> ' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS > > > >> >>>> -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS > > > >> >>>> $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign > > > >> >>>> -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o > > > >> >>>> - > > > >> >>>> libbpf: 2129190-4a0 doesn't provide kernel version > > > >> >>>> Failed to parse test case 'Basic BPF llvm compiling test' > > > >> >>>> test child finished with -1 > > > >> >>>> ---- end ---- > > > >> >>>> Test LLVM searching and compiling subtest 0: FAILED! > > > >> >>>> ----------------------------------------------------- > > > >> >>>> > > > >> >>>> > > > >> >>>> Basic BPF LLVM compiling test is failing. > > > >> >>>> It reports that bpf could not able to read the kernel version from uname. > > > >> >>>> Is it because of this that 'perf test LLVM' is failing ? > > > >> >>>> > > > >> >>>> My uname says: > > > >> >>>> 4.9-xxxx-amd-x86-64 > > > >> >>> > > > >> >>> > > > >> >>> 'perf test LLVM' requires a well formed uname string (%d.%d.%d). Please see > > > >> >>> tools/perf/fetch_kernel_version . If your distro uses another method to > > > >> >>> report > > > >> >>> kernel version number (from your uname string, sublevel can't be > > > >> >>> determined), > > > >> >>> please help us improve that function like what we have done for ubuntu > > > >> >>> (checking /proc/version_signature). > > > >> >>> > > > >> >>> > > > >> >>> > > > >> >>>> My kernel version is: 4.9.20 (from Makefile) > > > >> >>>> > > > >> >>>> So, I dont think there is any problem with kernel version. > > > >> >>>> > > > >> >>>> If any body have come across this problem please let me know how to > > > >> >>>> resolve this issue. > > > >> >>>> > > > >> >>>> > > > >> >>>> > > > >> >>>> Thank You! > > > >> >>>> > > > >> >>>> Regards, > > > >> >>>> Pintu > > > >> >>> > > > >> >>> > > > >> >>> > > > > > > > > > > > > -- > > > > Masami Hiramatsu > > > > > > -- > > Masami Hiramatsu -- Masami Hiramatsu _______________________________________________ Kernelnewbies mailing list Kernelnewbies@kernelnewbies.org https://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies