From: tanxiaojun@huawei.com (Tan Xiaojun)
To: linux-arm-kernel@lists.infradead.org
Subject: [arm64] OOPS when using /proc/kcore to disassemble the kernel symbols in "perf top"
Date: Wed, 19 Apr 2017 16:01:41 +0800 [thread overview]
Message-ID: <58F71965.2000008@huawei.com> (raw)
In-Reply-To: <8499425b-2b92-5b1f-a89f-6e7cd443232f@redhat.com>
On 2017/4/19 12:14, Pratyush Anand wrote:
> Hi tan,
>
> On Saturday 15 April 2017 02:18 PM, Tan Xiaojun wrote:
>> Hi,
>>
>> My test server is Hisilicon D03/D05 (arm64).
>> Kernel source code is 4.11-rc6 (up to date) and config (as an attachment in the end) is generated by defconfig.
>> (Old version does not seem to have this problem. Linux-4.1 is fine and other versions I have not tested yet.)
>
> I tested with mustang(ARM64) and 4.11-rc6 and could not reproduce it.
>
This is important. Thanks a lot.
>>
>> When I do "perf top" and annotate a random kernel symbol (like vsnprintf or others), the system report an OOPS below:
>> (The probability of occurrence is very high, almost every time.)
>>
>> $ perf top
>>
>> Annotate vsnprintf ---- choose it
>> Zoom into perf(7066) thread
>> Zoom into the Kernel DSO
>> Browse map details
>> Run scripts for samples of thread [perf]
>> Run scripts for samples of symbol [vsnprintf]
>> Run scripts for all samples
>> Exit
>>
>> log:
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913498] Unable to handle kernel paging request at virtual address ffffdb16aa14028c
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913502] pgd = ffff803f70b29000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913506] [ffffdb16aa14028c] *pgd=0000000000000000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913511] Internal error: Oops: 96000004 [#1] PREEMPT SMP
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913514] Modules linked in:
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913520] CPU: 6 PID: 9703 Comm: perf Not tainted 4.11.0-rc6-00029-gb9b3322 #3
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913523] Hardware name: Huawei Taishan 2180 /BC11SPCC, BIOS 1.31 06/23/2016
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913526] task: ffff803f6ff99a00 task.stack: ffff803f4c104000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913531] PC is at __memcpy+0x38/0x180
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913535] LR is at vread+0x148/0x284
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913538] pc : [<ffff0000083926b8>] lr : [<ffff0000081ba2a0>] pstate: 00000145
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913540] sp : ffff803f4c107c70
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913542] x29: ffff803f4c107c70 x28: ffff803f5ef73000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913548] x27: 000000000000032c x26: ffff803f6ff99a00
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913552] x25: ffff00000839d28c x24: ffff803f7f801380
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913557] x23: 000000000000032c x22: ffff803f5ef73000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913561] x21: 000000000000028c x20: ffff00000839d28c
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913565] x19: 000000000000032c x18: 0000ffffaa6cc2d0
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913569] x17: 0000ffffab9dc350 x16: ffff0000081f5f04
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913573] x15: 0000317ba8000000 x14: 001c19d1d0000000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913577] x13: 00000003e8000000 x12: 0000000000000006
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913581] x11: 0000000000000007 x10: 0000000000000000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913586] x9 : 0000000000000000 x8 : ffff000008e6d3d8
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913590] x7 : 00005b16aa140000 x6 : ffff803f5ef73000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913594] x5 : 0000000000000d74 x4 : 0000000000000004
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913598] x3 : 0000000000000000 x2 : 0000000000000328
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913602] x1 : ffffdb16aa14028c x0 : ffff803f5ef73000
>
> So, source pointer for the memcpy(2nd arg) seems wrong. (Unable to handle kernel paging request at virtual address ffffdb16aa14028c, and x1 is ffffdb16aa14028c).
>
> I tried to look into code around source pointer,and they seem to be correct.
>
> in read_kcore(): start is source pointer.
> start = kc_offset_to_vaddr(*fpos - elf_buflen);
> vread(buf, (char *)start, tsz);
>
> in vread() -> aligned_vread(): addr is source pointer.
> offset = offset_in_page(addr);
> p = vmalloc_to_page(addr);
> *map = kmap_atomic(p);
> memcpy(buf, map + offset, length);
>
> They all look fine and should work. Since, it always works with 4.1 on your platform, can you please try a git bisect to see which commit is causing trouble for you.
>
> ~Pratyush
>
Yes, Pratyush, you are right. I am working on it now.
Thank you for your reply.
Xiaojun.
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913606]
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913609] Process perf (pid: 9703, stack limit = 0xffff803f4c104000)
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913612] Stack: (0xffff803f4c107c70 to 0xffff803f4c108000)
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913615] 7c60: ffff803f4c107d00 ffff000008267a18
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913619] 7c80: 000000000000032c 0000000036dd9c10 ffff000008f75160 ffff803f4c107eb8
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913622] 7ca0: 0000000000000000 ffff803f6ff99a00 ffff803f5ef73000 ffff000008e6d3d8
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913625] 7cc0: ffff00000839d28c 000000000000032c 0000000000000024 ffff803f5ef73000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913629] 7ce0: 000000000000032c 000000000000032c ffff803f6ff99a00 ffff000008e684a0
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913632] 7d00: ffff803f4c107d90 ffff000008259d00 ffff803f720c3d00 fffffffffffffffb
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913635] 7d20: 0000000036dd9c10 ffff803f4c107eb8 0000000080000000 0000000000000015
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913638] 7d40: 0000000000000124 000000000000003f ffff000008942000 ffff803f6ff99a00
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913641] 7d60: ffff803f6ff08310 ffff803f6ff99a00 ffff803f6ff99a00 ffff803f6ff99a00
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913644] 7d80: 0000000d00000124 0000000000002000 ffff803f4c107db0 ffff0000081f3810
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913647] 7da0: ffff803f6ff08300 ffff803f4c107eb8 ffff803f4c107e30 ffff0000081f4ab0
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913650] 7dc0: 000000000000032c ffff803f6ff08300 0000000000000000 0000000000000000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913653] 7de0: ffff803f4c107e10 ffff0000081f49ac ffff803f6ff08300 0000000000000000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913656] 7e00: 0000000036dd9c10 ffff803f4c107eb8 ffff803f4c107e30 ffff0000081f4a8c
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913659] 7e20: 000000000000032c ffff803f6ff08300 ffff803f4c107e70 ffff0000081f5f48
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913662] 7e40: ffff803f6ff08303 ffff803f6ff08300 ffffffffffffffff 0000ffffab9dc37c
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913664] 7e60: 0000000000000200 0000ffffab9dcbdc 0000000000000000 ffff000008082f8c
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913667] 7e80: 0000000000000200 0000803ff70f9000 ffffffffffffffff ffff000008082f5c
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913670] 7ea0: 0000000036dd9c10 000000000000032c ffffffffffffffff 000000000839f28c
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913673] 7ec0: 000000000000002a 0000000036dd9c10 000000000000032c 0000ffffaa6d42c8
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913676] 7ee0: 0000ffffaa6cc49c 0000ffffaa6d41c0 0000ffffaa6d48b0 0000000000000000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913679] 7f00: 000000000000003f 0000000000000003 0000000000000020 0000000000000007
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913682] 7f20: 0000000000000006 00000003e8000000 001c19d1d0000000 0000317ba8000000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913685] 7f40: 0000000000000000 0000ffffab9dc350 0000ffffaa6cc2d0 0000000000622000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913688] 7f60: 0000000000001000 0000000036dd9c10 000000000000032c 00000000006f1038
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913691] 7f80: 000000000000002b 000000000000002a 000000000839f28c 0000000000000001
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913694] 7fa0: 0000ffffaa6d3990 0000ffffaa6cc4e0 0000ffffab9dc368 0000ffffaa6cc4a0
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913697] 7fc0: 0000ffffab9dc37c 0000000080000000 000000000000002a 000000000000003f
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913700] 7fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913702] Call trace:
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913705] Exception stack(0xffff803f4c107aa0 to 0xffff803f4c107bd0)
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913708] 7aa0: 000000000000032c 0001000000000000 ffff803f4c107c70 ffff0000083926b8
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913712] 7ac0: 00000000014200ca 0000000000000000 ffff803f71b1ec38 0000000000000000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913715] 7ae0: ffff803f6ff99a00 0000000036dda000 0000000000000000 0000000000000000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913718] 7b00: 000000000000000c ffff000008f6c610 ffff803f4c107b60 ffff0000082c0ae0
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913721] 7b20: ffff803f7047a030 ffff000008f76000 0000000000000000 ffff803f7200a800
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913724] 7b40: ffff803f5ef73000 ffffdb16aa14028c 0000000000000328 0000000000000000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913727] 7b60: 0000000000000004 0000000000000d74 ffff803f5ef73000 00005b16aa140000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913729] 7b80: ffff000008e6d3d8 0000000000000000 0000000000000000 0000000000000007
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913732] 7ba0: 0000000000000006 00000003e8000000 001c19d1d0000000 0000317ba8000000
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913735] 7bc0: ffff0000081f5f04 0000ffffab9dc350
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913739] [<ffff0000083926b8>] __memcpy+0x38/0x180
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913743] [<ffff000008267a18>] read_kcore+0x230/0x3b0
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913747] [<ffff000008259d00>] proc_reg_read+0x64/0x90
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913751] [<ffff0000081f3810>] __vfs_read+0x28/0x108
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913754] [<ffff0000081f4ab0>] vfs_read+0x80/0x13c
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913757] [<ffff0000081f5f48>] SyS_read+0x44/0xa0
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913761] [<ffff000008082f8c>] __sys_trace_return+0x0/0x4
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913765] Code: 36080064 78402423 780024c3 36100064 (b8404423)
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913768] ---[ end trace 6710f03ffe50aedc ]---
>> Apr 17 05:03:59 EulerOS kernel: [ 339.913772] note: perf[9703] exited with preempt_count 2
>>
>> Call relationship:
>> read_kcore -> vread -> aligned_vread -> memcpy -> __memcpy
>>
>> Maybe you can give me some ideas.
>>
>> Thanks a lot.
>>
>> Xiaojun.
>>
>>
>>
>>
>
> .
>
next prev parent reply other threads:[~2017-04-19 8:01 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-02-23 6:04 [PATCH] perf/core: Fix to check perf_cpu_time_max_percent Tan Xiaojun
2017-02-24 9:15 ` [tip:perf/urgent] perf/core: Fix the perf_cpu_time_max_percent check tip-bot for Tan Xiaojun
2017-02-25 8:10 ` Tan Xiaojun
2017-02-25 8:10 ` Tan Xiaojun
2017-02-25 9:51 ` Peter Zijlstra
2017-02-25 9:51 ` Peter Zijlstra
2017-02-27 0:36 ` Tan Xiaojun
2017-02-27 0:36 ` Tan Xiaojun
2017-03-13 18:35 ` Vince Weaver
2017-03-14 1:52 ` Tan Xiaojun
2017-04-15 7:58 ` [BUG arm64] OOPS when using /proc/kcore to disassemble the kernel symbols in "perf top" Tan Xiaojun
[not found] ` <58F1DE6A.7050307@huawei.com>
2017-04-19 4:14 ` [arm64] " Pratyush Anand
2017-04-19 8:01 ` Tan Xiaojun [this message]
2017-04-19 11:49 ` Mark Rutland
2017-04-20 1:38 ` Tan Xiaojun
2017-04-21 5:46 ` Tan Xiaojun
2017-04-21 9:34 ` Mark Rutland
2017-04-24 1:02 ` Tan Xiaojun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=58F71965.2000008@huawei.com \
--to=tanxiaojun@huawei.com \
--cc=linux-arm-kernel@lists.infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.