From: Petr Mladek <pmladek@suse.com>
To: Torsten Duwe <duwe@lst.de>
Cc: Kamalesh Babulal <kamalesh@linux.vnet.ibm.com>,
Balbir Singh <bsingharora@gmail.com>,
Jessica Yu <jeyu@redhat.com>,
linux-kernel@vger.kernel.org,
Steven Rostedt <rostedt@goodmis.org>,
Jiri Kosina <jkosina@suse.cz>,
live-patching@vger.kernel.org, Miroslav Benes <mbenes@suse.cz>,
linuxppc-dev@lists.ozlabs.org
Subject: Re: [PATCH v8 8/8] livepatch: Detect offset for the ftrace location during build
Date: Tue, 16 Feb 2016 14:57:42 +0100 [thread overview]
Message-ID: <20160216135742.GT12548@pathway.suse.cz> (raw)
In-Reply-To: <20160216103907.GB25103@lst.de>
On Tue 2016-02-16 11:39:07, Torsten Duwe wrote:
> On Tue, Feb 16, 2016 at 04:00:30PM +0530, Kamalesh Babulal wrote:
> > * Torsten Duwe <duwe@lst.de> [2016-02-16 09:23:02]:
> > >
> > > N.b.: if you try to livepatch/trace such a leaf function without
> > > global dependencies, it will crash if that function got called with
> > > a different TOC value. Hence this whole testing.
> > >
> >
> > I am running out of ideas on how to generate this crash, any pointers
> > will be helpful.
>
> Petr discovered some modular SCSI helper function does it.
The kernel crashes here when I build it _with_ modules and
enable a tracer.
I see this:
$> lsmod
Module Size Used by
af_packet 46767 0
dm_mod 148307 0
e1000 184154 0
rtc_generic 2617 0
ext4 716333 1
crc16 2307 1 ext4
mbcache 14237 1 ext4
jbd2 143496 1 ext4
sd_mod 49002 3
sr_mod 22532 0
cdrom 65024 1 sr_mod
ibmvscsi 34139 2
scsi_transport_srp 18979 1 ibmvscsi
sg 44568 0
scsi_mod 289641 5 sg,scsi_transport_srp,ibmvscsi,sd_mod,sr_mod
autofs4 48256 0
echo function >/sys/kernel/debug/tracing/current_tracer
BANG!
Some dugging has shown an Oops in the fucntion int_to_scsilun()
called from ibmvscsi_queuecommand(). So, I rebooted and
did the following test:
$> echo ibmvscsi_queuecommand >/sys/kernel/debug/tracing/set_ftrace_filter
$> echo function > /sys/kernel/debug/tracing/current_tracer
$> echo 1 > /sys/kernel/debug/tracing/tracing_on
$> cat /sys/kernel/debug/tracing/trace
# tracer: function
#
# entries-in-buffer/entries-written: 7/7 #P:4
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / delay
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
# | | | |||| | |
bash-3488 [000] .... 100.278622: ibmvscsi_queuecommand <-scsi_dispatch_cmd
kworker/1:2-223 [001] .... 101.048569: ibmvscsi_queuecommand <-scsi_dispatch_cmd
kworker/1:2-223 [001] .... 103.048575: ibmvscsi_queuecommand <-scsi_dispatch_cmd
jbd2/sda3-8-1021 [003] .... 104.008645: ibmvscsi_queuecommand <-scsi_dispatch_cmd
jbd2/sda3-8-1021 [003] .... 104.008883: ibmvscsi_queuecommand <-scsi_dispatch_cmd
<idle>-0 [000] ..s. 104.017672: ibmvscsi_queuecommand <-scsi_dispatch_cmd
<idle>-0 [003] ..s. 104.017771: ibmvscsi_queuecommand <-scsi_dispatch_cmd
It means that ibmvscsi_queuecommand can be traced. Then I did
c79:/sys/kernel/debug/tracing # echo int_to_scsilun >set_ftrace_filter
BANG!
Unable to handle kernel paging request for data at address 0xd00000000108b148
Faulting instruction address: 0xd000000000bde35c
Oops: Kernel access of bad area, sig: 11 [#1]
SMP NR_CPUS=2048 NUMA pSeries
Modules linked in: af_packet(E) dm_mod(E) e1000(E) rtc_generic(E) ext4(E) crc16(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sd_mod(E) ibmvscsi(E) scsi_transport_srp(E) sg(E) scsi_mod(E) autofs4(E)
CPU: 1 PID: 223 Comm: kworker/1:2 Tainted: G E 4.5.0-rc2-11-default+ #90
Workqueue: events_freezable_power_ disk_events_workfn
task: c0000000f7d99aa0 ti: c0000000f7cb8000 task.ti: c0000000f7cb8000
NIP: d000000000bde35c LR: d000000000bcffec CTR: d000000000bcffe0
REGS: c0000000f7cbb3b0 TRAP: 0300 Tainted: G E (4.5.0-rc2-11-default+)
MSR: 8000000100009033 <SF,EE,ME,IR,DR,RI,LE,TM[E]> CR: 24c82220 XER: 00000000
CFAR: d000000000bdd144 DAR: d00000000108b148 DSISR: 40000000 SOFTE: 0
GPR00: d0000000010a21cc c0000000f7cbb630 d0000000010aeda0 0000000000008200
GPR04: c0000000fa0b33fc 000000000000014a c0000000fa0b3408 0000000000000010
GPR08: 0000000000000008 c00000000370a4e0 0000000000000000 d00000000108b128
GPR12: d000000000bcffe0 c000000007e80300 c0000000000d8b18 c0000000fe04ba80
GPR16: 0000000000000000 c0000000f7b6f208 0000000000000000 0000000000010000
GPR20: c0000000f7b6f144 c0000000f7b6f140 d000000000bcbcf0 0000000000000000
GPR24: 0000000000000200 0000000000000000 0000000000000001 c0000000f7b6f810
GPR28: c0000000f7b6f000 c0000000f7b6f000 c0000000fa0b33a0 c0000000fe837e00
NIP [d000000000bde35c] scsi_inq_str+0x21b0/0x41ac [scsi_mod]
LR [d000000000bcffec] int_to_scsilun+0xc/0x60 [scsi_mod]
Call Trace:
[c0000000f7cbb630] [d0000000010a21cc] ibmvscsi_queuecommand+0x10c/0x4e0 [ibmvscsi] (unreliable)
[c0000000f7cbb6e0] [d000000000bcbea8] scsi_dispatch_cmd+0xe8/0x2c0 [scsi_mod]
[c0000000f7cbb760] [d000000000bceb0c] scsi_request_fn+0x50c/0x8b0 [scsi_mod]
[c0000000f7cbb850] [c000000000407280] __blk_run_queue+0x60/0x90
[c0000000f7cbb880] [c000000000413640] blk_execute_rq_nowait+0x100/0x1a0
[c0000000f7cbb8d0] [c000000000413768] blk_execute_rq+0x88/0x170
[c0000000f7cbb9b0] [d000000000bca048] scsi_execute+0x108/0x1d0 [scsi_mod]
[c0000000f7cbba20] [d000000000bca2e8] scsi_execute_req_flags+0xc8/0x150 [scsi_mod]
[c0000000f7cbbae0] [d0000000012209e4] sr_check_events+0xb4/0x340 [sr_mod]
[c0000000f7cbbb90] [d0000000011c00b4] cdrom_check_events+0x44/0x80 [cdrom]
[c0000000f7cbbbc0] [d000000001220fa4] sr_block_check_events+0x44/0x60 [sr_mod]
[c0000000f7cbbbe0] [c0000000004222f8] disk_check_events+0x78/0x1b0
[c0000000f7cbbc50] [c0000000000d0610] process_one_work+0x1a0/0x480
[c0000000f7cbbce0] [c0000000000d0998] worker_thread+0xa8/0x5c0
[c0000000f7cbbd80] [c0000000000d8c24] kthread+0x114/0x140
[c0000000f7cbbe30] [c000000000009538] ret_from_kernel_thread+0x5c/0xa4
Instruction dump:
396bc360 f8410018 e98b0020 7d8903a6 4e800420 00000000 00000000 0044fb30
c0000000 3d62fffe 396bc388 60000000 <e98b0020> 7d8903a6 4e800420 00000000
---[ end trace 3b830c669dd7adb5 ]---
Note that ibmvscsi_queuecommand() handle TOC and int_to_scsilun()
does not handle TOC
$> objdump -hdr drivers/scsi/ibmvscsi/ibmvscsi.ko
00000000000020c0 <ibmvscsi_queuecommand>:
20c0: 00 00 4c 3c addis r2,r12,0
20c0: R_PPC64_REL16_HA .TOC.
20c4: 00 00 42 38 addi r2,r2,0
20c4: R_PPC64_REL16_LO .TOC.+0x4
20c8: a6 02 08 7c mflr r0
20cc: 10 00 01 f8 std r0,16(r1)
20d0: 01 00 00 48 bl 20d0 <ibmvscsi_queuecommand+0x10>
20d0: R_PPC64_REL24 _mcount
20d4: a6 02 08 7c mflr r0
$> objdump -hdr drivers/scsi/scsi_mod.ko
[...]
000000000000ffe0 <int_to_scsilun>:
ffe0: a6 02 08 7c mflr r0
ffe4: 10 00 01 f8 std r0,16(r1)
ffe8: 01 00 00 48 bl ffe8 <int_to_scsilun+0x8>
ffe8: R_PPC64_REL24 _mcount
ffec: a6 02 08 7c mflr r0
fff0: 10 00 01 f8 std r0,16(r1)
fff4: e1 ff 21 f8 stdu r1,-32(r1)
I am able to trace int_to_scsilun() if I add a global call
into this fucntion.
I could trace any function when everything is built into the kernel.
I am not sure where the problem is but even normal tracing does not
work with this patchset and when -pg -mprofile-kernel is used.
Best Regards,
Petr
next prev parent reply other threads:[~2016-02-16 13:57 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-02-10 17:29 [PATCH v8 0/8] ftrace with regs + live patching for ppc64 LE (ABI v2) Torsten Duwe
2016-01-28 15:32 ` [PATCH v8 8/8] livepatch: Detect offset for the ftrace location during build Torsten Duwe
2016-02-12 16:13 ` Balbir Singh
2016-02-12 16:45 ` Petr Mladek
2016-02-13 1:33 ` Balbir Singh
2016-02-16 5:47 ` Kamalesh Babulal
2016-02-16 8:23 ` Torsten Duwe
2016-02-16 10:30 ` Kamalesh Babulal
2016-02-16 10:39 ` Torsten Duwe
2016-02-16 13:57 ` Petr Mladek [this message]
2016-02-17 3:08 ` Michael Ellerman
2016-02-23 17:00 ` Torsten Duwe
2016-02-24 6:37 ` Balbir Singh
2016-02-24 6:55 ` Balbir Singh
2016-02-24 9:23 ` Torsten Duwe
2016-02-24 11:22 ` Balbir Singh
2016-02-24 7:51 ` Kamalesh Babulal
2016-02-10 16:21 ` [PATCH v8 1/8] ppc64 (le): prepare for -mprofile-kernel Torsten Duwe
2016-02-17 10:55 ` Michael Ellerman
2016-02-17 11:30 ` Torsten Duwe
2016-02-17 11:39 ` Michael Ellerman
2016-02-10 16:22 ` [PATCH v8 2/8] ppc64le FTRACE_WITH_REGS implementation Torsten Duwe
2016-02-10 16:22 ` [PATCH v8 3/8] ppc use ftrace_modify_all_code default Torsten Duwe
2016-02-10 16:25 ` [PATCH v8 4/8] ppc64 ftrace_with_regs configuration variables Torsten Duwe
2016-02-11 7:48 ` Balbir Singh
2016-02-11 8:39 ` Kamalesh Babulal
2016-02-11 9:35 ` Balbir Singh
2016-02-11 13:00 ` Murali Sampath
2016-02-11 13:00 ` Murali Sampath
2016-02-11 8:42 ` Torsten Duwe
2016-02-11 9:34 ` Balbir Singh
2016-02-15 10:27 ` Michael Ellerman
2016-02-15 12:56 ` Jiri Kosina
2016-02-15 14:04 ` Torsten Duwe
2016-02-15 22:21 ` Torsten Duwe
2016-02-16 4:53 ` Balbir Singh
2016-02-16 10:09 ` Michael Ellerman
2016-02-16 10:35 ` Torsten Duwe
2016-02-10 16:27 ` [PATCH v8 5/8] ppc64 ftrace_with_regs: disable profiling for some files Torsten Duwe
2016-02-10 16:34 ` [PATCH v8 6/8] Implement kernel live patching for ppc64le (ABIv2) Torsten Duwe
2016-02-11 9:01 ` Miroslav Benes
2016-02-10 16:36 ` [PATCH v8 7/8] Enable LIVEPATCH to be configured on ppc64le and add livepatch.o if it is selected Torsten Duwe
2016-02-11 6:18 ` [PATCH v8 0/8] ftrace with regs + live patching for ppc64 LE (ABI v2) Balbir Singh
2016-02-11 8:38 ` Torsten Duwe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160216135742.GT12548@pathway.suse.cz \
--to=pmladek@suse.com \
--cc=bsingharora@gmail.com \
--cc=duwe@lst.de \
--cc=jeyu@redhat.com \
--cc=jkosina@suse.cz \
--cc=kamalesh@linux.vnet.ibm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=live-patching@vger.kernel.org \
--cc=mbenes@suse.cz \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).