* Re: in stable 4.4.131 kprobes seems to be broken [not found] <alpine.LRH.2.00.1807092352480.31641@sjc-ads-6991.cisco.com> @ 2018-07-10 7:05 ` Greg Kroah-Hartman 2018-07-10 7:32 ` Victor Kamensky 0 siblings, 1 reply; 5+ messages in thread From: Greg Kroah-Hartman @ 2018-07-10 7:05 UTC (permalink / raw) To: Victor Kamensky, stable; +Cc: Masami Hiramatsu, systemtap, xe-linux-external [Adding the stable list] On Mon, Jul 09, 2018 at 11:53:37PM -0700, Victor Kamensky wrote: > Hi Masami, > > I am not sure maybe you are aware about this issue already, but > I am writing in case if you are not. > > After move to 4.4.131 stable very basic SystemTap script that involves > kprobes stopped working - it crashes kernel. > > The SystemTap script like this: > > probe kernel.function("meminfo_proc_show") { > println("meminfo_proc_show called") > } > > which effectively will translate into kprobe entry in meminfo_proc_show > function, crashes like this. > > BUG: unable to handle kernel paging request at ffffffffc063f002 > IP: [<ffffffffa3c3558b>] resume_execution+0xe5/0x151 > PGD 24613067 ca > PUD 24615067 PMD 1523e9067 PTE 3370d061 > Oops: 0003 [#1] SMP Modules linked in: <snip> > CPU: 3 PID: 20078 Comm: <snip> Tainted: G O 4.4.131 #1 > task: ffff880031ead780 ti: ffff880032238000 task.ti: ffff880032238000 > t /RIP: 0010:[<ffffffffa3c3558b>] [<ffffffffa3c3558b>] resume_execution+0xe5/0x151 > RSP: 0018:ffff88017fccaec8 EFLAGS: 00010006 > RAX: ffffffffa3dad154 RBX: ffffffffa3dad152 RCX: 000000003f9c0ff9 > RDX: ffffffffc063f002 RSI: ffff88017fccaf58 RDI: 0000000000000041 > RBP: ffff8801549f83c0 R08: 0000000000000001 R09: 0000000000000001 > R10: ffff88003223bdd8 R11: 0000000000000246 R12: ffff88003223bdd8 > R13: ffff88017fccaf58 R14: ffffffffc063f000 R15: ffff88017fccee40 > FS: 00007f9f34520240(0000) GS:ffff88017fcc0000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: ffffffffc063f002 CR3: 0000000152b46000 CR4: 0000000000100670 > Stack: > ffff88017fccee40 ffff88017fccaf58 ffff8801549f83c0 0000000000000000 > ffff880033cde500 0000000000000001 ffffffffa3c35633 ffff88017fccaf58 > ffff880031ead780 0000000000000000 ffffffffa3c04e9d 0000000000004000 > Call Trace: > <#DB> > [<ffffffffa3c35633>] ? kprobe_debug_handler+0x3c/0xc8 > [<ffffffffa3c04e9d>] ? do_debug+0x7a/0x182 > [<ffffffffa4163afe>] ? debug+0x3e/0x70 > [<ffffffffa3dad14d>] ? meminfo_proc_open+0x1c/0x1c > [<ffffffffa3d73111>] ? seq_buf_alloc+0x23/0x3c > <<EOE>> > [<ffffffffa3d735a8>] ? seq_read+0x1a6/0x40d > [<ffffffffa3da5700>] ? proc_reg_read+0x47/0x65 > [<ffffffffa3da56b9>] ? proc_reg_write+0x65/0x65 > [<ffffffffa3d55ebf>] ? __vfs_read+0x34/0xea > [<ffffffffa3d564f9>] ? rw_verify_area+0x7a/0xc9 > [<ffffffffa3d565dc>] ? vfs_read+0x94/0xf9 > [<ffffffffa3d570a4>] ? SyS_read+0x61/0xb4 > [<ffffffffa4161b8a>] ? entry_SYSCALL_64_fastpath+0x1e/0x8e > Code: 7d 70 00 75 3e 49 8b 95 80 00 00 00 49 39 d6 73 2b 48 89 d0 4c 29 f0 > 48 8d 48 05 48 83 f9 0e 77 1b b9 fb ff ff ff 48 01 d8 29 d1 <c6> 02 e9 01 c8 > 89 42 01 c7 45 70 01 00 00 00 eb 07 c7 45 70 ff RIP [<ffffffffa3c3558b>] > resume_execution+0xe5/0x151 > RSP <ffff88017fccaec8> > CR2: ffffffffc063f002 > > Decoding crash: > > (gdb) p &resume_execution > $1 = (void (*)(struct kprobe *, struct pt_regs *, > struct kprobe_ctlblk *)) 0xffffffff810354a6 <resume_execution> > (gdb) b *(0xffffffff810354a6 + 0xe5) > Breakpoint 1 at 0xffffffff8103558b: file <snip>/kernel-source/arch/x86/kernel/kprobes/core.c, line 126. > > corresponds to: > > static nokprobe_inline void > __synthesize_relative_insn(void *from, void *to, u8 op) > { > struct __arch_relative_insn { > u8 op; > s32 raddr; > } __packed *insn; > > insn = (struct __arch_relative_insn *)from; > insn->raddr = (s32)((long)(to) - ((long)(from) + 5)); > insn->op = op; <---------------------------------------- > } > > > (gdb) p /x &kprobe_debug_handler > $1 = 0xffffffff810355f7 > (gdb) b *(0xffffffff810355f7 + 0x3c) > Breakpoint 1 at 0xffffffff81035633: file <snip>/kernel-source/arch/x86/kernel/kprobes/core.c, line 932. > > corresponds to > > static void resume_execution(struct kprobe *p, struct pt_regs *regs, > struct kprobe_ctlblk *kcb) > { > <snip> > if (p->ainsn.boostable == 0) { > if ((regs->ip > copy_ip) && > (regs->ip - copy_ip) + 5 < MAX_INSN_SIZE) { > /* > * These instructions can be executed directly if it > * jumps back to correct address. > */ > synthesize_reljump((void *)regs->ip, > (void *)orig_ip + (regs->ip - copy_ip)); <------- > p->ainsn.boostable = 1; > } else { > p->ainsn.boostable = -1; > } > } > > Further digging points to the following couple commits that 4.4.y stable > picked up: > > 1dd26ec7 kprobes/x86: Fix to set RWX bits correctly before releasing trampoline > 76bee4 kprobes/x86: Set kprobes pages read-only > > Effectively the way I read: the problem is that linux-4.4.y branch is > missing your following commit > > commit 804dec5bda9b4fcdab5f67fe61db4a0498af5221 > Author: Masami Hiramatsu <mhiramat@kernel.org> > Date: Wed Mar 29 14:00:25 2017 +0900 > > kprobes/x86: Do not modify singlestep buffer while resuming > > Do not modify singlestep execution buffer (kprobe.ainsn.insn) > while resuming from single-stepping, instead, modifies > the buffer to add a jump back instruction at preparing > buffer. > > Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> > Cc: Ananth N Mavinakayanahalli <ananth@linux.vnet.ibm.com> > > and as result synthesize_reljump function call still exists from > resume_execution function but it does not have proper page permission > manipulation code around it, so making kprobes pages as > read-only has averse effect: I.e when called from resume_execution > function it hits read-only page and crashes. > > Please advise how to proceed besides obvious backing out above mentioned > two commits. So if you apply the above commit, all works properly for you? thanks, greg k-h ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: in stable 4.4.131 kprobes seems to be broken 2018-07-10 7:05 ` in stable 4.4.131 kprobes seems to be broken Greg Kroah-Hartman @ 2018-07-10 7:32 ` Victor Kamensky 2018-07-10 12:26 ` Jack Wang 0 siblings, 1 reply; 5+ messages in thread From: Victor Kamensky @ 2018-07-10 7:32 UTC (permalink / raw) To: Greg Kroah-Hartman; +Cc: stable, Masami Hiramatsu, systemtap, xe-linux-external On Tue, 10 Jul 2018, Greg Kroah-Hartman wrote: > [Adding the stable list] > > On Mon, Jul 09, 2018 at 11:53:37PM -0700, Victor Kamensky wrote: >> Hi Masami, >> >> I am not sure maybe you are aware about this issue already, but >> I am writing in case if you are not. >> >> After move to 4.4.131 stable very basic SystemTap script that involves >> kprobes stopped working - it crashes kernel. >> >> The SystemTap script like this: >> >> probe kernel.function("meminfo_proc_show") { >> println("meminfo_proc_show called") >> } >> >> which effectively will translate into kprobe entry in meminfo_proc_show >> function, crashes like this. >> >> BUG: unable to handle kernel paging request at ffffffffc063f002 >> IP: [<ffffffffa3c3558b>] resume_execution+0xe5/0x151 >> PGD 24613067 ca >> PUD 24615067 PMD 1523e9067 PTE 3370d061 >> Oops: 0003 [#1] SMP Modules linked in: <snip> >> CPU: 3 PID: 20078 Comm: <snip> Tainted: G O 4.4.131 #1 >> task: ffff880031ead780 ti: ffff880032238000 task.ti: ffff880032238000 >> t /RIP: 0010:[<ffffffffa3c3558b>] [<ffffffffa3c3558b>] resume_execution+0xe5/0x151 >> RSP: 0018:ffff88017fccaec8 EFLAGS: 00010006 >> RAX: ffffffffa3dad154 RBX: ffffffffa3dad152 RCX: 000000003f9c0ff9 >> RDX: ffffffffc063f002 RSI: ffff88017fccaf58 RDI: 0000000000000041 >> RBP: ffff8801549f83c0 R08: 0000000000000001 R09: 0000000000000001 >> R10: ffff88003223bdd8 R11: 0000000000000246 R12: ffff88003223bdd8 >> R13: ffff88017fccaf58 R14: ffffffffc063f000 R15: ffff88017fccee40 >> FS: 00007f9f34520240(0000) GS:ffff88017fcc0000(0000) knlGS:0000000000000000 >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> CR2: ffffffffc063f002 CR3: 0000000152b46000 CR4: 0000000000100670 >> Stack: >> ffff88017fccee40 ffff88017fccaf58 ffff8801549f83c0 0000000000000000 >> ffff880033cde500 0000000000000001 ffffffffa3c35633 ffff88017fccaf58 >> ffff880031ead780 0000000000000000 ffffffffa3c04e9d 0000000000004000 >> Call Trace: >> <#DB> >> [<ffffffffa3c35633>] ? kprobe_debug_handler+0x3c/0xc8 >> [<ffffffffa3c04e9d>] ? do_debug+0x7a/0x182 >> [<ffffffffa4163afe>] ? debug+0x3e/0x70 >> [<ffffffffa3dad14d>] ? meminfo_proc_open+0x1c/0x1c >> [<ffffffffa3d73111>] ? seq_buf_alloc+0x23/0x3c >> <<EOE>> >> [<ffffffffa3d735a8>] ? seq_read+0x1a6/0x40d >> [<ffffffffa3da5700>] ? proc_reg_read+0x47/0x65 >> [<ffffffffa3da56b9>] ? proc_reg_write+0x65/0x65 >> [<ffffffffa3d55ebf>] ? __vfs_read+0x34/0xea >> [<ffffffffa3d564f9>] ? rw_verify_area+0x7a/0xc9 >> [<ffffffffa3d565dc>] ? vfs_read+0x94/0xf9 >> [<ffffffffa3d570a4>] ? SyS_read+0x61/0xb4 >> [<ffffffffa4161b8a>] ? entry_SYSCALL_64_fastpath+0x1e/0x8e >> Code: 7d 70 00 75 3e 49 8b 95 80 00 00 00 49 39 d6 73 2b 48 89 d0 4c 29 f0 >> 48 8d 48 05 48 83 f9 0e 77 1b b9 fb ff ff ff 48 01 d8 29 d1 <c6> 02 e9 01 c8 >> 89 42 01 c7 45 70 01 00 00 00 eb 07 c7 45 70 ff RIP [<ffffffffa3c3558b>] >> resume_execution+0xe5/0x151 >> RSP <ffff88017fccaec8> >> CR2: ffffffffc063f002 >> >> Decoding crash: >> >> (gdb) p &resume_execution >> $1 = (void (*)(struct kprobe *, struct pt_regs *, >> struct kprobe_ctlblk *)) 0xffffffff810354a6 <resume_execution> >> (gdb) b *(0xffffffff810354a6 + 0xe5) >> Breakpoint 1 at 0xffffffff8103558b: file <snip>/kernel-source/arch/x86/kernel/kprobes/core.c, line 126. >> >> corresponds to: >> >> static nokprobe_inline void >> __synthesize_relative_insn(void *from, void *to, u8 op) >> { >> struct __arch_relative_insn { >> u8 op; >> s32 raddr; >> } __packed *insn; >> >> insn = (struct __arch_relative_insn *)from; >> insn->raddr = (s32)((long)(to) - ((long)(from) + 5)); >> insn->op = op; <---------------------------------------- >> } >> >> >> (gdb) p /x &kprobe_debug_handler >> $1 = 0xffffffff810355f7 >> (gdb) b *(0xffffffff810355f7 + 0x3c) >> Breakpoint 1 at 0xffffffff81035633: file <snip>/kernel-source/arch/x86/kernel/kprobes/core.c, line 932. >> >> corresponds to >> >> static void resume_execution(struct kprobe *p, struct pt_regs *regs, >> struct kprobe_ctlblk *kcb) >> { >> <snip> >> if (p->ainsn.boostable == 0) { >> if ((regs->ip > copy_ip) && >> (regs->ip - copy_ip) + 5 < MAX_INSN_SIZE) { >> /* >> * These instructions can be executed directly if it >> * jumps back to correct address. >> */ >> synthesize_reljump((void *)regs->ip, >> (void *)orig_ip + (regs->ip - copy_ip)); <------- >> p->ainsn.boostable = 1; >> } else { >> p->ainsn.boostable = -1; >> } >> } >> >> Further digging points to the following couple commits that 4.4.y stable >> picked up: >> >> 1dd26ec7 kprobes/x86: Fix to set RWX bits correctly before releasing trampoline >> 76bee4 kprobes/x86: Set kprobes pages read-only >> >> Effectively the way I read: the problem is that linux-4.4.y branch is >> missing your following commit >> >> commit 804dec5bda9b4fcdab5f67fe61db4a0498af5221 >> Author: Masami Hiramatsu <mhiramat@kernel.org> >> Date: Wed Mar 29 14:00:25 2017 +0900 >> >> kprobes/x86: Do not modify singlestep buffer while resuming >> >> Do not modify singlestep execution buffer (kprobe.ainsn.insn) >> while resuming from single-stepping, instead, modifies >> the buffer to add a jump back instruction at preparing >> buffer. >> >> Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org> >> Cc: Ananth N Mavinakayanahalli <ananth@linux.vnet.ibm.com> >> >> and as result synthesize_reljump function call still exists from >> resume_execution function but it does not have proper page permission >> manipulation code around it, so making kprobes pages as >> read-only has averse effect: I.e when called from resume_execution >> function it hits read-only page and crashes. >> >> Please advise how to proceed besides obvious backing out above mentioned >> two commits. > > So if you apply the above commit, all works properly for you? That commit seems to be part of some x86 kprobes restructuring, I hesitated to try it. After your email tried to cherry-pick to 4.4.y and it does not go in smoothly at all, quite a bit of conflicts. I may not phrase it correctly: My point was that in 4.4 there are more places where synthesize_reljump function is called, which latter got removed. And this function modifies kprobes page, so lifting up read-only protection needs to be done in more places compared to latter kernels. Whether it can be done correctly and proper thing to do in do_debug/resume_execution function exception processing context I cannot tell. Thanks, Victor > thanks, > > greg k-h > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: in stable 4.4.131 kprobes seems to be broken 2018-07-10 7:32 ` Victor Kamensky @ 2018-07-10 12:26 ` Jack Wang 2018-07-10 13:31 ` Greg Kroah-Hartman 0 siblings, 1 reply; 5+ messages in thread From: Jack Wang @ 2018-07-10 12:26 UTC (permalink / raw) To: kamensky Cc: Greg Kroah-Hartman, stable, mhiramat, systemtap, xe-linux-external, amakhalov > That commit seems to be part of some x86 kprobes restructuring, I > hesitated to try it. After your email tried to cherry-pick to 4.4.y > and it does not go in smoothly at all, quite a bit of conflicts. > > I may not phrase it correctly: My point was that in 4.4 there > are more places where synthesize_reljump function is called, which > latter got removed. And this function modifies kprobes page, so > lifting up read-only protection needs to be done in more places > compared to latter kernels. Whether it can be done correctly and > proper thing to do in do_debug/resume_execution function > exception processing context I cannot tell. > > Thanks, > Victor > > > thanks, > > > > greg k-h > > Hi Victor, Hi Greg, Alexey Makhalov (+cc) ported the fix to kernel 4.4 2 weeks ago. https://www.spinics.net/lists/stable/msg245223.html It applies cleanly to 4.4.139. Regards, Jack ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: in stable 4.4.131 kprobes seems to be broken 2018-07-10 12:26 ` Jack Wang @ 2018-07-10 13:31 ` Greg Kroah-Hartman 2018-07-11 13:22 ` Masami Hiramatsu 0 siblings, 1 reply; 5+ messages in thread From: Greg Kroah-Hartman @ 2018-07-10 13:31 UTC (permalink / raw) To: Jack Wang Cc: kamensky, stable, mhiramat, systemtap, xe-linux-external, amakhalov On Tue, Jul 10, 2018 at 02:26:43PM +0200, Jack Wang wrote: > > That commit seems to be part of some x86 kprobes restructuring, I > > hesitated to try it. After your email tried to cherry-pick to 4.4.y > > and it does not go in smoothly at all, quite a bit of conflicts. > > > > I may not phrase it correctly: My point was that in 4.4 there > > are more places where synthesize_reljump function is called, which > > latter got removed. And this function modifies kprobes page, so > > lifting up read-only protection needs to be done in more places > > compared to latter kernels. Whether it can be done correctly and > > proper thing to do in do_debug/resume_execution function > > exception processing context I cannot tell. > > > > Thanks, > > Victor > > > > > thanks, > > > > > > greg k-h > > > > Hi Victor, Hi Greg, > > Alexey Makhalov (+cc) ported the fix to kernel 4.4 2 weeks ago. > https://www.spinics.net/lists/stable/msg245223.html > > It applies cleanly to 4.4.139. Ah, nice, that's already queued up and will be in the next 4.4.y release in a few days. thanks, greg k-h ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: in stable 4.4.131 kprobes seems to be broken 2018-07-10 13:31 ` Greg Kroah-Hartman @ 2018-07-11 13:22 ` Masami Hiramatsu 0 siblings, 0 replies; 5+ messages in thread From: Masami Hiramatsu @ 2018-07-11 13:22 UTC (permalink / raw) To: Greg Kroah-Hartman Cc: Jack Wang, kamensky, stable, mhiramat, systemtap, xe-linux-external, amakhalov On Tue, 10 Jul 2018 15:31:06 +0200 Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: > On Tue, Jul 10, 2018 at 02:26:43PM +0200, Jack Wang wrote: > > > That commit seems to be part of some x86 kprobes restructuring, I > > > hesitated to try it. After your email tried to cherry-pick to 4.4.y > > > and it does not go in smoothly at all, quite a bit of conflicts. > > > > > > I may not phrase it correctly: My point was that in 4.4 there > > > are more places where synthesize_reljump function is called, which > > > latter got removed. And this function modifies kprobes page, so > > > lifting up read-only protection needs to be done in more places > > > compared to latter kernels. Whether it can be done correctly and > > > proper thing to do in do_debug/resume_execution function > > > exception processing context I cannot tell. > > > > > > Thanks, > > > Victor > > > > > > > thanks, > > > > > > > > greg k-h > > > > > > Hi Victor, Hi Greg, > > > > Alexey Makhalov (+cc) ported the fix to kernel 4.4 2 weeks ago. > > https://www.spinics.net/lists/stable/msg245223.html > > > > It applies cleanly to 4.4.139. > > Ah, nice, that's already queued up and will be in the next 4.4.y release > in a few days. Thanks guys, I think Victor is correct. That bugfix patch should be picked up. Thank you, -- Masami Hiramatsu <mhiramat@kernel.org> ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-07-11 13:26 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <alpine.LRH.2.00.1807092352480.31641@sjc-ads-6991.cisco.com>
2018-07-10 7:05 ` in stable 4.4.131 kprobes seems to be broken Greg Kroah-Hartman
2018-07-10 7:32 ` Victor Kamensky
2018-07-10 12:26 ` Jack Wang
2018-07-10 13:31 ` Greg Kroah-Hartman
2018-07-11 13:22 ` Masami Hiramatsu
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.