From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.linuxfoundation.org ([140.211.169.12]:60058 "EHLO mail.linuxfoundation.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750992AbeGJHFr (ORCPT ); Tue, 10 Jul 2018 03:05:47 -0400 Date: Tue, 10 Jul 2018 09:05:44 +0200 From: Greg Kroah-Hartman To: Victor Kamensky , stable@vger.kernel.org Cc: Masami Hiramatsu , systemtap@sourceware.org, xe-linux-external@cisco.com Subject: Re: in stable 4.4.131 kprobes seems to be broken Message-ID: <20180710070543.GA26996@kroah.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: stable-owner@vger.kernel.org List-ID: [Adding the stable list] On Mon, Jul 09, 2018 at 11:53:37PM -0700, Victor Kamensky wrote: > Hi Masami, > > I am not sure maybe you are aware about this issue already, but > I am writing in case if you are not. > > After move to 4.4.131 stable very basic SystemTap script that involves > kprobes stopped working - it crashes kernel. > > The SystemTap script like this: > > probe kernel.function("meminfo_proc_show") { > println("meminfo_proc_show called") > } > > which effectively will translate into kprobe entry in meminfo_proc_show > function, crashes like this. > > BUG: unable to handle kernel paging request at ffffffffc063f002 > IP: [] resume_execution+0xe5/0x151 > PGD 24613067 ca > PUD 24615067 PMD 1523e9067 PTE 3370d061 > Oops: 0003 [#1] SMP Modules linked in: > CPU: 3 PID: 20078 Comm: Tainted: G O 4.4.131 #1 > task: ffff880031ead780 ti: ffff880032238000 task.ti: ffff880032238000 > t /RIP: 0010:[] [] resume_execution+0xe5/0x151 > RSP: 0018:ffff88017fccaec8 EFLAGS: 00010006 > RAX: ffffffffa3dad154 RBX: ffffffffa3dad152 RCX: 000000003f9c0ff9 > RDX: ffffffffc063f002 RSI: ffff88017fccaf58 RDI: 0000000000000041 > RBP: ffff8801549f83c0 R08: 0000000000000001 R09: 0000000000000001 > R10: ffff88003223bdd8 R11: 0000000000000246 R12: ffff88003223bdd8 > R13: ffff88017fccaf58 R14: ffffffffc063f000 R15: ffff88017fccee40 > FS: 00007f9f34520240(0000) GS:ffff88017fcc0000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: ffffffffc063f002 CR3: 0000000152b46000 CR4: 0000000000100670 > Stack: > ffff88017fccee40 ffff88017fccaf58 ffff8801549f83c0 0000000000000000 > ffff880033cde500 0000000000000001 ffffffffa3c35633 ffff88017fccaf58 > ffff880031ead780 0000000000000000 ffffffffa3c04e9d 0000000000004000 > Call Trace: > <#DB> > [] ? kprobe_debug_handler+0x3c/0xc8 > [] ? do_debug+0x7a/0x182 > [] ? debug+0x3e/0x70 > [] ? meminfo_proc_open+0x1c/0x1c > [] ? seq_buf_alloc+0x23/0x3c > <> > [] ? seq_read+0x1a6/0x40d > [] ? proc_reg_read+0x47/0x65 > [] ? proc_reg_write+0x65/0x65 > [] ? __vfs_read+0x34/0xea > [] ? rw_verify_area+0x7a/0xc9 > [] ? vfs_read+0x94/0xf9 > [] ? SyS_read+0x61/0xb4 > [] ? entry_SYSCALL_64_fastpath+0x1e/0x8e > Code: 7d 70 00 75 3e 49 8b 95 80 00 00 00 49 39 d6 73 2b 48 89 d0 4c 29 f0 > 48 8d 48 05 48 83 f9 0e 77 1b b9 fb ff ff ff 48 01 d8 29 d1 02 e9 01 c8 > 89 42 01 c7 45 70 01 00 00 00 eb 07 c7 45 70 ff RIP [] > resume_execution+0xe5/0x151 > RSP > CR2: ffffffffc063f002 > > Decoding crash: > > (gdb) p &resume_execution > $1 = (void (*)(struct kprobe *, struct pt_regs *, > struct kprobe_ctlblk *)) 0xffffffff810354a6 > (gdb) b *(0xffffffff810354a6 + 0xe5) > Breakpoint 1 at 0xffffffff8103558b: file /kernel-source/arch/x86/kernel/kprobes/core.c, line 126. > > corresponds to: > > static nokprobe_inline void > __synthesize_relative_insn(void *from, void *to, u8 op) > { > struct __arch_relative_insn { > u8 op; > s32 raddr; > } __packed *insn; > > insn = (struct __arch_relative_insn *)from; > insn->raddr = (s32)((long)(to) - ((long)(from) + 5)); > insn->op = op; <---------------------------------------- > } > > > (gdb) p /x &kprobe_debug_handler > $1 = 0xffffffff810355f7 > (gdb) b *(0xffffffff810355f7 + 0x3c) > Breakpoint 1 at 0xffffffff81035633: file /kernel-source/arch/x86/kernel/kprobes/core.c, line 932. > > corresponds to > > static void resume_execution(struct kprobe *p, struct pt_regs *regs, > struct kprobe_ctlblk *kcb) > { > > if (p->ainsn.boostable == 0) { > if ((regs->ip > copy_ip) && > (regs->ip - copy_ip) + 5 < MAX_INSN_SIZE) { > /* > * These instructions can be executed directly if it > * jumps back to correct address. > */ > synthesize_reljump((void *)regs->ip, > (void *)orig_ip + (regs->ip - copy_ip)); <------- > p->ainsn.boostable = 1; > } else { > p->ainsn.boostable = -1; > } > } > > Further digging points to the following couple commits that 4.4.y stable > picked up: > > 1dd26ec7 kprobes/x86: Fix to set RWX bits correctly before releasing trampoline > 76bee4 kprobes/x86: Set kprobes pages read-only > > Effectively the way I read: the problem is that linux-4.4.y branch is > missing your following commit > > commit 804dec5bda9b4fcdab5f67fe61db4a0498af5221 > Author: Masami Hiramatsu > Date: Wed Mar 29 14:00:25 2017 +0900 > > kprobes/x86: Do not modify singlestep buffer while resuming > > Do not modify singlestep execution buffer (kprobe.ainsn.insn) > while resuming from single-stepping, instead, modifies > the buffer to add a jump back instruction at preparing > buffer. > > Signed-off-by: Masami Hiramatsu > Cc: Ananth N Mavinakayanahalli > > and as result synthesize_reljump function call still exists from > resume_execution function but it does not have proper page permission > manipulation code around it, so making kprobes pages as > read-only has averse effect: I.e when called from resume_execution > function it hits read-only page and crashes. > > Please advise how to proceed besides obvious backing out above mentioned > two commits. So if you apply the above commit, all works properly for you? thanks, greg k-h