From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 550AEC43612 for ; Tue, 8 Jan 2019 04:44:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1FE92218A1 for ; Tue, 8 Jan 2019 04:44:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1546922662; bh=v5nU2a3xaCgrR9VGbQtPWzhpekPikp+rMRzB33VfNAw=; h=From:To:Cc:Subject:Date:List-ID:From; b=LXKroARs9jyutKn16F8hmvo2akkOTcTJSs1x+4NRmvNB81UUoo/+sg05GOS59axKf pt5F19DJQYkfZ0Cg31jeT+0Xfbs5OXdnpQtn5ImGvOWqF937QaJQodTj0sAJPiplps 6gD2DlwssWam0Erbt/lXRdUUhoDVBA3nTWi2Nnhc= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727560AbfAHEoV (ORCPT ); Mon, 7 Jan 2019 23:44:21 -0500 Received: from mail.kernel.org ([198.145.29.99]:52428 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727374AbfAHEoV (ORCPT ); Mon, 7 Jan 2019 23:44:21 -0500 Received: from localhost.localdomain (NE2965lan1.rev.em-net.ne.jp [210.141.244.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id D2C1B2087F; Tue, 8 Jan 2019 04:44:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1546922660; bh=v5nU2a3xaCgrR9VGbQtPWzhpekPikp+rMRzB33VfNAw=; h=From:To:Cc:Subject:Date:From; b=JFSQIBfsDpFw4G2lZTEV2TxIsT2yF/BaYWAb0Y1mxqAddtHLENw0B/Cn33eM80PzJ lfjlZl2BVeii3pblL1QBQa5FgzOPn8wYKTGEPjD20ojxMUx6j1rl51q24sU8WouCb5 Pbyx6HN0XuD9Pc6It2D1UCyCUaMm6zgnLbt6j/CQ= From: Masami Hiramatsu To: Ingo Molnar Cc: Masami Hiramatsu , peterz@infradead.org, Mathieu Desnoyers , linux-kernel , Andrea Righi , Steven Rostedt , stable@vger.kernel.org Subject: [PATCH v2 0/3] kprobes: Fix kretprobe issues Date: Tue, 8 Jan 2019 13:43:55 +0900 Message-Id: <154692263564.1133.17363562046971295490.stgit@devbox> X-Mailer: git-send-email 2.13.6 User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello, This is v2 series of fixing kretprobe incorrect stacking order patches. In this version, I fixed a lack of kprobes.h including and added new patch for kretprobe trampoline recursion issue. (and add Cc:stable) (1) kprobe incorrct stacking order problem On recent talk with Andrea, I started more precise investigation on the kernel panic with kretprobes on notrace functions, which Francis had been reported last year ( https://lkml.org/lkml/2017/7/14/466 ). See the investigation details in https://lkml.kernel.org/r/154686789378.15479.2886543882215785247.stgit@devbox When we put a kretprobe on ftrace_ops_assist_func() and put another kretprobe on probed-function, below happens -> ->fentry ->ftrace_ops_assist_func() ->int3 ->kprobe_int3_handler() ...->pre_handler_kretprobe() push the return address (*fentry*) of ftrace_ops_assist_func() to top of the kretprobe list and replace it with kretprobe_trampoline. <-kprobe_int3_handler() <-(int3) ->kprobe_ftrace_handler() ...->pre_handler_kretprobe() push the return address (caller) of probed-function to top of the kretprobe list and replace it with kretprobe_trampoline. <-(kprobe_ftrace_handler()) <-(ftrace_ops_assist_func()) [kretprobe_trampoline] ->tampoline_handler() pop the return address (caller) from top of the kretprobe list <-(trampoline_handler()) [run caller with incorrect stack information] <-() !!KERNEL PANIC!! Therefore, this kernel panic happens only when we put 2 k*ret*probes on ftrace_ops_assist_func() and other functions. If we put kprobes, it doesn't cause any issue, since it doesn't change the return address. To fix (or just avoid) this issue, we can introduce a frame pointer verification to skip wrong order entries. And I also would like to blacklist those functions because those are part of ftrace-based kprobe handling routine. (2) kretprobe trampoline recursion problem This was found by Andrea in the previous thread https://lkml.kernel.org/r/20190107183444.GA5966@xps-13 ---- echo "r:event_1 __fdget" >> kprobe_events echo "r:event_2 _raw_spin_lock_irqsave" >> kprobe_events echo 1 > events/kprobes/enable [DEADLOCK] ---- Because kretprobe trampoline_handler uses spinlock for protecting hash table, if we probe the spinlock itself, it causes deadlock. Thank you Andrea and Steve for discovering this root cause!! This bug has been introduced with the asm-coded trampoline code, since previously it used another kprobe for hooking the function return placeholder (which only has a nop) and trampoline handler was called from that kprobe. To fix this bug, I introduced a dummy kprobe and set it in current_kprobe as we did in old days. Thank you, --- Masami Hiramatsu (3): x86/kprobes: Verify stack frame on kretprobe kprobes: Mark ftrace mcount handler functions nokprobe x86/kprobes: Fix to avoid kretprobe recursion arch/x86/kernel/kprobes/core.c | 48 ++++++++++++++++++++++++++++++++++++++-- include/linux/kprobes.h | 1 + kernel/trace/ftrace.c | 6 ++++- 3 files changed, 52 insertions(+), 3 deletions(-) -- Masami Hiramatsu (Linaro)