From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2108DC88E50 for ; Fri, 11 Sep 2026 14:10:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Cc:To:In-Reply-To:References :Message-Id:Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=dPfI0hp8i5ODnb7Ti+2mvm8Dl2bKUUBUiZ4EzuwrKLQ=; b=uWaAJ24CahfldDaVv73gN1vZlW 4PoAW2LRpcDiAlzInpr8Qmi7MMuGxgNsPCa2ZbpN62E1zNeC14YKjiJUNTFegHeR/qlHi1iPVSPy/ xxftJ14m4jaXV41Ad/mYcKjq/WjWbX4MgyTEU0wufsmgxArjV6OZ5FnNt46o4aM32LE28UHGXq0TP tPhRIOMfGW+7SMhLIWOn6608tzu19JYPZGFAajX5kgn9+TJToEue1kU1HaxQ5RDY4mpDkiJkGbOg9 6cE9KkOXlKiirWf59Ob5IdXECjfc3u297FFlt1BXiq0e+1pZKOCO66oJ6a2dnSsr9gl6IJKgRkHSK BGzq66wQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1x51xF-0000000GrIc-4B4s; Fri, 11 Sep 2026 14:10:14 +0000 Received: from mail-yw1-x112b.google.com ([2607:f8b0:4864:20::112b]) by bombadil.infradead.org with esmtps (Exim 4.99.1 #2 (Red Hat Linux)) id 1x51x0-0000000GrBg-3Ken for linux-arm-kernel@lists.infradead.org; Fri, 11 Sep 2026 14:10:02 +0000 Received: by mail-yw1-x112b.google.com with SMTP id 00721157ae682-885cf67963eso6051227b3.0 for ; Fri, 11 Sep 2026 07:09:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135798; x=1789740598; darn=lists.infradead.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=dPfI0hp8i5ODnb7Ti+2mvm8Dl2bKUUBUiZ4EzuwrKLQ=; b=Mr0GGpSIMSxHKBb7IK8qmICVxNJcMLmeM9DS6r/8i5+ihUu94nsz8nrxk1fnuO/2nb 8kNgsAVNzp1XuupvNMKbXoeltSi+pqcB/lTjyHfnU2wMMXapV4Fj1vXFD+DEjmSvFi1E wGgFGfcvvWjfpTMofE0tGCcIf6PMCitSX+VgrF1qRQGqhDVM+VjJYZIVCsD3tHGOeUc/ 9TrUxhk+SbymKC9gdbh35SfwZ9kuf3yqViO2xHEHISB9ZkzZJEDnmNctpZP3qCZG6GF7 g88icq6iXr2hZQdTKYRj2kIGbLe3QUZSwV/ty8K7DBzpDzYsArTUTx6UiD1oVwYZzbZK G2RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135798; x=1789740598; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=dPfI0hp8i5ODnb7Ti+2mvm8Dl2bKUUBUiZ4EzuwrKLQ=; b=hVNWCV2HJldmKQRez6zJoNGP8kB2G1eifxnqSl9HYNxEVwekGNyTEcBN7oLGj9XIMI +w5U5KaC1XktP4kDdhSHPyHEPjGub2fDvjJIa9733SYm5eKvwZDdl7uZveDUaOYtGwHv rCg43lqCDDghP2Wq1mGeDOfgrfKyigjSw6aoQYHxJiPXPf9b+VybRwXtUF5ysjnGaiXc W4BC0P1rM0TiMUxjVck2Y1jatD68XgQaOqxfdNkfmjgzJAYPQdVsMRD2JwWwUpoxT2YB 2Kk53RX3tr+ErDl6DK5ilPtiMmiwrGOUcUvtH6/fUkPCMrgyXZG50w68LGFT4/s79xpy ndNw== X-Forwarded-Encrypted: i=1; AKwUvBx5qT2OXAnenOfMzUvonqAf/Z+KBMk/qvCnihJ3pECMp3RgdlUhBOgQyF+O1NOFp4HtQeARwr0ZYRGCSWhhkn1H@lists.infradead.org X-Gm-Message-State: AFuF++kvl5rieY8vCzbfifu8Q6LpVNWpSwH317u3EjviRWsPVHMLLWuh i1MtJh5HNdO4lyKALV6N/STtrJaKPkgHzUn5HyiHl2yswtHtdxp5s9WDMxuKtddG+4Q= X-Gm-Gg: AYBFou0BKdb6ENcVrBfHjvsdcasgYnxBYxew+o+4pAZKVijspvtJPhdCNJ8nhbYD4L5 t5b5xnaBBfvmXL2yFgLYKlwjFKCfd7MeW1rln7qOwZyv4jqmCBmQLFgRZv3q4F6PGbCrGMVxkIh EYcqeOodFyAMMj8jdzPR7GjFfN6NEpx4byFmMxrRY4a7AZ3ZYJ1m7nxyvDXrlGAZnFeRLZ7qwSJ ba+v6kORGWe2JIIg/HjnH6/+t/BEEQSsAPEydhCxhGi70+RRamfjBfnROVIs+3rFWT71cpqCoOK fucNTzR2hZA2HhEiSxP8YgtAsWkKQOaQTUri3iF0l4lsRmGEfiyg4EuppKjayGrL8a+b65AfI76 h1eX85cPQrZpmQQ/tANBxeLnsdq2FbAcSzapFRo+tJrxwuBEe4eT1YGDhCKw0u49/ROlxQXBP9g Ghw5/UMu9XKhQd6eGgbX+yxno3hdacIZX8kfZnu+7yrhRVRk6dfrNwhdpNcTeR4zN0fVfnmEfvL 2WYDWYg01bYD/1P23tp7A0qNgsIHenusADC/GLkZfXw+b2nWeF+Wuub X-Received: by 2002:a05:690c:e64c:20b0:873:5c7b:c107 with SMTP id 00721157ae682-884b338f76bmr10673077b3.63.1789135797554; Fri, 11 Sep 2026 07:09:57 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e8047a4dsm255725685a.23.2026.09.11.07.09.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:09:56 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:43 +0000 Subject: [PATCH RFC v2 05/15] ftrace: Mark modules hosting direct-call trampolines for Tasks RCU MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-5-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135736; l=7014; i=josef@toxicpanda.com; h=from:subject:message-id; bh=0t7KiskdL/z6tLq58DM0lyjoNIyrb4NbcoG2+xIJz7Y=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QIYpNGNhK8IIkgKHr1QxZn5VS4Cl+AINmPPfcsvZi4ZGk+7mTHv8oPmVPnKO/4ckp4yjA+b+rY+ zCDlRK/TVmQM= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.9.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260911_070958_865132_E94F8EFF X-CRM114-Status: GOOD ( 23.02 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org An out-of-line direct trampoline registered with register_ftrace_direct() is kept alive only by Tasks RCU while a task executes it or is preempted in something it called; ftrace_shutdown()'s synchronize_rcu_tasks() is what stops rmmod freeing it under such a task. Once preemption becomes a Tasks RCU quiescent state, such a trampoline must hold current->rcu_tramp_nesting across its call-out like the ftrace and BPF trampolines do, so document that in register_ftrace_direct(). That still leaves the few instructions before the increment and after the decrement. For BPF images those are in dynamically allocated text that rcu_tasks_ip_in_trampoline() already treats as protected, but the in-tree samples (and any similar user) place their trampolines in module .text. Add a sticky module::ftrace_direct_tramp flag, set by every register/modify path when the direct address is module text, and have rcu_tasks_ip_in_trampoline() treat a task interrupted anywhere in such a module as a potential reader. Other modules' text is unaffected. Assisted-by: LLM Signed-off-by: Josef Bacik --- include/linux/module.h | 7 +++++++ kernel/rcu/tasks.h | 23 +++++++++++++++++++++-- kernel/trace/ftrace.c | 39 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+), 2 deletions(-) diff --git a/include/linux/module.h b/include/linux/module.h index 96cc98568eea..ea4727f53fab 100644 --- a/include/linux/module.h +++ b/include/linux/module.h @@ -521,6 +521,13 @@ struct module { unsigned int num_ftrace_callsites; unsigned long *ftrace_callsites; #endif +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + /* + * An ftrace direct-call trampoline lives in this module's text; see + * rcu_tasks_ip_in_trampoline(). Sticky once set. + */ + bool ftrace_direct_tramp; +#endif #ifdef CONFIG_KPROBES void *kprobes_text_start; unsigned int kprobes_text_size; diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h index 0e46d8fe4d8e..1b9fe1bfa591 100644 --- a/kernel/rcu/tasks.h +++ b/kernel/rcu/tasks.h @@ -1114,16 +1114,35 @@ bool __weak arch_rcu_tasks_ip_in_trampoline(unsigned long ip) * deliberately does not consult is_ftrace_trampoline() and friends: text * being torn down may already be unregistered there while a task still * stands on it; - * - in core text the architecture flags via arch_rcu_tasks_ip_in_trampoline(). + * - in core text the architecture flags via arch_rcu_tasks_ip_in_trampoline(); + * - in the text of a module that hosts an ftrace direct-call trampoline, + * which covers the instructions before that trampoline's increment and + * after its decrement (see ftrace_direct_mark_module()). * * A false positive only defers the quiescent state to the task's next * context switch. */ bool rcu_tasks_ip_in_trampoline(unsigned long ip) { + bool ret = true; + if (core_kernel_text(ip)) return arch_rcu_tasks_ip_in_trampoline(ip); - return !is_module_text_address(ip); + +#ifdef CONFIG_MODULES + scoped_guard(rcu) { + struct module *mod = __module_text_address(ip); + +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS + if (mod) + ret = READ_ONCE(mod->ftrace_direct_tramp); +#else + if (mod) + ret = false; +#endif + } +#endif + return ret; } NOKPROBE_SYMBOL(rcu_tasks_ip_in_trampoline); diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index 53d5db60bfa5..14f27b887231 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -6076,6 +6076,29 @@ static void reset_direct(struct ftrace_ops *ops, unsigned long addr) ops->trampoline = 0; } +/* + * A direct trampoline may live in module text rather than in dynamically + * allocated text that rcu_tasks_ip_in_trampoline() recognises on its own (see + * samples/ftrace/ftrace-direct*.c). The trampoline itself must hold + * current->rcu_tramp_nesting across its call-out (see register_ftrace_direct()); + * marking the owning module here covers the instructions before that increment + * and after the decrement, where a task interrupted in the module's text must + * not be treated as Tasks-RCU quiescent, so that ftrace_shutdown()'s + * synchronize_rcu_tasks() still keeps the module text from being freed under + * it. + */ +static void ftrace_direct_mark_module(unsigned long addr) +{ +#ifdef CONFIG_MODULES + struct module *mod; + + guard(rcu)(); + mod = __module_text_address(addr); + if (mod) + WRITE_ONCE(mod->ftrace_direct_tramp, true); +#endif +} + /** * register_ftrace_direct - Call a custom trampoline directly * for multiple functions registered in @ops @@ -6090,6 +6113,17 @@ static void reset_direct(struct ftrace_ops *ops, unsigned long addr) * and save the parameters of the function being traced, and restore them * (or inject new ones if needed), before returning. * + * Nothing but Tasks RCU keeps the trampoline at @addr alive while a task is + * executing it or is preempted in something it called. On architectures that + * select ARCH_HAS_RCU_TASKS_PREEMPT_QS a preemption is a Tasks RCU quiescent + * state unless current->rcu_tramp_nesting is non-zero, so the trampoline must + * increment it before calling out and decrement it before returning, as the + * ftrace and BPF trampolines do (see rcu_tasks_trampoline_enter() and + * samples/ftrace/ftrace-direct.h). The few instructions before the increment + * and after the decrement are covered by the irq-exit IP check: automatically + * for trampolines outside kernel and module text (e.g. BPF images), and via + * ftrace_direct_mark_module() for trampolines in module text. + * * Returns: * 0 on success * -EINVAL - The @ops object was already registered with this call or @@ -6169,6 +6203,7 @@ int register_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) ops->flags |= MULTI_FLAGS; ops->trampoline = FTRACE_REGS_ADDR; ops->direct_call = addr; + ftrace_direct_mark_module(addr); err = register_ftrace_function_nolock(ops); if (err) @@ -6237,6 +6272,8 @@ __modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) lockdep_assert_held_once(&direct_mutex); + ftrace_direct_mark_module(addr); + /* Enable the tmp_ops to have the same functions as the direct ops */ ftrace_ops_init(&tmp_ops); tmp_ops.func_hash = ops->func_hash; @@ -6419,6 +6456,7 @@ int update_ftrace_direct_add(struct ftrace_ops *ops, struct ftrace_hash *hash) hlist_for_each_entry(entry, &hash->buckets[i], hlist) { if (__ftrace_lookup_ip(direct_functions, entry->ip)) goto out_unlock; + ftrace_direct_mark_module(entry->direct); } } @@ -6702,6 +6740,7 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, b tmp = __ftrace_lookup_ip(direct_hash, entry->ip); if (!tmp) continue; + ftrace_direct_mark_module(entry->direct); tmp->direct = entry->direct; } } -- 2.55.0