From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA28036B92A for ; Fri, 11 Sep 2026 14:10:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.182 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135823; cv=none; b=AGTlvyg6VvZqKmzAe9l2wNe84+d1i+KeFTbwB/1/tR/Pp1YgCKdW3OB/xLdQptJthJmGwuemjFLQSj/v5J5RGzTOwxyV8Bo4IUYm2Un+K6tXzXd7cJXJFJfG9FOAhVISoCveMx+PR92dFD7Rg1wD6ZtCvQ8Gpit7P15yozDW+rw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135823; c=relaxed/simple; bh=hPcqGdFxo+b+ijS7jNB+8GMowmEDvopzKnpNmpphVrY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=YCe78PjlvBdBCLWYj8VM7NZCoZIwzy5vZjwxn4PDz/IwNfEpp8G+Ayy3Heue1SplovyzcuFapLI3kYJYEaQk1ThpFz+BVcRKxjxd8KQcVLjrhPvBpRafNqkwFQYTpF3cLfIe46BZbiKli+svfGfM6IJIV1jS9Vr5yE8aIV4z6EQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=V+OtdwTM; arc=none smtp.client-ip=209.85.160.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="V+OtdwTM" Received: by mail-qt1-f182.google.com with SMTP id d75a77b69052e-5306baf6b53so15000091cf.1 for ; Fri, 11 Sep 2026 07:10:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135821; x=1789740621; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=32pR0m9lJbbl6oefWflQ2FM2YWt9ii7Ub9HEHMaojiA=; b=V+OtdwTMNyS3KHM0tSWV1vt7ohG1FUf1Q3StY2+KLLu4iWNF2+qEFTo9UfRK0qZRCe MuSvri91VbNM6fYtPkKZVA0ARJn+BFjMdt7eSiBU62RuIlQ1j37shBSP4SpX4kOEf9VT llv2p15WUz/Y7A96x/5zZcrrGNG0oUP/2HdSmgkyZaBV3lN+ecuY8C59yhxMXXs59tIc +tNmgnUHigv9VK9TsY5UajA5O11AcjRXHIATasLlcllLm+WX6uxW2klNqqu44l89Y11B tmt1uSMklLnVTID9H0og/lCPTdFJvPw9Bhfp+68asnowa4kgzIvR2cvi6YezQK3zzRLt Po6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135821; x=1789740621; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=32pR0m9lJbbl6oefWflQ2FM2YWt9ii7Ub9HEHMaojiA=; b=S1XVeEVI8rwri0WCB3nW4ZB2wMwmG+c5zkLBRse3xhz5OsBAbQxWbgCpBqtpfyQMIK okOUWW5rT34wZHL/aGQNdD01lVsr9i+Iu34Fe8dFkeNJsBNzr29/vTQDBjAtrk8+/rYy jwjKbjrsecZtHIxzReCV1fyD/Y6YXoErV7Ubpn/YHxpcGNTUewC4XiW51GW3FeBBZxZa v3TDjj6wT0KR9Gj8FHFyPqoYr4YmyPv4x0NBKESkWEtxH10kgx7JWtLvY5cs/QsJ8il0 z3AygDzRE5oQMW8XSATppskXk/rIFHr7xK3NzQDHSCV2rbNF8WlG3MaXoOh2fsoqS8/U j+nA== X-Forwarded-Encrypted: i=1; AKwUvBzHzTsnmZTcU6YTNL+4kbTXtgx9Y5WY5eEKJwY/PojyOGHYv7BrY+rV8LpdLl3AQyim0bGeM8JyJSjM4fq1z2AVZmo=@vger.kernel.org X-Gm-Message-State: AFuF++nsZxOKsxc0Wuh0JFTrBlj1XBnTt70uJ8OklTewbQ/gF/E87Hvl 1yUapGZECTR9YVfL3fao/Zcv1owpNf591N2uvuWaa9ytARDL3blOd9mZP+tsFhtFmZA= X-Gm-Gg: AYBFou14Kf9qVmuKnHoK6dOWg7NAFjsf9CwbxtxHft0J68FAXq5IkD394pTkmUJCiXt BhTO6fTqhAdna6HEBFVtLKlUGN04XEzGFl9aF9AsnRy7N8ZAjiKYK+EOwEor/KzKvhzLwZi4d3t 3B3PPBCBKU2ubPjhJgYA0DLYes5FFJtpLmF7FhSdPuMXv2ChquRwvGD6XDDtLqg1G/5s8WOZC0/ hwkmg4Bs6R15pwPKyBXJx/b4NJpcTZsQ21gsUWoQT4L6GTfsnjavD1KP93tFduobaldc5nkA8ql KOg6K63bWXhcoL9Zni5bBrjIsJdDqiuVml/XnrmJxsf6CyktEWO7oxq+6Ubx4fOu0Ya08X5xD/3 BlY2JHws/OJPXNCobMjfJz01UUN68qLsOw95bmAG28GJFTQMndn2HW4cd8z3J3ARACVCBI3uEwS +uqUTPX/mec/W+UjhzEOcBIBjdH5NfIEqDPjpempErlWZ0QBdthN4uDbwFRsZ38QWxe5HG8NjUX 3y8rAow++aBAjgNxqhh3YcgLReGVyYj7tCDlcu5ddPLkNvXOCvZY6LbK9q7Hnm2Coo= X-Received: by 2002:a05:622a:4114:b0:530:da2a:757 with SMTP id d75a77b69052e-530da2a1198mr24113211cf.33.1789135818762; Fri, 11 Sep 2026 07:10:18 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-530e0aaddbdsm2707391cf.2.2026.09.11.07.10.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:10:17 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:49 +0000 Subject: [PATCH RFC v2 11/15] samples: ftrace: Maintain Tasks RCU trampoline nesting in direct-call trampolines Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-11-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135737; l=10811; i=josef@toxicpanda.com; h=from:subject:message-id; bh=hPcqGdFxo+b+ijS7jNB+8GMowmEDvopzKnpNmpphVrY=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QKhN0qkThHUh5I4+Grit8p3XEg0LAgV/aMtyHMka/HOmaDw0JgyPUhuISBQoEJ4d9x0FD6DquBp sygcZlr+QrA8= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA Follow the register_ftrace_direct() contract in the sample modules: on x86-64 and arm64, have each hand-written trampoline increment current->rcu_tramp_nesting before calling its C handler and decrement it before returning, via a small shared samples/ftrace/ftrace-direct.h. %r11 and x12/w13 are used as scratch; both are caller-saved, non-argument registers and therefore dead on entry to and exit from an fentry trampoline. The header pulls in the generated asm-offsets.h only on those two architectures, since it is not generally safe to include from C (PPC32's TASK_SIZE and arm64's TRAMP_VALIAS clash with the C definitions; the latter is worked around locally with push_macro/pop_macro). Other architectures get empty macros and are unchanged. Assisted-by: LLM Signed-off-by: Josef Bacik --- samples/ftrace/ftrace-direct-modify.c | 9 ++++ samples/ftrace/ftrace-direct-multi-modify.c | 9 ++++ samples/ftrace/ftrace-direct-multi.c | 5 +++ samples/ftrace/ftrace-direct-too.c | 5 +++ samples/ftrace/ftrace-direct.c | 5 +++ samples/ftrace/ftrace-direct.h | 64 +++++++++++++++++++++++++++++ 6 files changed, 97 insertions(+) diff --git a/samples/ftrace/ftrace-direct-modify.c b/samples/ftrace/ftrace-direct-modify.c index 164d9dd6fd92..eb8230fa4242 100644 --- a/samples/ftrace/ftrace-direct-modify.c +++ b/samples/ftrace/ftrace-direct-modify.c @@ -2,6 +2,7 @@ #include #include #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -73,7 +74,9 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " call my_direct_func1\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp1, .-my_tramp1\n" @@ -85,7 +88,9 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " call my_direct_func2\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp2, .-my_tramp2\n" @@ -141,11 +146,13 @@ asm ( " .globl my_tramp1\n" " my_tramp1:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #16\n" " stp x9, x30, [sp]\n" " bl my_direct_func1\n" " ldp x30, x9, [sp]\n" " add sp, sp, #16\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp1, .-my_tramp1\n" @@ -153,11 +160,13 @@ asm ( " .globl my_tramp2\n" " my_tramp2:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #16\n" " stp x9, x30, [sp]\n" " bl my_direct_func2\n" " ldp x30, x9, [sp]\n" " add sp, sp, #16\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp2, .-my_tramp2\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct-multi-modify.c b/samples/ftrace/ftrace-direct-multi-modify.c index b03766c6217b..c8f1062e5d1a 100644 --- a/samples/ftrace/ftrace-direct-multi-modify.c +++ b/samples/ftrace/ftrace-direct-multi-modify.c @@ -2,6 +2,7 @@ #include #include #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -77,10 +78,12 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func1\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp1, .-my_tramp1\n" @@ -92,10 +95,12 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func2\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp2, .-my_tramp2\n" @@ -154,6 +159,7 @@ asm ( " .globl my_tramp1\n" " my_tramp1:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -162,6 +168,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp1, .-my_tramp1\n" @@ -169,6 +176,7 @@ asm ( " .globl my_tramp2\n" " my_tramp2:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -177,6 +185,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp2, .-my_tramp2\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct-multi.c b/samples/ftrace/ftrace-direct-multi.c index 3fe6ddaf0b69..bc6a88dd4ffc 100644 --- a/samples/ftrace/ftrace-direct-multi.c +++ b/samples/ftrace/ftrace-direct-multi.c @@ -3,6 +3,7 @@ #include /* for handle_mm_fault() */ #include +#include "ftrace-direct.h" #include #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include @@ -56,10 +57,12 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " movq 8(%rbp), %rdi\n" " call my_direct_func\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp, .-my_tramp\n" @@ -101,6 +104,7 @@ asm ( " .globl my_tramp\n" " my_tramp:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -109,6 +113,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp, .-my_tramp\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct-too.c b/samples/ftrace/ftrace-direct-too.c index bf2411aa6fd7..247e418644a2 100644 --- a/samples/ftrace/ftrace-direct-too.c +++ b/samples/ftrace/ftrace-direct-too.c @@ -3,6 +3,7 @@ #include /* for handle_mm_fault() */ #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -61,6 +62,7 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " pushq %rsi\n" " pushq %rdx\n" @@ -70,6 +72,7 @@ asm ( " popq %rdx\n" " popq %rsi\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp, .-my_tramp\n" @@ -110,6 +113,7 @@ asm ( " .globl my_tramp\n" " my_tramp:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #48\n" " stp x9, x30, [sp]\n" " stp x0, x1, [sp, #16]\n" @@ -119,6 +123,7 @@ asm ( " ldp x0, x1, [sp, #16]\n" " ldp x2, x3, [sp, #32]\n" " add sp, sp, #48\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp, .-my_tramp\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct.c b/samples/ftrace/ftrace-direct.c index 5368c8c39cbb..9e1964baf28b 100644 --- a/samples/ftrace/ftrace-direct.c +++ b/samples/ftrace/ftrace-direct.c @@ -3,6 +3,7 @@ #include /* for wake_up_process() */ #include +#include "ftrace-direct.h" #if !defined(CONFIG_ARM64) && !defined(CONFIG_PPC32) #include #endif @@ -54,9 +55,11 @@ asm ( " pushq %rbp\n" " movq %rsp, %rbp\n" CALL_DEPTH_ACCOUNT + RCU_TASKS_TRAMP_ENTER " pushq %rdi\n" " call my_direct_func\n" " popq %rdi\n" + RCU_TASKS_TRAMP_EXIT " leave\n" ASM_RET " .size my_tramp, .-my_tramp\n" @@ -97,6 +100,7 @@ asm ( " .globl my_tramp\n" " my_tramp:" " hint 34\n" // bti c + RCU_TASKS_TRAMP_ENTER " sub sp, sp, #32\n" " stp x9, x30, [sp]\n" " str x0, [sp, #16]\n" @@ -104,6 +108,7 @@ asm ( " ldp x30, x9, [sp]\n" " ldr x0, [sp, #16]\n" " add sp, sp, #32\n" + RCU_TASKS_TRAMP_EXIT " ret x9\n" " .size my_tramp, .-my_tramp\n" " .popsection\n" diff --git a/samples/ftrace/ftrace-direct.h b/samples/ftrace/ftrace-direct.h new file mode 100644 index 000000000000..d0313f33f47f --- /dev/null +++ b/samples/ftrace/ftrace-direct.h @@ -0,0 +1,64 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _SAMPLES_FTRACE_DIRECT_H +#define _SAMPLES_FTRACE_DIRECT_H + +#include + +/* + * A direct-call trampoline is entered with no lock, refcount or RCU marker + * held; only Tasks RCU keeps it (and, for a module, its text) alive while a + * task is inside it or preempted in something it called. On architectures + * that select ARCH_HAS_RCU_TASKS_PREEMPT_QS a preemption is a Tasks RCU + * quiescent state unless current->rcu_tramp_nesting is non-zero, so the + * trampoline must raise it before calling out and drop it afterwards, exactly + * like the ftrace and BPF trampolines do. See rcu_tasks_trampoline_enter() + * and register_ftrace_direct(). The instructions before the increment and + * after the decrement are covered by ftrace_direct_mark_module(). + * + * These expand to instruction strings for use inside the samples' asm() + * trampolines. The scratch register is caller-saved and not an argument + * register, so it is dead on entry to and exit from an fentry trampoline. + * + * The generated asm-offsets.h is only pulled in on the architectures that need + * it here: it is not generally safe to include from C (e.g. PPC32's TASK_SIZE + * and arm64's TRAMP_VALIAS clash with the C definitions), which is why the + * samples themselves guard their own include of it. + */ +#if defined(CONFIG_TASKS_RCU) && defined(CONFIG_X86_64) + +#include + +#define RCU_TASKS_TRAMP_ENTER \ + " movq %gs:current_task(%rip), %r11\n" \ + " incl " __stringify(TASK_rcu_tramp_nesting) "(%r11)\n" +#define RCU_TASKS_TRAMP_EXIT \ + " movq %gs:current_task(%rip), %r11\n" \ + " decl " __stringify(TASK_rcu_tramp_nesting) "(%r11)\n" + +#elif defined(CONFIG_TASKS_RCU) && defined(CONFIG_ARM64) + +/* arm64's asm-offsets.h redefines TRAMP_VALIAS from . */ +#pragma push_macro("TRAMP_VALIAS") +#undef TRAMP_VALIAS +#include +#pragma pop_macro("TRAMP_VALIAS") + +#define RCU_TASKS_TRAMP_ENTER \ + " mrs x12, sp_el0\n" \ + " ldr w13, [x12, #" __stringify(TSK_RCU_TRAMP_NESTING) "]\n" \ + " add w13, w13, #1\n" \ + " str w13, [x12, #" __stringify(TSK_RCU_TRAMP_NESTING) "]\n" +#define RCU_TASKS_TRAMP_EXIT \ + " mrs x12, sp_el0\n" \ + " ldr w13, [x12, #" __stringify(TSK_RCU_TRAMP_NESTING) "]\n" \ + " sub w13, w13, #1\n" \ + " str w13, [x12, #" __stringify(TSK_RCU_TRAMP_NESTING) "]\n" + +#else + +#define RCU_TASKS_TRAMP_ENTER +#define RCU_TASKS_TRAMP_EXIT + +#endif + +#endif /* _SAMPLES_FTRACE_DIRECT_H */ -- 2.55.0