From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9BDD933343C for ; Fri, 11 Sep 2026 14:10:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.178 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135817; cv=none; b=k2HBREtxBM0/vrYnOkFDD5qP4KVC+46ZCov9AFQQ30cupEuy6igd0u8jwUDkgwXuVkLvTz/c9KjS4f0WY394I8bqurl1KgaF24jcdAf72tgfeVec5n/SFgto0xqwXemleeJapDtGaE+cGeKCjtaBPgL4j/rLXFWjTu02jTAwy34= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135817; c=relaxed/simple; bh=xqTPn5S4ER1XAZhtIYfJFNG5V6jdP8hXJAwXIfHAJeM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=OT8ZcpG/nZumm6YfdZnyjYvVYRI0ic6mwhIfC+Ba+q/4EWprf2APtZHb2vZiYdMzEBTQ3au2u9IlIBEB57OaiLlo87arYShVVArn6xPlzqNiXNPU6sVxEo+aJtu/XCrlM/MV9skyxe1KjatJObb7MlyIZkWmX1XEIbHX+v2CbP4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=RL7TN91v; arc=none smtp.client-ip=209.85.222.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="RL7TN91v" Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-939ebaf1956so82022485a.2 for ; Fri, 11 Sep 2026 07:10:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135813; x=1789740613; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=q0Z6kl89L/mcJKSDFUzFWtYRNb/tGGuag8yv3GYWCr4=; b=RL7TN91v33ejtXycFyxg3fUnBjQf8abI+hin2UNGZI2wBeWwf49htfb9Aj8vUw2Muq k9u8JLOpfzYnL+lg0qTqWffW0ltVG7wjneIpkZseOnt26NGyUT2jtGsu3wJxAwhF9TfT Q7tUWoFJCzyfPptfXsbE6n8fgKUbf9pfjLsavhbujWDYGXt48pF1kZryFdJgRiRIX4ge pIeqsVTdFT92F/zIFJZMB18PnPCOKsOH1fvdO3mIZVGLVrTrMhmrC6bHOTAOhAsTrZJJ W95Cqb6b3057kaKanyW0Z8xbp2flwvBJ4Aa1e2hOY5E02BLr7W9PPTVAWk0kBW/CyAoq rIDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135813; x=1789740613; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=q0Z6kl89L/mcJKSDFUzFWtYRNb/tGGuag8yv3GYWCr4=; b=LFYZUXehoUB7+DDg7yzCnYQYGerqF+qQsxIYBDKT0zuoMU8/1SZ1a6nICWx4p9iKPf EdlCPIbxmNP1R+mJjrjoyofNH6kwdMGOGW5M+K8VLXs/CeAaUKtRpinfzBvvRtdDa+gp VY/hO99eJmzeWWIH5tUxUTlOBvx9RcRdsmBFIHCYzxQdsLq6+M8E3Cg2zm8etDS+l94T wjv4Y2ZaId96aXTpJ+qhJXZ+fduuo4wYJ74FVW11E3UpmdISqw+MmrYvs+/mMANNWvZ4 al965g9cW1fHXhKPe0/5WjS5KFpPu1Y96axBIcMbf98UowgjsazMz7dyMMo4VogUdrn5 eUuA== X-Forwarded-Encrypted: i=1; AKwUvBweTQ1RvMpPo0uXpzGoKxxhh2C/suu80oeo7rxaPVHuRu30X2yyGITM9QEn4smfdEDnk2XwQsJ/BQLIPmRNXeIgdPU=@vger.kernel.org X-Gm-Message-State: AFuF++mg0ca5mkd1/CwL3vqOdkQFk08k007QyPHEXcI0ZdOn1GQYHm6M FUOgf0eWQcCW2NN+uH9p3NvT6h4eIj7S5ygCzndaMc9pOPSNTlDg51VBfHVDGy7gr2c= X-Gm-Gg: AYBFou1Nr3CTBRZ+IEow+bCHRmN0R5n0iivMHVa8sVRLgtjqbTKtFatnXVGzMT5hvtB NtuHCcFnPvhVCsV4bhXagjFtoJaHFxuhxTubqABlb7GkmFSss6FEeZYgh7EmUTp0y1IR6WOXwwN 6dNn0/TCKJzIQ0IPhBTbtHyoMcB7qdXm9j6DprSIwMaO8fRzTmbo4tc4i+hFXeZ9JzUXyCUeQEC xv3ADB5x5kauGaL37rz4yVGQfVGuAIKXedRU4ZKcHNx+Nm7/awzaVqIXFaHgJGsqmfFmVieOjvW 1TkKxY7+UvaAYtamSSFURsip75/eqHk9C3DJPGTCGECbBr3TDUn2fNMVNb3+lS+d/EFlVT/+YU5 c5T8QFwX13p8571Xq8MeL7aXOAD2gydKB/gbT1j0lgfpeQG97wbJSniYMHE/sOvFTkXqVqLF5fA toP8sXlGVANTxl5jjn70INbH8OSFTZ5iGKLx/+uPDIuiymPdb6QLBbK8qyQg3JUMMK3CCVnmSL/ kBPI6pSdruVKj+WzBeZfU0c3O4kxw0004lGAulf6qhi9EHmXNgXu3WD X-Received: by 2002:a05:620a:6287:b0:939:6dea:374b with SMTP id af79cd13be357-939ea1af385mr553869585a.47.1789135812492; Fri, 11 Sep 2026 07:10:12 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e80bbab7sm247620785a.34.2026.09.11.07.10.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:10:10 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:47 +0000 Subject: [PATCH RFC v2 09/15] arm64: ftrace: Maintain Tasks RCU trampoline nesting in ftrace_caller Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-9-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135736; l=5217; i=josef@toxicpanda.com; h=from:subject:message-id; bh=xqTPn5S4ER1XAZhtIYfJFNG5V6jdP8hXJAwXIfHAJeM=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QDdDEdIwM05h7b7JvpgB0Ncpvko1HStiAd9VzLg7h8HjWsQnjthe7VWx1bLRiDUbfVMTbRgUiKe YsgeGrk28wQI= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA Bracket the call out to the ftrace_ops callback in ftrace_caller with an increment/decrement of current->rcu_tramp_nesting, using x12/w13 which are scratch there. The read-modify-write is not atomic, but only current modifies the count and every interrupting user is balanced, so nothing is lost. ftrace_caller itself, including the early CALL_OPS direct path and the late direct tail that carry a BPF trampoline address in x17 with the count at zero, is static kernel text: add an ftrace_static_tramp_end marker after ftrace_stub_direct_tramp and provide arch_rcu_tasks_ip_in_trampoline() covering [ftrace_caller, ftrace_static_tramp_end) so the irq-exit check treats a task interrupted anywhere in it as inside a trampoline. The hook is built only under CONFIG_RCU_TASKS_PREEMPT_QS, which arm64 does not select until a later patch. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/arm64/kernel/asm-offsets.c | 3 +++ arch/arm64/kernel/entry-ftrace.S | 35 +++++++++++++++++++++++++++++++++++ arch/arm64/kernel/ftrace.c | 16 ++++++++++++++++ 3 files changed, 54 insertions(+) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 9c853ed3ceab..f6655a284f18 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -39,6 +39,9 @@ int main(void) DEFINE(TSK_STACK, offsetof(struct task_struct, stack)); #ifdef CONFIG_STACKPROTECTOR DEFINE(TSK_STACK_CANARY, offsetof(struct task_struct, stack_canary)); +#endif +#ifdef CONFIG_TASKS_RCU + DEFINE(TSK_RCU_TRAMP_NESTING, offsetof(struct task_struct, rcu_tramp_nesting)); #endif BLANK(); DEFINE(THREAD_CPU_CONTEXT, offsetof(struct task_struct, thread.cpu_context)); diff --git a/arch/arm64/kernel/entry-ftrace.S b/arch/arm64/kernel/entry-ftrace.S index 025140caafe7..46a102e7199a 100644 --- a/arch/arm64/kernel/entry-ftrace.S +++ b/arch/arm64/kernel/entry-ftrace.S @@ -14,6 +14,33 @@ #include #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS +/* + * Tasks RCU trampoline nesting, see rcu_tasks_trampoline_enter(). The whole + * of ftrace_caller is treated as trampoline text by the irq-exit IP check (see + * arch_rcu_tasks_ip_in_trampoline()), so these only need to bracket the call + * out to ops->func; everything before the increment and after the decrement, + * including the direct-call tails that carry a BPF trampoline address in x17, + * is covered by that. The count is only modified by current and every nested + * user (interrupts) is balanced, so a plain ldr/add/str is sufficient. + */ + .macro rcu_tasks_tramp_enter, tsk:req, tmp:req +#ifdef CONFIG_TASKS_RCU + mrs \tsk, sp_el0 + ldr \tmp, [\tsk, #TSK_RCU_TRAMP_NESTING] + add \tmp, \tmp, #1 + str \tmp, [\tsk, #TSK_RCU_TRAMP_NESTING] +#endif + .endm + + .macro rcu_tasks_tramp_exit, tsk:req, tmp:req +#ifdef CONFIG_TASKS_RCU + mrs \tsk, sp_el0 + ldr \tmp, [\tsk, #TSK_RCU_TRAMP_NESTING] + sub \tmp, \tmp, #1 + str \tmp, [\tsk, #TSK_RCU_TRAMP_NESTING] +#endif + .endm + /* * Due to -fpatchable-function-entry=2, the compiler has placed two NOPs before * the regular function prologue. For an enabled callsite, ftrace_init_nop() and @@ -94,6 +121,8 @@ SYM_CODE_START(ftrace_caller) stp x29, x30, [sp, #FREGS_SIZE] add x29, sp, #FREGS_SIZE + rcu_tasks_tramp_enter x12, w13 + /* Prepare arguments for the tracer func */ sub x0, x30, #AARCH64_INSN_SIZE // ip (callsite's BL insn) mov x1, x9 // parent_ip (callsite's LR) @@ -111,6 +140,8 @@ SYM_INNER_LABEL(ftrace_call, SYM_L_GLOBAL) bl ftrace_stub // func(ip, parent_ip, op, regs) #endif + rcu_tasks_tramp_exit x12, w13 + /* * At the callsite x0-x8 and x19-x30 were live. Any C code will have preserved * x19-x29 per the AAPCS, and we created frame records upon entry, so we need @@ -178,6 +209,10 @@ SYM_CODE_START(ftrace_stub_direct_tramp) SYM_CODE_END(ftrace_stub_direct_tramp) #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ +/* End of [ftrace_caller, ...) for arch_rcu_tasks_ip_in_trampoline(). */ +SYM_CODE_START(ftrace_static_tramp_end) +SYM_CODE_END(ftrace_static_tramp_end) + #else /* CONFIG_DYNAMIC_FTRACE_WITH_ARGS */ /* diff --git a/arch/arm64/kernel/ftrace.c b/arch/arm64/kernel/ftrace.c index e1a3c0b3a051..1b7ac2afed0d 100644 --- a/arch/arm64/kernel/ftrace.c +++ b/arch/arm64/kernel/ftrace.c @@ -17,6 +17,22 @@ #include #include +#ifdef CONFIG_RCU_TASKS_PREEMPT_QS +extern void ftrace_static_tramp_end(void); + +/* + * See rcu_tasks_ip_in_trampoline(). ftrace_caller and ftrace_stub_direct_tramp + * are core kernel text but must be treated as trampolines: a task preempted in + * them may be carrying an ops pointer (x11) or a direct-call BPF trampoline + * address (x17) whose lifetime is guarded only by Tasks RCU. + */ +bool arch_rcu_tasks_ip_in_trampoline(unsigned long ip) +{ + return ip >= (unsigned long)ftrace_caller && + ip < (unsigned long)ftrace_static_tramp_end; +} +#endif + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS struct fregs_offset { const char *name; -- 2.55.0