From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1822B362153 for ; Fri, 11 Sep 2026 14:10:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.174 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135812; cv=none; b=TRwzoufQYdqzbAApSbvnXlEA0A9n0BW8BBemkcK9WtDe7JP5D9Kbz/hHebrWpUcj8uU4vkIhl99r5KdT9WVh0ivi24SW1aV5Ixy4BY5Ydhe/Nu9fTEdobaT3WskGOCLi4ffPZNfiOeVpA7jPj7w5rS+Ryaeo1KEyygQxEB69ISU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789135812; c=relaxed/simple; bh=ZoHASPcgXPkyj8nhWq6Qxe/k1AFcC7iFexyU3IrpvR8=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=nD5t7ACbbhktLqPvNU/1tZCevXZgfDYV2S8UYITPGuCJ2mIVLJJPzsOkcPvwkJryTjCz5AYOlsOEZbZ7S/zhdlfI7A0Vm/6I51WSajzPxLFXvssYvEZWGDmmJOqr1MeXMgV1xagdZY21QDhSLritsQBcHAZaToH9vRDRT4Tb7zA= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com; spf=pass smtp.mailfrom=toxicpanda.com; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b=QW7FHd+Y; arc=none smtp.client-ip=209.85.222.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=toxicpanda.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=toxicpanda.com header.i=@toxicpanda.com header.b="QW7FHd+Y" Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-939f48b80d6so47326185a.1 for ; Fri, 11 Sep 2026 07:10:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda.com; s=google; t=1789135808; x=1789740608; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=7VM45t8rF8CqlHWPsaSwdk7BD4ox5LDNpAeDrj6m5qM=; b=QW7FHd+YQu+cdbwUAwPHWl6S8SJLFgDyTumlkSiN1QY5ehnv0dCylT926iFmEqJpok 32Qx9mxvXEMsm1hYeNmA5d/m7nCtpNiLceDs1HQEYHE9dhdYkjFCYdLZ/3YToNKP3H3n wbpxURtuKoxXDKvhUsCs894yOY2p9P2SeT7jZH8/5JflFVem6A++FLKgvEzGOmXB2BZf V3z9XclxwiteNQC2991FKaQQlmm/RsTxVriIVBe0DTKXlKaO1bERJnX0MLH2/CpVUYJX 6+xcC3O4i4HysTWL0mdzZ/uR8/hnk/SpHCh8R3Sjw3O25/y9FG9K0NTTQjC201ow5QnU KLIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789135808; x=1789740608; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=7VM45t8rF8CqlHWPsaSwdk7BD4ox5LDNpAeDrj6m5qM=; b=ExERiPnmw/gSW4Amh66REm1LZ0zPobp2cu+T33KE6dsSTslPZPYzz+Muj3GmODUVOe Epm8JmSLHdy8tjkJH4fSleF7BxZhR2NXC6shjOykCdq8hgLnoiTp0P8o7T7n60pEC+2d XfjVIjiqqFzVcqnj8D+OEa2SXyk9yVkFZcpji5tXCsm1ldSBnH9nFs04EG2k3vYUAJCZ 6rEZkO4qGYIw28SRkoJDrvCPCTt1pagh427qaRp5PdENnDfufqBBdJl/yhC0h/yy67Dm BGv6Cor8SK8UjvReRkpYXT/555e/o4IAqFuXCR3okuo8wwEUoITad6n0J9Qn2wyjSlk1 30jQ== X-Forwarded-Encrypted: i=1; AKwUvBxAWYuWe1b2fyQRpbS4SpNZ2vA5lMHu2YFWuS0B54E6VtqBwyybn04x2cE6LZ+pzMr6/MjhngJV56LykPy0t540oqI=@vger.kernel.org X-Gm-Message-State: AFuF++kJwt0p4ccbxiKSAS5XYn/Oszgejgnjuqk9GhDmLecnyEPik7wk vaojtErtH6qbwW3qqsGBWPulNWyzhgwYQLzZOPT2azY+Mbk10jBkGderrZM6ZQKTrAQ= X-Gm-Gg: AYBFou3HBFZ2ebSGG8N+Ry9VLpBmiZ5xOmyXmTkhE+5HO1HMpgtIZ7iSi5bQJY2clEZ fES6nUmcVA4pCGLtPNNwzVLBNIeowCb4+kpuqLL4RGR3qHDnHutw4exNymwRdLg5x/X+1GGqLfS 1rgThiruGCBXsWWzux7W2rfNNk/NUVOGj7Gy5ty/Q4eTq1CarPpZoX298w4yjDS0ZxW18ffTR93 GqioGEdZurseGhSh0K2xD2kwgjQbXf4BPhXXd2vmVcQRJGToGii53tUkJjCRTJddYkKRQsf2xAg NnRmgIZjIFpLqpVMjvpdxwRJHrd/GmmLoNTfPPwZAyYiwml8grgJ9+7jtscMlHa4eH3M/noya9G EVFJP2E3x3jthkpc6nvJVB7CNd0rZK1CvqCUw0O7gvJsoqsgbYB1Y9EQQfzg+j7huAmlv/Ei23d gqAaAhZrNebHSSBZ382L/a8LnGvimcl52+AqcFGUJi1lpT4d4WPGLYLCveBsRACUe3/iNLIdc3h Kak4UBFkdXST4Dwk5UoV0aTuUN5olq9AyKN+vq37YCYziic9Yn5XcQ7 X-Received: by 2002:a05:620a:25c7:b0:939:bd4e:b2fe with SMTP id af79cd13be357-939ea2877e9mr555790285a.40.1789135808266; Fri, 11 Sep 2026 07:10:08 -0700 (PDT) Received: from toxicpanda.com (ec2-34-228-114-98.compute-1.amazonaws.com. [34.228.114.98]) by smtp.gmail.com with ESMTPSA id af79cd13be357-939e7f184d9sm254193785a.12.2026.09.11.07.10.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2026 07:10:05 -0700 (PDT) From: Josef Bacik Date: Fri, 11 Sep 2026 14:08:46 +0000 Subject: [PATCH RFC v2 08/15] bpf, x86: Maintain Tasks RCU trampoline nesting in the BPF trampoline Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20260911-b4-rcu-tasks-preempt-qs-v2-8-eaaa61ed2da4@toxicpanda.com> References: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> In-Reply-To: <20260911-b4-rcu-tasks-preempt-qs-v2-0-eaaa61ed2da4@toxicpanda.com> To: "Paul E. McKenney" , Frederic Weisbecker , Neeraj Upadhyay , Joel Fernandes , Boqun Feng , Thomas Gleixner , Peter Zijlstra , Steven Rostedt , Masami Hiramatsu , Mark Rutland , Jiri Olsa , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , x86@kernel.org, Catalin Marinas , Will Deacon , Puranjay Mohan , Xu Kuohai Cc: Andy Lutomirski , Josh Triplett , Uladzislau Rezki , Mathieu Desnoyers , Lai Jiangshan , Zqiang , Juergen Gross , Luis Chamberlain , Ihor Solodrai , linux-kernel@vger.kernel.org, rcu@vger.kernel.org, linux-trace-kernel@vger.kernel.org, bpf@vger.kernel.org, linux-arm-kernel@lists.infradead.org, xen-devel@lists.xenproject.org, Josef Bacik X-Mailer: b4 0.15.2 X-Developer-Signature: v=1; a=openssh-sha256; t=1789135736; l=4464; i=josef@toxicpanda.com; h=from:subject:message-id; bh=ZoHASPcgXPkyj8nhWq6Qxe/k1AFcC7iFexyU3IrpvR8=; b=U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgUBr36M/n0nWN0DNbnxwzIiCZez6MG JiruuNaSCI/zXsAAAAGcGF0YXR0AAAAAAAAAAZzaGE1MTIAAABTAAAAC3NzaC1lZDI1NTE5AAAA QNGMQXlnJ9mLTB5EylkME94aCeeA53rjkqD+SHndTEJHVSfeCPYIUtSGAIP6GARv7tJy5dM1hBK jP1T+H+zt3A0= X-Developer-Key: i=josef@toxicpanda.com; a=openssh; fpr=SHA256:C8kOX2QUJCMqnCX+KEeoqRAjLo9L+ELOSH2NSAJHqGA Emit an increment of current->rcu_tramp_nesting once the trampoline's frame is set up and a decrement before the final register restore, so that a task preempted while running fentry/fexit/fmod_ret/LSM programs or the __bpf_tramp_enter()/__bpf_tramp_exit() glue is not treated as Tasks-RCU quiescent. Drop the count around the call to the original function: that may run arbitrarily long without sleeping and must not pin a Tasks RCU grace period, and the trampoline frame above it is held by im->pcref rather than by Tasks RCU (see bpf_tramp_image_put()). The fmod_ret early-exit branch and the ip_after_call -> ip_epilogue poke both skip the decrement/increment pair around the original call, so the count stays balanced on every path. The sequence is "mov r11, gs:[current_task]; inc/dec dword [r11 + off]"; r11 is scratch at every emission point and (u32)¤t_task is a valid sign-extended %gs-absolute with the current per-CPU layout, the same form the JIT already uses for this_cpu_off. The image is dynamically allocated text, so the instructions outside the bracketed region are covered by the irq-exit IP check. Assisted-by: LLM Signed-off-by: Josef Bacik --- arch/x86/net/bpf_jit_comp.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 2853e87797a7..a375c1b7bd50 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -722,6 +722,31 @@ static void emit_indirect_jump(u8 **pprog, int bpf_reg, u8 *ip) *pprog = prog; } +/* + * Tasks RCU trampoline nesting, see rcu_tasks_trampoline_enter(). + * + * mov r11, QWORD PTR gs:[current_task] + * inc/dec DWORD PTR [r11 + offsetof(struct task_struct, rcu_tramp_nesting)] + * + * r11 (AUX_REG) is scratch in the trampoline at every point this is emitted. + */ +static void emit_rcu_tasks_tramp_nesting(u8 **pprog, bool enter) +{ +#ifdef CONFIG_TASKS_RCU + u8 *prog = *pprog; + + /* mov r11, gs:[abs32] */ + EMIT2(0x65, 0x4C); + EMIT3(0x8B, 0x1C, 0x25); + EMIT((u32)(unsigned long)¤t_task, 4); + /* inc/dec dword ptr [r11 + disp32] */ + EMIT3(0x41, 0xFF, enter ? 0x83 : 0x8B); + EMIT(offsetof(struct task_struct, rcu_tramp_nesting), 4); + + *pprog = prog; +#endif +} + static void emit_return(u8 **pprog, u8 *ip) { u8 *prog = *pprog; @@ -3610,6 +3635,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im /* mov QWORD PTR [rbp - rbx_off], rbx */ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_6, -rbx_off); + /* + * From here until the matching decrement before the final return, a + * preemption of this task is not a Tasks RCU quiescent state. The + * instructions above this point are covered by the irq-exit IP check. + */ + emit_rcu_tasks_tramp_nesting(&prog, true); + func_meta = nr_regs; /* Store number of argument registers of the traced function */ emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta); @@ -3670,6 +3702,13 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im LOAD_TRAMP_TAIL_CALL_CNT_PTR(stack_size); } + /* + * The original function may run for a long time without + * sleeping; do not let it pin a Tasks RCU grace period. The + * trampoline frame above it is held by im->pcref + * (__bpf_tramp_enter()), not by Tasks RCU, across the call. + */ + emit_rcu_tasks_tramp_nesting(&prog, false); if (flags & BPF_TRAMP_F_ORIG_STACK) { emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, 8); EMIT2(0xff, 0xd3); /* call *rbx */ @@ -3680,6 +3719,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im goto cleanup; } } + emit_rcu_tasks_tramp_nesting(&prog, true); /* remember return value in a stack for bpf prog to access */ emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); im->ip_after_call = image + (prog - (u8 *)rw_image); @@ -3741,6 +3781,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im if (save_ret) emit_ldx(&prog, BPF_DW, BPF_REG_0, BPF_REG_FP, -8); + /* Remaining instructions are covered by the irq-exit IP check. */ + emit_rcu_tasks_tramp_nesting(&prog, false); + emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off); EMIT1(0xC9); /* leave */ -- 2.55.0