From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89221C52D7C for ; Tue, 13 Aug 2024 04:30:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 16FC16B00A0; Tue, 13 Aug 2024 00:30:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1200B6B00A1; Tue, 13 Aug 2024 00:30:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F2D326B00A2; Tue, 13 Aug 2024 00:29:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D16236B00A0 for ; Tue, 13 Aug 2024 00:29:59 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8619FA059A for ; Tue, 13 Aug 2024 04:29:59 +0000 (UTC) X-FDA: 82445944518.07.B41B66A Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf25.hostedemail.com (Postfix) with ESMTP id DCA23A0014 for ; Tue, 13 Aug 2024 04:29:57 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=lAoZgFKT; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf25.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723523341; a=rsa-sha256; cv=none; b=F75n+SQ1715MmMtCQh7Ty1JnCtaU5LBkFkIsSSoYlQkv0L6tpCuH6L23nDaf51Lrzfh5/D BJj6qI1wK5Tgz8sczEWg+0TRZjDeoFD1LJkEFUxAUx8aphvpRc98C976+P7yO8AndIop2Z Nb53qXDy7GmnwGwIUEX4/NGExCvEOW0= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=lAoZgFKT; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf25.hostedemail.com: domain of andrii@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=andrii@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723523341; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LlaKZRu4PzVHPdtAjWlZSFDNQe6FVy7RczckRzr/Mps=; b=PCVlyrsuuwCEJ2Z04npjZ62iYU2NahBuDd2xmvbLOHaluhVCmbHMiJsc+KPxuMBlgtHu46 z8dK9tH7YNyQj8A/EkbLEAriugZUQA74qfIPjFqcoi/CwS7re5uTFzxaZV+udgox7hswIU +P5tT8sRif5vGAiEqEyczkQBstjiblg= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 1919461551; Tue, 13 Aug 2024 04:29:57 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id AE5C7C4AF0E; Tue, 13 Aug 2024 04:29:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1723523396; bh=OvL70QsrDczQ8iqrV8LJBOmNylKoE4hwbFie5j0/JPQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lAoZgFKTFKJp/F2BB4J2mNKQlWMd/tPk5d1k6V28UkayC+B9VzjbEtAD7etxoMPox KbZZ8pfGDdndUki76uP9d1Kumyi0kPPI6sR/mMdVZYktNRy+zZF2VF4FgLznIFMRDF Wnc8+PRlNA7CWER/Q74/5uVayvtjMRD+3OsBCDbKWF0OmUwmbbCxQaVEn1ciPrBF0a yiCvHr2Ob6iZoAIhxLuxutP9ciBwGSQhbNGS+RQr7bpqmB3oYXkxFk1emantskGrWr tE2e4ayy7/6Y+TPrt6ChbeoQK4OEY08fTqQfq2h7dQ7/HpRo+auNBL31UQ1bu5IBxW QkhqNdR7Z5qiA== From: Andrii Nakryiko To: linux-trace-kernel@vger.kernel.org, peterz@infradead.org, oleg@redhat.com Cc: rostedt@goodmis.org, mhiramat@kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, jolsa@kernel.org, paulmck@kernel.org, willy@infradead.org, surenb@google.com, akpm@linux-foundation.org, linux-mm@kvack.org, Andrii Nakryiko Subject: [PATCH v3 05/13] perf/uprobe: split uprobe_unregister() Date: Mon, 12 Aug 2024 21:29:09 -0700 Message-ID: <20240813042917.506057-6-andrii@kernel.org> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20240813042917.506057-1-andrii@kernel.org> References: <20240813042917.506057-1-andrii@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: DCA23A0014 X-Stat-Signature: gkgbjc43iimcoimo7a4yihg44qssfzp1 X-Rspam-User: X-HE-Tag: 1723523397-589974 X-HE-Meta: U2FsdGVkX19Qd9W4idb8pr0vVaGFDEHUJUZN0qBM5oqKHQ/qx7LtEV0hTQl9CfcDhJzYpN63NTXi1xX82gHeaqj/YQyHnREXPm+TaBhNI22AJ36GC5VZrz1UbqjZdcCj2+rFZdY2Ewu+1/pkLjs5tMC95bZ5pn9+5ToxJq8LpgTcnKJ/AYJpzB5ErLDaaQf0FDh7gVfxoZNdPNuxVn5WcI8lIzEojssgZ5DntbV3NzNV19CCQ/gR8D675ndt0j8QBe4aDJbdozCKM26c7W/vhDC36nIl3V/+gqQ4YKSuRSKYGBKl2Nah/TSjf5kWeA7CDp1SGaR1F7gmIxQB+QlLITQ4chzyCWEZcMojGmlIHud8v1UlFv+mXekIOBzZ1JRDwSsVqdjHFhIqj43ahbvABQe0ME6Bh3sQJR41f+qvoqkz/tmDtC8mx+rmEY/lYz09i3prHl+ZR5UJ4CuY6uCt4XzX6+yx9INP+oE0KHk0PCUSbSEjvyMDt/pCNgrK/1ZbH3URhQ08P4bexCmcpipFtgNIMSJHc3hcraHiNHKmsOcnl61pNUZFw2aTTNtFH4P0dlusij8vAXt+sXEHfiNw2v7NB3vEHHlQfxXxsjFgtPIoNlNH9GRkH5gk0LAXwKbj1VOCClvAHHyFB7AZ168668Z6MdvnzvVGBRLEocvya/6Cv3GeRPlCxG8zR31WAiZz6/h+zRxKqe7CNmFKLRs2qHJmuX6oUyxCy3GX1470WJ/8tURqhZYbiBH9yL5kfVH6QYNZ7izdwuXNKWtLHMnnlFNJNn+Ed1ppb5d+DXmmy6EcUEiMcufcbzRz4dxYi4hhyRS3Qquhq3+R44zJaltnU0tqxzGz7V6xVcWi4U+pYmn0yAb3sMI/qiP1bVyOQrqx9Yo+xnXAIEAyHCLaxzIqUBRQLaDg/pVqrOls3Q8ZL+ioTWjukh8EXVUTNZw0wuDWriGFtn/YE2WPeWGgYWy uoerRJ6O 483u4nZaZrulsP7Coey2DEk7DW0T1t0DLBqW8E852A0vONKYtG+mYteZwpaITHCwEl52+Zeb1M2rwbUufnkNNPdM2p1LpCzK0DK8Q4fKqKdF6Fkbe+dCsQv9J1aFhuw1ZRPdHD0u3qQatbV2LzIg2/5tZxGDGnWPC3s50qjFLK2MmKBE4rivMJB3zVtg95pKy7Ol8MlptQTPkiDxonPSW+fa2SSXJ+YwXTSaXiK3bS9goLW7fWmL4R+gy4L+HQ/U6mR7BZQYoE6MEmv9eq4Ef8R06po1sdMe+H+qBDlKqs+Jg+PGidXbrwlPrUTFgxRMArRw/dNdQ7bY2xCbYW4rFu70L9+ZgcbOFYlzC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Peter Zijlstra With uprobe_unregister() having grown a synchronize_srcu(), it becomes fairly slow to call. Esp. since both users of this API call it in a loop. Peel off the sync_srcu() and do it once, after the loop. We also need to add uprobe_unregister_sync() into uprobe_register()'s error handling path, as we need to be careful about returning to the caller before we have a guarantee that partially attached consumer won't be called anymore. This is an unlikely slow path and this should be totally fine to be slow in the case of a failed attach. Signed-off-by: Peter Zijlstra (Intel) Co-developed-by: Andrii Nakryiko Signed-off-by: Andrii Nakryiko --- include/linux/uprobes.h | 8 +++++-- kernel/events/uprobes.c | 21 +++++++++++++------ kernel/trace/bpf_trace.c | 5 ++++- kernel/trace/trace_uprobe.c | 6 +++++- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 3 ++- 5 files changed, 32 insertions(+), 11 deletions(-) diff --git a/include/linux/uprobes.h b/include/linux/uprobes.h index 29c935b0d504..e41cdae5597b 100644 --- a/include/linux/uprobes.h +++ b/include/linux/uprobes.h @@ -108,7 +108,8 @@ extern unsigned long uprobe_get_trap_addr(struct pt_regs *regs); extern int uprobe_write_opcode(struct arch_uprobe *auprobe, struct mm_struct *mm, unsigned long vaddr, uprobe_opcode_t); extern struct uprobe *uprobe_register(struct inode *inode, loff_t offset, loff_t ref_ctr_offset, struct uprobe_consumer *uc); extern int uprobe_apply(struct uprobe *uprobe, struct uprobe_consumer *uc, bool); -extern void uprobe_unregister(struct uprobe *uprobe, struct uprobe_consumer *uc); +extern void uprobe_unregister_nosync(struct uprobe *uprobe, struct uprobe_consumer *uc); +extern void uprobe_unregister_sync(void); extern int uprobe_mmap(struct vm_area_struct *vma); extern void uprobe_munmap(struct vm_area_struct *vma, unsigned long start, unsigned long end); extern void uprobe_start_dup_mmap(void); @@ -157,7 +158,10 @@ uprobe_apply(struct uprobe* uprobe, struct uprobe_consumer *uc, bool add) return -ENOSYS; } static inline void -uprobe_unregister(struct uprobe *uprobe, struct uprobe_consumer *uc) +uprobe_unregister_nosync(struct uprobe *uprobe, struct uprobe_consumer *uc) +{ +} +static inline void uprobe_unregister_sync(void) { } static inline int uprobe_mmap(struct vm_area_struct *vma) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 7de1aaf50394..0b6d4c0a0088 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -1094,11 +1094,11 @@ register_for_each_vma(struct uprobe *uprobe, struct uprobe_consumer *new) } /** - * uprobe_unregister - unregister an already registered probe. + * uprobe_unregister_nosync - unregister an already registered probe. * @uprobe: uprobe to remove * @uc: identify which probe if multiple probes are colocated. */ -void uprobe_unregister(struct uprobe *uprobe, struct uprobe_consumer *uc) +void uprobe_unregister_nosync(struct uprobe *uprobe, struct uprobe_consumer *uc) { int err; @@ -1112,12 +1112,15 @@ void uprobe_unregister(struct uprobe *uprobe, struct uprobe_consumer *uc) /* TODO : cant unregister? schedule a worker thread */ if (unlikely(err)) { uprobe_warn(current, "unregister, leaking uprobe"); - goto out_sync; + return; } put_uprobe(uprobe); +} +EXPORT_SYMBOL_GPL(uprobe_unregister_nosync); -out_sync: +void uprobe_unregister_sync(void) +{ /* * Now that handler_chain() and handle_uretprobe_chain() iterate over * uprobe->consumers list under RCU protection without holding @@ -1129,7 +1132,7 @@ void uprobe_unregister(struct uprobe *uprobe, struct uprobe_consumer *uc) */ synchronize_srcu(&uprobes_srcu); } -EXPORT_SYMBOL_GPL(uprobe_unregister); +EXPORT_SYMBOL_GPL(uprobe_unregister_sync); /** * uprobe_register - register a probe @@ -1187,7 +1190,13 @@ struct uprobe *uprobe_register(struct inode *inode, up_write(&uprobe->register_rwsem); if (ret) { - uprobe_unregister(uprobe, uc); + uprobe_unregister_nosync(uprobe, uc); + /* + * Registration might have partially succeeded, so we can have + * this consumer being called right at this time. We need to + * sync here. It's ok, it's unlikely slow path. + */ + uprobe_unregister_sync(); return ERR_PTR(ret); } diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c index 73c570b5988b..6b632710c98e 100644 --- a/kernel/trace/bpf_trace.c +++ b/kernel/trace/bpf_trace.c @@ -3184,7 +3184,10 @@ static void bpf_uprobe_unregister(struct bpf_uprobe *uprobes, u32 cnt) u32 i; for (i = 0; i < cnt; i++) - uprobe_unregister(uprobes[i].uprobe, &uprobes[i].consumer); + uprobe_unregister_nosync(uprobes[i].uprobe, &uprobes[i].consumer); + + if (cnt) + uprobe_unregister_sync(); } static void bpf_uprobe_multi_link_release(struct bpf_link *link) diff --git a/kernel/trace/trace_uprobe.c b/kernel/trace/trace_uprobe.c index 7eb79e0a5352..f7443e996b1b 100644 --- a/kernel/trace/trace_uprobe.c +++ b/kernel/trace/trace_uprobe.c @@ -1097,6 +1097,7 @@ static int trace_uprobe_enable(struct trace_uprobe *tu, filter_func_t filter) static void __probe_event_disable(struct trace_probe *tp) { struct trace_uprobe *tu; + bool sync = false; tu = container_of(tp, struct trace_uprobe, tp); WARN_ON(!uprobe_filter_is_empty(tu->tp.event->filter)); @@ -1105,9 +1106,12 @@ static void __probe_event_disable(struct trace_probe *tp) if (!tu->uprobe) continue; - uprobe_unregister(tu->uprobe, &tu->consumer); + uprobe_unregister_nosync(tu->uprobe, &tu->consumer); + sync = true; tu->uprobe = NULL; } + if (sync) + uprobe_unregister_sync(); } static int probe_event_enable(struct trace_event_call *call, diff --git a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c index 3c0515a27842..1fc16657cf42 100644 --- a/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +++ b/tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c @@ -475,7 +475,8 @@ static void testmod_unregister_uprobe(void) mutex_lock(&testmod_uprobe_mutex); if (uprobe.uprobe) { - uprobe_unregister(uprobe.uprobe, &uprobe.consumer); + uprobe_unregister_nosync(uprobe.uprobe, &uprobe.consumer); + uprobe_unregister_sync(); path_put(&uprobe.path); uprobe.uprobe = NULL; } -- 2.43.5