From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0E70A158531; Fri, 24 Jan 2025 16:16:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737735420; cv=none; b=NfyRyDiQwftFONpo+HeOCk9PKjPIFCdnb9Yk6wv7wNdPL4s7MyBWlg+TZHmdXKTgiYBTqZTCKCQ9sOG15tgBQIwu69akXIGSia1suXSJiOLsVLY33O9Fu7iEl2BcIdSatp/nvu2BewJRqRO8jhCFw5VSIolsYxJkn8N/nmAss00= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737735420; c=relaxed/simple; bh=G7A7X+TNPuzSM3j7u4W80znkfn4JRMQ0X24L5b8aaos=; h=Date:From:To:Cc:Subject:Message-Id:In-Reply-To:References: Mime-Version:Content-Type; b=GV/u9ZM2goSFIVaWeUWym04vEitAYLMNFKPv7APCXfHHqtpg3sWO8zuEXwg6Y5YwzLqseTB5X0CQvyPhq+oO16knJhLY/fdHUdCRwtilf7ZLYst+yrGtUnn7yY/WbFS5RCUfdb9gtGEADQhhEHarHEAvTUckce7B4QE/bSGEdpw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=J9Hbj24A; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="J9Hbj24A" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E1AE6C4CED2; Fri, 24 Jan 2025 16:16:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1737735419; bh=G7A7X+TNPuzSM3j7u4W80znkfn4JRMQ0X24L5b8aaos=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=J9Hbj24AMWvi4ArN9bfCG4bJxVzfSCX6+7HADJX3aqyIgKvRtqogZ0mKnJ7zwOZX1 D0zbwkblzHD/NKGZtJj85iZh0uQUdv+b6uRhIaCUEO6jOZOi4M9THiXj+eaERObsj3 7qob8m1u+2yTBxTrRxfeAz4373DYXh4fXNHhpB8HEN9vZ/fn/qruDIqe7d2G7AkJiX blAYpOgvdp80v+1iG/WXXzERgbWLtgd96rXsCcE4VCUHRi5jhlFXjf1kw+iU1EocmU NyWUYpUvaFolQFWWnAPKKT1F0glY47MJVPd0YpPosyJnOzCOY5hJeSXrwj1Ftt7TP2 ExAgO2LjObthg== Date: Sat, 25 Jan 2025 01:16:55 +0900 From: Masami Hiramatsu (Google) To: Nam Cao Cc: Steven Rostedt , naveen@kernel.org, anil.s.keshavamurthy@intel.com, davem@davemloft.net, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, john.ogness@linutronix.de Subject: Re: kprobe on local function Message-Id: <20250125011655.449075f32f50de6fa7ff20b8@kernel.org> In-Reply-To: <20250123024224.d0h8d4HV@linutronix.de> References: <20250122075939.qw7HsuMd@linutronix.de> <20250122172936.09bc8c9a@gandalf.local.home> <20250123104842.92336327f898e546c5b67536@kernel.org> <20250123024224.d0h8d4HV@linutronix.de> X-Mailer: Sylpheed 3.8.0beta1 (GTK+ 2.24.33; x86_64-pc-linux-gnu) Precedence: bulk X-Mailing-List: linux-trace-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Thu, 23 Jan 2025 03:42:24 +0100 Nam Cao wrote: > On Thu, Jan 23, 2025 at 10:48:42AM +0900, Masami Hiramatsu wrote: > > On Wed, 22 Jan 2025 17:29:36 -0500 > > Steven Rostedt wrote: > > > > > On Wed, 22 Jan 2025 08:59:39 +0100 > > > Nam Cao wrote: > > > > I installed a kretprobe on the function "rt_mutex_slowunlock". Although the > > > > function is called, the probe is never hit. > > > > > > > > > > How did you install the kretprobe? > > Sorry for being unclear. I install it from within the kernel: > > struct kretprobe mutex_event = { > .kp.symbol_name = "rt_mutex_slowunlock", > .entry_handler = entry_handler, > .handler = return_handler, > }; > > register_kretprobe(&mutex_event); This is using symbol name to install kretprobe. Thus if there are rt_mutex_slowunlock.XXX symbols are generated by the compiler, it can not install kretprobe to those symbols. > > ... > > > Have you tried "perf probe" to install the probes? That uses dwarf info to > > > find all the functions. > > > > Yes, I recommend you to use `perf probe` (under tools/perf) to put kretprobes > > on those copies. See tools/perf/Documentation/perf-probe.txt for details. > > So a similar solution for in-kernel code is kallsyms_on_each_match_symbol(). > But I expect register_kretprobe() to do that by itself; or at least report > a failure if caller specifies the symbol name, but multiple symbols of the > same name are found. Because at least to me, my code "looks correct and > should work" but it doesn't. Now what you could know is that the kernel does not call the "rt_mutex_slowunlock" symbol and it may call another copied symbol. That's a good to know right? Note that this in-kernel kretprobe API is a low-layer API, so that user is responsible for controlling it carefully. This kretprobe API itself does NOT know which copy of the optimized symbols are used. (there could be inlined instance too) If you need to know how many copies are generated or inlined, you need to use the debuginfo as "perf probe" tool does, which can not handled by the kernel itself because it is too big to be loaded. That is why I made the "perf probe" as an external tool. Thank you, > > Thank you both for your replies, > Nam -- Masami Hiramatsu (Google)