Re: [PATCH 3/4] nfsd: add some stub tracepoints around key vfs functions

[Chuck Lever <chuck.lever@oracle.com>]

On 3/6/25 11:28 AM, Jeff Layton wrote:
> On Thu, 2025-03-06 at 09:29 -0500, Chuck Lever wrote:
>> On 3/6/25 7:38 AM, Jeff Layton wrote:
>>> Sargun set up kprobes to add some of these tracepoints. Convert them to
>>> simple static tracepoints. These are pretty sparse for now, but they
>>> could be expanded in the future as needed.
>>
>> I have mixed feelings about this.

To be very clear: I'm always up for better observability! The details of
this patch are where I start to have some hesitation.


>> - Probably tracepoints should replace the existing dprintk call sites.
>>   dprintk is kind of useless for heavy traffic.
>>
> 
> I'm fine with removing dprintks as we go.

Removing them was controversial a few years ago when I first brought
this up... I would very much like to see these call sites gone, even if
we don't have immediate replacements in the form of trace points.


>> - Seems like other existing tracepoints could report most of the same
>>   information. fh_verify, for example, has a tracepoint that reports
>>   the file handle. There's an svc proc tracepoint, and an NFSv4 COMPOUND
>>   tracepoint that can report XID and procedure.
>>
> 
> The problem there is the lack of context. Yes, I can see that
> fh_verify() got called, but on a busy server it can be hard to tell why
> it got called. I see things like the fh_verify() tracepoint working in
> conjunction with these new tracepoints. IOW, you could match up the
> xids and see which fh_verify() was called for which operation.

If we're talking about NFSv3 only, sunrpc:svc_process records the XID,
nfsd thread, NFSv3 procedure name, and NFSD namespace of each incoming
RPC call. You also get the NFS client's IP address.

You can also enable nfsd:nfsd_fh_verify to capture several of those
items, plus the NFS file handle.

The kernel process information will be identical for both the svc_proc
and nfsd_fh_verify trace points -- that will tie the two records
together so you can match an XID to an NFS procedure and its file handle
argument.

If you want to see a little more you can enable the function_graph
plug-in for nfsd_dispatch().

Another approach is adding trace points in the XDR layer to capture
all of the arguments of incoming RPC calls:

https://web.git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/log/?h=topic-xdr-tracepoints


-- 
Chuck Lever