Profiling knfsd

Profiling knfsd

[Eric Veith1]

Hello everybody,

I'm facing the somewhat difficult task of looking under the hood of the
knfsd and profiling its behaviour on WRITE/COMMIT calls. Basically, I'd
like to answer the following questions:

- What amount of payload is comming in?
- What WRITE flags are set (UNSTABLE/DATA_SYNC/FILE_SYNC)?
- What happens to the incoming RPCs (are they stored in a queue, or cached
if UNSTABLE is set? If yes, what is the queue/cache size)?
- What does the nfsd write out to disk?
- How many NFS I/O operations are a kernel VFS (?) operation, and how many
FS IOPs does this make?

Regarding the last question, I already have a way of finding out what I/O
operations containing what payload are done with what delay.

I'm a little at loss here because I have only a very vague idea of where to
look. I though using systemtap and tapping some functions would give me
answers to the questions above. Could somebody give me please some
pointers, be it function names, header files, or a general idea of the call
flow a NFS WRITE packet creates from being received over a socket until it
is written out to disk?

Thank you very much in advance.

                  Eric

--
Eric MSP Veith <eric.veith@de.ibm.com>
Hechtsheimer Str. 2
DE-55131 Mainz
Germany

IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Erich Clementi
Geschäftsführung: Martin Jetter (Vorsitzender), Reinhard Reschke, Christoph
Grandpierre, Matthias Hartmann, Michael Diemer, Martina Koederitz
Sitz der Gesellschaft: Stuttgart
Registergericht: Amtsgericht Stuttgart, HRB 14562
WEEE-Reg.-Nr. DE 99369940

Re: Profiling knfsd

[J. Bruce Fields]

On Tue, May 25, 2010 at 03:36:53PM +0200, Eric Veith1 wrote:
> 
> Hello everybody,
> 
> I'm facing the somewhat difficult task of looking under the hood of the
> knfsd and profiling its behaviour on WRITE/COMMIT calls. Basically, I'd
> like to answer the following questions:
> 
> - What amount of payload is comming in?
> - What WRITE flags are set (UNSTABLE/DATA_SYNC/FILE_SYNC)?
> - What happens to the incoming RPCs (are they stored in a queue, or cached
> if UNSTABLE is set? If yes, what is the queue/cache size)?
> - What does the nfsd write out to disk?
> - How many NFS I/O operations are a kernel VFS (?) operation, and how many
> FS IOPs does this make?
> 
> Regarding the last question, I already have a way of finding out what I/O
> operations containing what payload are done with what delay.
> 
> I'm a little at loss here because I have only a very vague idea of where to
> look. I though using systemtap and tapping some functions would give me
> answers to the questions above. Could somebody give me please some
> pointers, be it function names, header files, or a general idea of the call
> flow a NFS WRITE packet creates from being received over a socket until it
> is written out to disk?

There's a little bit of information in /proc/net/rpc/nfsd and
/proc/fs/nfsd/pool_stats.

Greg Banks had some patches to improve nfsd statistics (and provide some
of the information you're asking for) that never got merged.  If someone
wanted to take over the task of forward-porting them and fixing the
final problems, that might be one way to be to learn the code in
question.

Unfortunately I don't know of any really good documentation; you
probably just have to dive in, read the code, and experiment.

What's the high-level question you're trying to answer?

--b.

> 
> Thank you very much in advance.
> 
>                   Eric
> 
> --
> Eric MSP Veith <eric.veith@de.ibm.com>
> Hechtsheimer Str. 2
> DE-55131 Mainz
> Germany
> 
> IBM Deutschland GmbH
> Vorsitzender des Aufsichtsrats: Erich Clementi
> Geschäftsführung: Martin Jetter (Vorsitzender), Reinhard Reschke, Christoph
> Grandpierre, Matthias Hartmann, Michael Diemer, Martina Koederitz
> Sitz der Gesellschaft: Stuttgart
> Registergericht: Amtsgericht Stuttgart, HRB 14562
> WEEE-Reg.-Nr. DE 99369940
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Profiling knfsd

[Eric Veith1]

Bruce,

thanks for your answer!

"J. Bruce Fields" <bfields@fieldses.org> wrote on 05/25/2010 11:23:12 PM:
> Unfortunately I don't know of any really good documentation; you
> probably just have to dive in, read the code, and experiment.

In the meantime, I've been able to follow the call path quite a bit using
systemtap. Whoohoo, what a nice tool for this. Where I still have problems
is with anything below the actula fs/nfsd/nfsproc.c code, i.e. the VFS
calls and where a chunk of data is actually handled over to the FS.

> What's the high-level question you're trying to answer?

In a short summary: "How many I/O operations are caused by a NFS WRITE
call?" It includes things like following a WRITE call through until it
reaches the FS and the storage below and finding out whether the Kernel or
the FS (which is GPFS) do caching on unstable WRITEs.

The question originates from a performance problem I was once facing where
the DS' controller couldn't handle the client's writeout behaviour when
streaming a very large file. We more or less guessed the DS part, and now
I'd really like to see where the bottleneck is. I'd like to follow a WRITE
through the whole kernel through GPFS until it reaches the DS' controller,
see how many IOPs this WRITE really causes, what the latencies are and
whether there's a write queue that is painfully full at some point.

TIA!

                  Eric

--
Eric MSP Veith <eric.veith@de.ibm.com>
Hechtsheimer Str. 2
DE-55131 Mainz
Germany

IBM Deutschland GmbH
Vorsitzender des Aufsichtsrats: Erich Clementi
Geschäftsführung: Martin Jetter (Vorsitzender), Reinhard Reschke, Christoph
Grandpierre, Matthias Hartmann, Michael Diemer, Martina Koederitz
Sitz der Gesellschaft: Stuttgart
Registergericht: Amtsgericht Stuttgart, HRB 14562
WEEE-Reg.-Nr. DE 99369940