about /proc/net/rpc/nfsd

about /proc/net/rpc/nfsd

[Jordi Prats]

Hi all,
I've been researching the meaning of the contents of the 
/proc/net/rpc/nfsd file using the 2.6.23 kernel sources.

$ cat /proc/net/rpc/nfsd
rc 0 959398 336415498
fh 0 0 0 0 0
io 743250564 1224442800
th 16 9193 1322.293 121.816 246.724 134.568 138.140 84.223 32.349 37.372 
7.716 28.454
ra 32 2044764 555 263 206 115 119 82 89 98 64 50896
net 337376442 0 337375903 1956
rpc 337333487 12 12 0 0
proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 41 322956721 70371 521208 10500197 134 2096930 779219 48538 
4760 1 0 40535 4351 11538 0 0 19386 57668 47 0 222804
proc4 2 0 0

This is what I've learned:

* rc (reply cache): <hits> <misses> <nocache> (according a thread on 
this list from Chuck Leaver and Thomas Talpey with me)
- hits: client it's retransmiting (a bad thing! o hits is good)
- misses: a operation that requires caching
- nocache: a operation that no requires caching

* fh (filehandle): <stale> <total-lookups> <anonlookups> 
<dir-not-in-cache> <nodir-not-in-cache>
- stale: *supose* to be file handle errors (like when you resize the 
underlying filesystem)
- total-lookups, anonlookups, dir-not-in-cache, nodir-not-in-cache: do 
not appear (and I always seen it as zeros). So I supose they are unused.

* io (input/output): <bytes-read> <bytes-written>
- bytes-read: bytes read directly from disk
- bytes-written: bytes written to disk

* th (threads): <threads> <fullcnt> <10%-20%> <20%-30%> ... <90%-100%> 
<100%>
  - threads: number of nfsd threads
  - fullcnt: number of times that the last 10% of threads  (so all 
threads) are busy.
  - 10%-20%, 20%-30% ... 90%-100%: 10 numbers representing 10-20%, 
20-30% to 100%. Counts the number of times a given interval are busy. 
For example you have 10 threats and you get this:
12 11 10 9 8 7 6 5 4 3

12 times the thread 1 was busy, 11 times the thread 2 was busy, 10 time 
the thread 3 was busy, and so on...

* ra (read-ahead): <cache-size> <10%> <20%> ... <100%> <not-found>
- cache-size: always the double of number threads
- 10%, 20% ... 100%: how deep it found what was looking for. I *suppose* 
this means how far the cached block is from the original block that was 
first requested.
- not-found: not found in the read-ahead cache

* net: <netcnt> <netudpcnt> <nettcpcnt> <nettcpconn>
- netcnt: counts every read
- netudpcnt: counts every UDP packet it receives
- nettcpcnt: counts every time it receives data from a TCP connection
- nettcpconn: count every TCP connection it receives

* rpc: <rpccnt> <rpcbadfmt+rpcbadauth+rpcbadclnt> <rpcbadfmt> 
<rpcbadauth> <rpcbadclnt>
- rpccnt: counts all rpc operations
- rpcbadfmt: counts if while processing a RPC it encounters the 
following errors: err_bad_dir, err_bad_rpc, err_bad_prog, err_bad_vers, 
err_bad_proc, err_bad
- rpcbadauth: bad authentication. It does not count if you try to mount 
from a machine that it's not in your exports file
- rpcbadclnt: unused

I've had no time to investigate about proc<N>. I *supose* that N is the 
protocol version (2 for NFS2, 3 for NFS3 and 4 for NFSv4) So they are 
stats about operations from a given protocol. I'll research this in short.


Sorry for this so long mail... That's all what I've got reading the 
source files. There's anything incorrect? There's some "educated guests" 
  So, could anyone please confirm or refute them?

If you what me to do something with this I'd be glad of it!

regards,
Jordi




-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: about /proc/net/rpc/nfsd

[Jordi Prats]

Just a extended explanation about busy threads:

You have 10 threats and you get this:
th ... 12 11 10 9 8 7 6 5 4 3

  12 times the thread 1 was busy, 11 times the thread 2 was busy (and 
also the thread 1), 10 time  the thread 3 was busy (and also 1 and 2), 
and so on...

So if you increase the third number all the threads before this range 
was also busy. So if you what to know how may threads are usually busy 
you should look for the bigger number.

For example:
th 16 21109 650.836 171.150 154.441 41.955 49.185 24.350 8.245 13.590 
7.330 27.259

Here most of the time they are idle, but you also have peaks of activity 
as you see on the last numbers.

Note that this is what I've understood. That does not mean I'm certain 
about I'm saying on this email!

regards,
Jordi



Jordi Prats wrote:
> Hi all,
> I've been researching the meaning of the contents of the 
> /proc/net/rpc/nfsd file using the 2.6.23 kernel sources.
> 
> $ cat /proc/net/rpc/nfsd
> rc 0 959398 336415498
> fh 0 0 0 0 0
> io 743250564 1224442800
> th 16 9193 1322.293 121.816 246.724 134.568 138.140 84.223 32.349 37.372 
> 7.716 28.454
> ra 32 2044764 555 263 206 115 119 82 89 98 64 50896
> net 337376442 0 337375903 1956
> rpc 337333487 12 12 0 0
> proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> proc3 22 41 322956721 70371 521208 10500197 134 2096930 779219 48538 
> 4760 1 0 40535 4351 11538 0 0 19386 57668 47 0 222804
> proc4 2 0 0
> 
> This is what I've learned:
> 
> * rc (reply cache): <hits> <misses> <nocache> (according a thread on 
> this list from Chuck Leaver and Thomas Talpey with me)
> - hits: client it's retransmiting (a bad thing! o hits is good)
> - misses: a operation that requires caching
> - nocache: a operation that no requires caching
> 
> * fh (filehandle): <stale> <total-lookups> <anonlookups> 
> <dir-not-in-cache> <nodir-not-in-cache>
> - stale: *supose* to be file handle errors (like when you resize the 
> underlying filesystem)
> - total-lookups, anonlookups, dir-not-in-cache, nodir-not-in-cache: do 
> not appear (and I always seen it as zeros). So I supose they are unused.
> 
> * io (input/output): <bytes-read> <bytes-written>
> - bytes-read: bytes read directly from disk
> - bytes-written: bytes written to disk
> 
> * th (threads): <threads> <fullcnt> <10%-20%> <20%-30%> ... <90%-100%> 
> <100%>
>   - threads: number of nfsd threads
>   - fullcnt: number of times that the last 10% of threads  (so all 
> threads) are busy.
>   - 10%-20%, 20%-30% ... 90%-100%: 10 numbers representing 10-20%, 
> 20-30% to 100%. Counts the number of times a given interval are busy. 
> For example you have 10 threats and you get this:
> 12 11 10 9 8 7 6 5 4 3
> 
> 12 times the thread 1 was busy, 11 times the thread 2 was busy, 10 time 
> the thread 3 was busy, and so on...
> 
> * ra (read-ahead): <cache-size> <10%> <20%> ... <100%> <not-found>
> - cache-size: always the double of number threads
> - 10%, 20% ... 100%: how deep it found what was looking for. I *suppose* 
> this means how far the cached block is from the original block that was 
> first requested.
> - not-found: not found in the read-ahead cache
> 
> * net: <netcnt> <netudpcnt> <nettcpcnt> <nettcpconn>
> - netcnt: counts every read
> - netudpcnt: counts every UDP packet it receives
> - nettcpcnt: counts every time it receives data from a TCP connection
> - nettcpconn: count every TCP connection it receives
> 
> * rpc: <rpccnt> <rpcbadfmt+rpcbadauth+rpcbadclnt> <rpcbadfmt> 
> <rpcbadauth> <rpcbadclnt>
> - rpccnt: counts all rpc operations
> - rpcbadfmt: counts if while processing a RPC it encounters the 
> following errors: err_bad_dir, err_bad_rpc, err_bad_prog, err_bad_vers, 
> err_bad_proc, err_bad
> - rpcbadauth: bad authentication. It does not count if you try to mount 
> from a machine that it's not in your exports file
> - rpcbadclnt: unused
> 
> I've had no time to investigate about proc<N>. I *supose* that N is the 
> protocol version (2 for NFS2, 3 for NFS3 and 4 for NFSv4) So they are 
> stats about operations from a given protocol. I'll research this in short.
> 
> 
> Sorry for this so long mail... That's all what I've got reading the 
> source files. There's anything incorrect? There's some "educated guests" 
>   So, could anyone please confirm or refute them?
> 
> If you what me to do something with this I'd be glad of it!
> 
> regards,
> Jordi
> 
> 
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >> http://get.splunk.com/
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
> 
> 

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: about /proc/net/rpc/nfsd

[Gabriel Barazer]

On 10/22/2007 11:36:56 PM +0200, Jordi Prats <jprats@cesca.es> wrote:
> Hi all,
> I've been researching the meaning of the contents of the 
> /proc/net/rpc/nfsd file using the 2.6.23 kernel sources.
> 
> 
> Sorry for this so long mail... That's all what I've got reading the 
> source files. There's anything incorrect? There's some "educated guests" 
>   So, could anyone please confirm or refute them?
> 
> If you what me to do something with this I'd be glad of it!

Wow this is an awesome job you've done here to document precisely this 
proc file. I've tagged your mail and copied it in a safe place.  I'm 
sure this would be very useful to keep a place where all these proc 
files are explained like this, maybe on the linux-nfs wiki ?

Great job Jordi !

Gabriel

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: about /proc/net/rpc/nfsd

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]

Gabriel Barazer wrote:
> On 10/22/2007 11:36:56 PM +0200, Jordi Prats <jprats@cesca.es> wrote:
>> Hi all,
>> I've been researching the meaning of the contents of the 
>> /proc/net/rpc/nfsd file using the 2.6.23 kernel sources.
>>
>>
>> Sorry for this so long mail... That's all what I've got reading the 
>> source files. There's anything incorrect? There's some "educated guests" 
>>   So, could anyone please confirm or refute them?
>>
>> If you what me to do something with this I'd be glad of it!
> 
> Wow this is an awesome job you've done here to document precisely this 
> proc file. I've tagged your mail and copied it in a safe place.  I'm 
> sure this would be very useful to keep a place where all these proc 
> files are explained like this, maybe on the linux-nfs wiki ?

This belongs in the NFS server performance chapter of the NFS How-To. 
The How-To's maintainer is csmithere@gmail.com.

[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 315 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
email;internet:chuck dot lever at nospam oracle dot com
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard


[-- Attachment #3: Type: text/plain, Size: 314 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

[-- Attachment #4: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: about /proc/net/rpc/nfsd

[Jordi Prats]

Hi,
What would you thing about creating a nfsd man page on section 5 with 
what does it mean every value?

I think on wiki pages should go how to interpret to enhance performance, 
but not the meaning of every value you can get on statistics.

So, I'm thinking this as two different things to answer the following 
questions:
- What is this value? (on a man page)
- How do I interpret this value to enhance performance? (on wiki's page 
about performance)

What do you think?

regards,
Jordi

Chuck Lever wrote:
> Gabriel Barazer wrote:
>> On 10/22/2007 11:36:56 PM +0200, Jordi Prats <jprats@cesca.es> wrote:
>>> Hi all,
>>> I've been researching the meaning of the contents of the 
>>> /proc/net/rpc/nfsd file using the 2.6.23 kernel sources.
>>>
>>>
>>> Sorry for this so long mail... That's all what I've got reading the 
>>> source files. There's anything incorrect? There's some "educated 
>>> guests"   So, could anyone please confirm or refute them?
>>>
>>> If you what me to do something with this I'd be glad of it!
>>
>> Wow this is an awesome job you've done here to document precisely this 
>> proc file. I've tagged your mail and copied it in a safe place.  I'm 
>> sure this would be very useful to keep a place where all these proc 
>> files are explained like this, maybe on the linux-nfs wiki ?
> 
> This belongs in the NFS server performance chapter of the NFS How-To. 
> The How-To's maintainer is csmithere@gmail.com.

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: about /proc/net/rpc/nfsd

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 1625 bytes --]

Jordi Prats wrote:
> Hi,
> What would you thing about creating a nfsd man page on section 5 with 
> what does it mean every value?
> 
> I think on wiki pages should go how to interpret to enhance performance, 
> but not the meaning of every value you can get on statistics.
> 
> So, I'm thinking this as two different things to answer the following 
> questions:
> - What is this value? (on a man page)
> - How do I interpret this value to enhance performance? (on wiki's page 
> about performance)
> 
> What do you think?

I don't have any objection to that.  What would you call it: nfsd(7)? 
I'm not sure in which man section proc files are documented.

> Chuck Lever wrote:
>> Gabriel Barazer wrote:
>>> On 10/22/2007 11:36:56 PM +0200, Jordi Prats <jprats@cesca.es> wrote:
>>>> Hi all,
>>>> I've been researching the meaning of the contents of the 
>>>> /proc/net/rpc/nfsd file using the 2.6.23 kernel sources.
>>>>
>>>>
>>>> Sorry for this so long mail... That's all what I've got reading the 
>>>> source files. There's anything incorrect? There's some "educated 
>>>> guests"   So, could anyone please confirm or refute them?
>>>>
>>>> If you what me to do something with this I'd be glad of it!
>>>
>>> Wow this is an awesome job you've done here to document precisely 
>>> this proc file. I've tagged your mail and copied it in a safe place.  
>>> I'm sure this would be very useful to keep a place where all these 
>>> proc files are explained like this, maybe on the linux-nfs wiki ?
>>
>> This belongs in the NFS server performance chapter of the NFS How-To. 
>> The How-To's maintainer is csmithere@gmail.com.


[-- Attachment #2: chuck.lever.vcf --]
[-- Type: text/x-vcard, Size: 315 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Chuck
org:Oracle Corporation;Corporate Architecture: Linux Projects Group
adr:;;1015 Granger Avenue;Ann Arbor;MI;48104;USA
email;internet:chuck dot lever at nospam oracle dot com
title:Principal Member of Staff
tel;work:+1 248 614 5091
x-mozilla-html:FALSE
version:2.1
end:vcard


[-- Attachment #3: Type: text/plain, Size: 314 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >> http://get.splunk.com/

[-- Attachment #4: Type: text/plain, Size: 140 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs