How to monitor Linux NFS client load?

How to monitor Linux NFS client load?

[Andrey Borzenkov]

[-- Attachment #1: Type: Text/Plain, Size: 1106 bytes --]

Recently we have the case of very high latencies on NFS reads as 
reported by application (SAP R/3). NFS server was NetApp FAS; according 
to NetApp statistic, average volume read latencies were in order 10ms, 
while SAP stats gave 30-50ms. Systems were interconnected by dedicated 
1Gb/s Cisco switches (3750G) with ca. 30% max load on interfaces.

On advice of my colleague we changed sunrpc.tcp_slot_table_entries from 
default 16 to 128 which seemed to make situation much better - without 
changing load pattern of filer in any visible way.

Now, I can understand, why we observed much higher latency on system and 
why changing (what effectively is) queue depth helped. But I am totally 
frustrated that there does not appear to be *any* possibility to detect 
this situation on Linux side and to get a real numbers of real NFS IO 
latencies or number of requests waiting to be executed (and I do not 
even dream about per-mount point stats).

I am grateful for any hints how can we monitor Linux NFS client and get 
real-life numbers of what happens inside. Thank you!

-andrey

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: How to monitor Linux NFS client load?

[Trond Myklebust]

On Sat, 2009-07-11 at 21:03 +0400, Andrey Borzenkov wrote:
> Recently we have the case of very high latencies on NFS reads as 
> reported by application (SAP R/3). NFS server was NetApp FAS; according 
> to NetApp statistic, average volume read latencies were in order 10ms, 
> while SAP stats gave 30-50ms. Systems were interconnected by dedicated 
> 1Gb/s Cisco switches (3750G) with ca. 30% max load on interfaces.
> 
> On advice of my colleague we changed sunrpc.tcp_slot_table_entries from 
> default 16 to 128 which seemed to make situation much better - without 
> changing load pattern of filer in any visible way.
> 
> Now, I can understand, why we observed much higher latency on system and 
> why changing (what effectively is) queue depth helped. But I am totally 
> frustrated that there does not appear to be *any* possibility to detect 
> this situation on Linux side and to get a real numbers of real NFS IO 
> latencies or number of requests waiting to be executed (and I do not 
> even dream about per-mount point stats).
> 
> I am grateful for any hints how can we monitor Linux NFS client and get 
> real-life numbers of what happens inside. Thank you!

See the nfs-iostat utility in the nfs-utils package:

http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=tools/nfs-iostat/nfs-iostat.py;h=9626d42609b9485c7fda0c9ef69d698f9fa929fd;hb=HEAD

Trond

Re: How to monitor Linux NFS client load?

[Trond Myklebust]

On Sat, 2009-07-11 at 18:44 -0400, Trond Myklebust wrote:
> On Sat, 2009-07-11 at 21:03 +0400, Andrey Borzenkov wrote:
> > I am grateful for any hints how can we monitor Linux NFS client and get 
> > real-life numbers of what happens inside. Thank you!
> 
> See the nfs-iostat utility in the nfs-utils package:
> 
> http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=tools/nfs-iostat/nfs-iostat.py;h=9626d42609b9485c7fda0c9ef69d698f9fa929fd;hb=HEAD

Oh, and recent versions of 'sar' (from the 'sysstat' package) should
also have support for NFS monitoring.

Trond

Re: How to monitor Linux NFS client load?

[Andrey Borzenkov]

[-- Attachment #1: Type: Text/Plain, Size: 1975 bytes --]

On Sunday 12 of July 2009 02:44:27 Trond Myklebust wrote:
> On Sat, 2009-07-11 at 21:03 +0400, Andrey Borzenkov wrote:
> > Recently we have the case of very high latencies on NFS reads as
> > reported by application (SAP R/3). NFS server was NetApp FAS;
> > according to NetApp statistic, average volume read latencies were
> > in order 10ms, while SAP stats gave 30-50ms. Systems were
> > interconnected by dedicated 1Gb/s Cisco switches (3750G) with ca.
> > 30% max load on interfaces.
> >
> > On advice of my colleague we changed sunrpc.tcp_slot_table_entries
> > from default 16 to 128 which seemed to make situation much better -
> > without changing load pattern of filer in any visible way.
> >
> > Now, I can understand, why we observed much higher latency on
> > system and why changing (what effectively is) queue depth helped.
> > But I am totally frustrated that there does not appear to be *any*
> > possibility to detect this situation on Linux side and to get a
> > real numbers of real NFS IO latencies or number of requests waiting
> > to be executed (and I do not even dream about per-mount point
> > stats).
> >
> > I am grateful for any hints how can we monitor Linux NFS client and
> > get real-life numbers of what happens inside. Thank you!
>
> See the nfs-iostat utility in the nfs-utils package:
>
> http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=tools/nfs-i
>ostat/nfs-iostat.py;h=9626d42609b9485c7fda0c9ef69d698f9fa929fd;hb=HEAD
>

This looks exactly like what is required, except ...

cn1:~ # ./nfs-iostat.py 2
Traceback (most recent call last):
  File "./nfs-iostat.py", line 559, in ?
    iostat_command(prog)
  File "./nfs-iostat.py", line 464, in iostat_command
    mountstats = parse_stats_file('/proc/self/mountstats')
  File "./nfs-iostat.py", line 434, in parse_stats_file
    f = file(filename)
IOError: [Errno 2] No such file or directory: '/proc/self/mountstats'

Which minimum kernel version is required for it?

Thank you!

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: How to monitor Linux NFS client load?

[Trond Myklebust]

On Sun, 2009-07-12 at 08:38 +0400, Andrey Borzenkov wrote:
> On Sunday 12 of July 2009 02:44:27 Trond Myklebust wrote:
> > On Sat, 2009-07-11 at 21:03 +0400, Andrey Borzenkov wrote:
> > > Recently we have the case of very high latencies on NFS reads as
> > > reported by application (SAP R/3). NFS server was NetApp FAS;
> > > according to NetApp statistic, average volume read latencies were
> > > in order 10ms, while SAP stats gave 30-50ms. Systems were
> > > interconnected by dedicated 1Gb/s Cisco switches (3750G) with ca.
> > > 30% max load on interfaces.
> > >
> > > On advice of my colleague we changed sunrpc.tcp_slot_table_entries
> > > from default 16 to 128 which seemed to make situation much better -
> > > without changing load pattern of filer in any visible way.
> > >
> > > Now, I can understand, why we observed much higher latency on
> > > system and why changing (what effectively is) queue depth helped.
> > > But I am totally frustrated that there does not appear to be *any*
> > > possibility to detect this situation on Linux side and to get a
> > > real numbers of real NFS IO latencies or number of requests waiting
> > > to be executed (and I do not even dream about per-mount point
> > > stats).
> > >
> > > I am grateful for any hints how can we monitor Linux NFS client and
> > > get real-life numbers of what happens inside. Thank you!
> >
> > See the nfs-iostat utility in the nfs-utils package:
> >
> > http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=blob;f=tools/nfs-i
> >ostat/nfs-iostat.py;h=9626d42609b9485c7fda0c9ef69d698f9fa929fd;hb=HEAD
> >
> 
> This looks exactly like what is required, except ...
> 
> cn1:~ # ./nfs-iostat.py 2
> Traceback (most recent call last):
>   File "./nfs-iostat.py", line 559, in ?
>     iostat_command(prog)
>   File "./nfs-iostat.py", line 464, in iostat_command
>     mountstats = parse_stats_file('/proc/self/mountstats')
>   File "./nfs-iostat.py", line 434, in parse_stats_file
>     f = file(filename)
> IOError: [Errno 2] No such file or directory: '/proc/self/mountstats'
> 
> Which minimum kernel version is required for it?
> 
> Thank you!

You need kernel 2.6.17 or newer.

Cheers
  Trond