Live performance tools?

Live performance tools?

[John Goerzen]

Hi,

We are deploying a number of Linux desktop clients running NFS against
our Linux NFS server.  We are seeing occasional odd spikes in traffic
that are causing performance troubles for all users.  We are trying to
isolate the cause of these spikes, but to date haven't been able to.
The best we can do is use iostat and verify that yes, the disk we
expected is seeing a lot of traffic.

nfsd threads seem to be impervious to lsof.  top also doesn't show much
about them, and of course you can't strace them.

Is there any tool out there that could give us any of this sort of info:

 * What IP addresses are generating high volumes of read or write
   traffic

 * What files on disk are being accessed frequently via NFS

 * Anything else that could help us pinpoint the trouble

nfsstat does not seem to provide fine enough detail for this.

Thanks,

-- John


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Live performance tools?

[Eli Stair]

Try lighting up an instance of ntop on the network you've got your NFS 
server(s) on.  Either span/mirror the switch ports so it sees all the 
traffic in promisc mode, or turn on sflow forwarding to the box.  As for 
determining what files are being accessed, you can use ethereal to 
capture traffic on the wire and analyze file ops occuring, but it's not 
fun to try and analyze #ops/file... you'd have to filter quite well, 
dump as XML/text and do some post-processing to generate your stats.

You could also start up inotify on the linux NFS server, watching the 
entire directory tree you're exporting via NFS, log the output, and 
later analyze the file ops as reported.  Maybe you should turn on some 
trending/monitoring of the clients themselves, I'd suggest ganglia for 
ease-of-deployment, but you could also use net-snmp and cacti, or 
monami... there are a number of ways to get data on traffic occuring at 
both ends of the transaction.

There's also an ncurses-based traffic tool that's sometimes handy for 
looking at NFS, but I don't recall its name offhand.


/eli

John Goerzen wrote:
> Hi,
> 
> We are deploying a number of Linux desktop clients running NFS against
> our Linux NFS server.  We are seeing occasional odd spikes in traffic
> that are causing performance troubles for all users.  We are trying to
> isolate the cause of these spikes, but to date haven't been able to.
> The best we can do is use iostat and verify that yes, the disk we
> expected is seeing a lot of traffic.
> 
> nfsd threads seem to be impervious to lsof.  top also doesn't show much
> about them, and of course you can't strace them.
> 
> Is there any tool out there that could give us any of this sort of info:
> 
>  * What IP addresses are generating high volumes of read or write
>    traffic
> 
>  * What files on disk are being accessed frequently via NFS
> 
>  * Anything else that could help us pinpoint the trouble
> 
> nfsstat does not seem to provide fine enough detail for this.
> 
> Thanks,
> 
> -- John
> 
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc.
> Still grepping through log files to find problems?  Stop.
> Now Search log events and configuration files using AJAX and a browser.
> Download your FREE copy of Splunk now >>  http://get.splunk.com/
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
> 


-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Live performance tools?

[John Goerzen]

On Tue August 14 2007 12:39:52 pm Eli Stair wrote:
> Try lighting up an instance of ntop on the network you've got your NFS
> server(s) on.  Either span/mirror the switch ports so it sees all the
> traffic in promisc mode, or turn on sflow forwarding to the box.  As for

I was using iftop, but the traffic was too "spikey" to be able to identify 
long-term trends with it.  I'll take a look at ntop.

> determining what files are being accessed, you can use ethereal to
> capture traffic on the wire and analyze file ops occuring, but it's not
> fun to try and analyze #ops/file... you'd have to filter quite well,

Yes, that's exactly the problem I was fearing.

> You could also start up inotify on the linux NFS server, watching the
> entire directory tree you're exporting via NFS, log the output, and

Oh, EXCELLENT idea.  I don't know why I didn't think of that.  That could be 
very useful indeed.

I'm not familiar with ganglia, cacti, or monami, but will investigate.

Thanks very much for the suggestions.  This will be helpful.

-- John

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Live performance tools?

[Steinar H. Gunderson]

On Tue, Aug 14, 2007 at 10:39:52AM -0700, Eli Stair wrote:
> You could also start up inotify on the linux NFS server, watching the 
> entire directory tree you're exporting via NFS, log the output, and 
> later analyze the file ops as reported.

blktrace might also be useful here.

/* Steinar */
-- 
Homepage: http://www.sesse.net/

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs