[NFS] NFS performance debuggins

[NFS] NFS performance debuggins

[Adrian von Bidder]

[-- Attachment #1.1: Type: text/plain, Size: 4149 bytes --]

Hi,

Environment:

several Debian based clients (Debian etch and etchnhalf kernels, this means 
2.6.18 or 2.6.24); Debian etch (2.6.18 kernel) NFS (v3) server.  Network 
seems basically ok ("ping -f -s 3000" works without losses, ifconfig and 
switch monitoring shows no errors) with no noticeable load.  Disks seem to 
have very little load either, NFS server has no other tasks.

Performance is sluggish :-(  Basically works, though -- no spurious errors.

tcpdump shows many "reply ERR 1448" etc. msgs whenever NFS activitiy is 
going on (both stat like with "find /home" or read/write with dd)

+++
16:49:24.778560 IP 10.0.1.2.2049 > 10.0.0.209.809066834: reply ERR 1448
16:49:24.790304 IP 10.0.1.2.2049 > 10.0.0.209.943279929: reply ERR 1448
16:49:24.801380 IP 10.0.1.2.2049 > 10.0.0.209.2001885801: reply ERR 1448
16:49:24.802173 IP 10.0.1.2.2049 > 10.0.0.209.860835666: reply ERR 1448
16:49:24.805286 IP 10.0.1.2.2049 > 10.0.0.209.1479697199: reply ERR 1332
16:49:24.807679 IP 10.0.1.2.2049 > 10.0.0.209.1096249460: reply ERR 1448
16:49:24.808358 IP 10.0.1.2.2049 > 10.0.0.209.2000902760: reply ERR 1332
16:49:24.809097 IP 10.0.1.2.2049 > 10.0.0.209.926298420: reply ERR 1448
16:49:24.809100 IP 10.0.1.2.2049 > 10.0.0.209.25105411: reply ERR 1332
16:49:24.817923 IP 10.0.1.2.2049 > 10.0.0.209.1366504235: reply ERR 1448
16:49:24.817927 IP 10.0.1.2.2049 > 10.0.0.209.352525071: reply ERR 1332
16:49:24.820397 IP 10.0.1.2.2049 > 10.0.0.209.269848846: reply ERR 1332
16:49:24.822097 IP 10.0.1.2.2049 > 10.0.0.209.1345540144: reply ERR 1448
16:49:24.822856 IP 10.0.1.2.2049 > 10.0.0.209.944780599: reply ERR 1448
16:49:24.825109 IP 10.0.1.2.2049 > 10.0.0.209.1395668559: reply ERR 1448
16:49:24.825112 IP 10.0.1.2.2049 > 10.0.0.209.1999335795: reply ERR 1332
16:49:24.827813 IP 10.0.1.2.2049 > 10.0.0.209.1685677906: reply ERR 1332
16:49:24.829439 IP 10.0.1.2.2049 > 10.0.0.209.1666084982: reply ERR 1448
16:49:24.829443 IP 10.0.1.2.2049 > 10.0.0.209.1415656037: reply ERR 1332
16:49:24.839013 IP 10.0.1.2.2049 > 10.0.0.209.911226680: reply ERR 1448
16:49:24.839017 IP 10.0.1.2.2049 > 10.0.0.209.1735414852: reply ERR 1332
16:49:24.841325 IP 10.0.1.2.2049 > 10.0.0.209.911358287: reply ERR 1332
16:49:24.842092 IP 10.0.1.2.2049 > 10.0.0.209.1364284211: reply ERR 1448
16:49:24.842800 IP 10.0.1.2.2049 > 10.0.0.209.258643250: reply ERR 1332
16:49:24.844256 IP 10.0.1.2.2049 > 10.0.0.209.1666017882: reply ERR 1448
16:49:24.844996 IP 10.0.1.2.2049 > 10.0.0.209.808595513: reply ERR 1448
16:49:24.845674 IP 10.0.1.2.2049 > 10.0.0.209.2000779112: reply ERR 1448
16:49:24.845677 IP 10.0.1.2.2049 > 10.0.0.209.1652175121: reply ERR 1332
16:49:24.847120 IP 10.0.1.2.2049 > 10.0.0.209.944722769: reply ERR 1448
16:49:24.847123 IP 10.0.1.2.2049 > 10.0.0.209.1682657874: reply ERR 1332
16:49:24.849334 IP 10.0.1.2.2049 > 10.0.0.209.944714835: reply ERR 1448
16:49:24.850873 IP 10.0.1.2.2049 > 10.0.0.209.1345861938: reply ERR 1448
16:49:24.918710 IP 10.0.1.2.2049 > 10.0.0.179.1936680564: reply ERR 1448
16:49:24.918719 IP 10.0.1.2.2049 > 10.0.0.179.1698508838: reply ERR 1448
16:49:24.921911 IP 10.0.1.2.2049 > 10.0.0.179.1633904741: reply ERR 1448
+++

Mount options: "rw,noatime,rsize=8192,wsize=8192,intr,hard,addr=10.0.1.2", 
it seems to pick tcp by default.  I had problems with UDP from some of the 
clients due to a strangely buggy VDSL switch in the path, so I haven't 
tried that again (I want to keep the DSL clients and the non-DSL clients 
identical if this is at all possible, so I can switch equipment around 
without reconfiguration.)

That performance is not optimal whith todays desktop environments (tons of 
small configuration files in both oo.org and kde) at login/program start on 
cold caches is one thing, but performance

Now where do I start debugging this?

-- 
Development costs of average proprietary and free software don't differ
radically because the methods are pretty much the same.  The huge
difference lies in the way the developers try to recoup their costs, not
in the costs they have to compensate.
        -- Florian Weimer on debian-security

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 388 bytes --]

[-- Attachment #2: Type: text/plain, Size: 247 bytes --]

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debuggins

[Trond Myklebust]

On Mon, 2008-06-23 at 16:59 +0200, Adrian von Bidder wrote:
> Hi,
> 
> Environment:
> 
> several Debian based clients (Debian etch and etchnhalf kernels, this means 
> 2.6.18 or 2.6.24); Debian etch (2.6.18 kernel) NFS (v3) server.  Network 
> seems basically ok ("ping -f -s 3000" works without losses, ifconfig and 
> switch monitoring shows no errors) with no noticeable load.  Disks seem to 
> have very little load either, NFS server has no other tasks.
> 
> Performance is sluggish :-(  Basically works, though -- no spurious errors.
> 
> tcpdump shows many "reply ERR 1448" etc. msgs whenever NFS activitiy is 
> going on (both stat like with "find /home" or read/write with dd)
> 
> +++
> 16:49:24.778560 IP 10.0.1.2.2049 > 10.0.0.209.809066834: reply ERR 1448
> 16:49:24.790304 IP 10.0.1.2.2049 > 10.0.0.209.943279929: reply ERR 1448
> 16:49:24.801380 IP 10.0.1.2.2049 > 10.0.0.209.2001885801: reply ERR 1448
> 16:49:24.802173 IP 10.0.1.2.2049 > 10.0.0.209.860835666: reply ERR 1448
> 16:49:24.805286 IP 10.0.1.2.2049 > 10.0.0.209.1479697199: reply ERR 1332
> 16:49:24.807679 IP 10.0.1.2.2049 > 10.0.0.209.1096249460: reply ERR 1448
> 16:49:24.808358 IP 10.0.1.2.2049 > 10.0.0.209.2000902760: reply ERR 1332
> 16:49:24.809097 IP 10.0.1.2.2049 > 10.0.0.209.926298420: reply ERR 1448
> 16:49:24.809100 IP 10.0.1.2.2049 > 10.0.0.209.25105411: reply ERR 1332
> 16:49:24.817923 IP 10.0.1.2.2049 > 10.0.0.209.1366504235: reply ERR 1448
> 16:49:24.817927 IP 10.0.1.2.2049 > 10.0.0.209.352525071: reply ERR 1332
> 16:49:24.820397 IP 10.0.1.2.2049 > 10.0.0.209.269848846: reply ERR 1332
> 16:49:24.822097 IP 10.0.1.2.2049 > 10.0.0.209.1345540144: reply ERR 1448
> 16:49:24.822856 IP 10.0.1.2.2049 > 10.0.0.209.944780599: reply ERR 1448
> 16:49:24.825109 IP 10.0.1.2.2049 > 10.0.0.209.1395668559: reply ERR 1448
> 16:49:24.825112 IP 10.0.1.2.2049 > 10.0.0.209.1999335795: reply ERR 1332
> 16:49:24.827813 IP 10.0.1.2.2049 > 10.0.0.209.1685677906: reply ERR 1332
> 16:49:24.829439 IP 10.0.1.2.2049 > 10.0.0.209.1666084982: reply ERR 1448
> 16:49:24.829443 IP 10.0.1.2.2049 > 10.0.0.209.1415656037: reply ERR 1332
> 16:49:24.839013 IP 10.0.1.2.2049 > 10.0.0.209.911226680: reply ERR 1448
> 16:49:24.839017 IP 10.0.1.2.2049 > 10.0.0.209.1735414852: reply ERR 1332
> 16:49:24.841325 IP 10.0.1.2.2049 > 10.0.0.209.911358287: reply ERR 1332
> 16:49:24.842092 IP 10.0.1.2.2049 > 10.0.0.209.1364284211: reply ERR 1448
> 16:49:24.842800 IP 10.0.1.2.2049 > 10.0.0.209.258643250: reply ERR 1332
> 16:49:24.844256 IP 10.0.1.2.2049 > 10.0.0.209.1666017882: reply ERR 1448
> 16:49:24.844996 IP 10.0.1.2.2049 > 10.0.0.209.808595513: reply ERR 1448
> 16:49:24.845674 IP 10.0.1.2.2049 > 10.0.0.209.2000779112: reply ERR 1448
> 16:49:24.845677 IP 10.0.1.2.2049 > 10.0.0.209.1652175121: reply ERR 1332
> 16:49:24.847120 IP 10.0.1.2.2049 > 10.0.0.209.944722769: reply ERR 1448
> 16:49:24.847123 IP 10.0.1.2.2049 > 10.0.0.209.1682657874: reply ERR 1332
> 16:49:24.849334 IP 10.0.1.2.2049 > 10.0.0.209.944714835: reply ERR 1448
> 16:49:24.850873 IP 10.0.1.2.2049 > 10.0.0.209.1345861938: reply ERR 1448
> 16:49:24.918710 IP 10.0.1.2.2049 > 10.0.0.179.1936680564: reply ERR 1448
> 16:49:24.918719 IP 10.0.1.2.2049 > 10.0.0.179.1698508838: reply ERR 1448
> 16:49:24.921911 IP 10.0.1.2.2049 > 10.0.0.179.1633904741: reply ERR 1448
> +++
> 
> Mount options: "rw,noatime,rsize=8192,wsize=8192,intr,hard,addr=10.0.1.2", 
> it seems to pick tcp by default.  I had problems with UDP from some of the 
> clients due to a strangely buggy VDSL switch in the path, so I haven't 
> tried that again (I want to keep the DSL clients and the non-DSL clients 
> identical if this is at all possible, so I can switch equipment around 
> without reconfiguration.)
> 
> That performance is not optimal whith todays desktop environments (tons of 
> small configuration files in both oo.org and kde) at login/program start on 
> cold caches is one thing, but performance
> 
> Now where do I start debugging this?

In the above dump 1448 is _not_ the error code, but rather the packet
length. You might therefore try using the tcpdump option '-vvv' to see
if you can obtain the actual error value (which should tell you why the
NFS server is rejecting your packets).
Alternatively, you might consider using wireshark/tshark, which can
display NFS packets in a much more readable fashion.

Cheers
  Trond


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debuggins

[J. Bruce Fields]

On Mon, Jun 23, 2008 at 04:59:57PM +0200, Adrian von Bidder wrote:
> Hi,
> 
> Environment:
> 
> several Debian based clients (Debian etch and etchnhalf kernels, this means 
> 2.6.18 or 2.6.24); Debian etch (2.6.18 kernel) NFS (v3) server.  Network 
> seems basically ok ("ping -f -s 3000" works without losses, ifconfig and 
> switch monitoring shows no errors) with no noticeable load.  Disks seem to 
> have very little load either, NFS server has no other tasks.
> 
> Performance is sluggish :-(  Basically works, though -- no spurious errors.

In what way exactly is it sluggish?

> tcpdump shows many "reply ERR 1448" etc. msgs whenever NFS activitiy is 
> going on (both stat like with "find /home" or read/write with dd)

I'm afraid I don't know how to read that tcpdump output.

--b.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[Adrian von Bidder]

[-- Attachment #1.1: Type: text/plain, Size: 1841 bytes --]

Hi again,

Thanks for your replies (You too, Trond)

On Monday 23 June 2008 21.28:36 you wrote:

[... NFS performance ...]

> In what way exactly is it sluggish?

Starting KDE, opening documents, sometimes also closing oo.org and saving 
documents takes several seconds longer than on local disk.

Certainly network latency (especially with these silly lots of small config 
files) takes some time, but I'm still surprised.  At the same time, I don't 
have data to compare a "known good" NFS against ours, so perhaps NFS is 
indeed so slow?

>
> > tcpdump shows many "reply ERR 1448" etc. msgs whenever NFS activitiy is
> > going on (both stat like with "find /home" or read/write with dd)
>
> I'm afraid I don't know how to read that tcpdump output.

tcpdump "-vvv" doesn't give more information on these packets; at the same 
time wireshark doesn't show anything suspicious except tons of wrong TCP 
checksums caused (I hope...) by offloading.  I'll have to look if I can get 
the raw traffic at the network switch to check this (but I think with 30% 
and more wrong tcp checksums, traffic would completely break down so I'm 
quite confident here.)


Slightly different topic: is there an NFS related mailing list I can 
subscribe to?  This one is apparently closed for new subscribers, and the 
bounce instructs me to send mail to majordomo-MogPR669STc76Z2rM5mHXA@public.gmane.org which 
bounces :-(  Reading others' NFS postings might just give me more ideas on 
where to look.


TODO today: play around with NFSv4 on the shaky assumption that nfsv3 is 
actually working but net latency is killing my performance.


cheers
-- vbi



-- 
The typewriting machine, when played with expression, is no more
annoying than the piano when played by a sister or near relation.
		-- Oscar Wilde

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 388 bytes --]

[-- Attachment #2: Type: text/plain, Size: 247 bytes --]

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[J. Bruce Fields]

On Tue, Jun 24, 2008 at 12:17:24PM +0200, Adrian von Bidder wrote:
> Hi again,
> 
> Thanks for your replies (You too, Trond)

The custom around here is to leave everyone on the cc: line.

> 
> On Monday 23 June 2008 21.28:36 you wrote:
> 
> [... NFS performance ...]
> 
> > In what way exactly is it sluggish?
> 
> Starting KDE, opening documents, sometimes also closing oo.org and saving 
> documents takes several seconds longer than on local disk.

"close" on nfs is an operation that requires a round-trip to the server
and waiting for the disk to commit any writes made before the close, so
if you've got to do a lot of those it can take time.  Fooling with the
journaling on the exported filesystem may help.

> Certainly network latency (especially with these silly lots of small config 
> files) takes some time, but I'm still surprised.  At the same time, I don't 
> have data to compare a "known good" NFS against ours, so perhaps NFS is 
> indeed so slow?
> 
> >
> > > tcpdump shows many "reply ERR 1448" etc. msgs whenever NFS activitiy is
> > > going on (both stat like with "find /home" or read/write with dd)
> >
> > I'm afraid I don't know how to read that tcpdump output.
> 
> tcpdump "-vvv" doesn't give more information on these packets; at the same 
> time wireshark doesn't show anything suspicious except tons of wrong TCP 
> checksums caused (I hope...) by offloading.

Yes, that's normal.

> I'll have to look if I can get 
> the raw traffic at the network switch to check this (but I think with 30% 
> and more wrong tcp checksums, traffic would completely break down so I'm 
> quite confident here.)
> 
> 
> Slightly different topic: is there an NFS related mailing list I can 
> subscribe to?  This one is apparently closed for new subscribers, and the 
> bounce instructs me to send mail to majordomo-MogPR669STc76Z2rM5mHXA@public.gmane.org which 
						^^^^
						vger

Where'd the typo in that address get introduced?

> bounces :-(  Reading others' NFS postings might just give me more ideas on 
> where to look.

Should be: http://vger.kernel.org/vger-lists.html#linux-nfs

> TODO today: play around with NFSv4 on the shaky assumption that nfsv3 is 
> actually working but net latency is killing my performance.

Delegations *might* help if the problem is really open latency.

--b.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[Adrian von Bidder]

[-- Attachment #1.1: Type: text/plain, Size: 1734 bytes --]

On Tuesday 24 June 2008 22.29:31 J. Bruce Fields wrote:
> On Tue, Jun 24, 2008 at 12:17:24PM +0200, Adrian von Bidder wrote:

> > Starting KDE, opening documents, sometimes also closing oo.org and
> > saving documents takes several seconds longer than on local disk.
>
> "close" on nfs is an operation that requires a round-trip to the server
> and waiting for the disk to commit any writes made before the close, so
> if you've got to do a lot of those it can take time.  Fooling with the
> journaling on the exported filesystem may help.

Are there tools to measure latencies on NFS?  Given a network dump, desired 
output would be histograms of latencies by file operation?  (Or maybe I can 
catch the information on the client, VFS side instead of NFS?

At this time, I really need to collect more data on where the problem is 
since all I'm doing right now is fooling around based on assumptions... :-(

OTOH I'd suspect KDE/oo.org startup to be mostly reads of those config 
files, so the problem shouldn't be close latencies.  Assumptions again.

> > TODO today: play around with NFSv4 on the shaky assumption that nfsv3
> > is actually working but net latency is killing my performance.
>
> Delegations *might* help if the problem is really open latency.

First tries showed
 * There are no acl on my files now
 * user id mapping seems funny: some users map to nobody, others map 
correctly.  Huh?
 * Performance seems to be ok (timing desktop applications is always 
difficult, and so far I'm working against on the production server with 
varying load anyway...)

Haven't investigated these yet...

cheers
-- vbi


-- 
Today is Sweetmorn, the 30th day of Confusion in the YOLD 3174

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 388 bytes --]

[-- Attachment #2: Type: text/plain, Size: 247 bytes --]

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[Trond Myklebust]

On Wed, 2008-06-25 at 09:02 +0200, Adrian von Bidder wrote:
> On Tuesday 24 June 2008 22.29:31 J. Bruce Fields wrote:
> > On Tue, Jun 24, 2008 at 12:17:24PM +0200, Adrian von Bidder wrote:
> 
> > > Starting KDE, opening documents, sometimes also closing oo.org and
> > > saving documents takes several seconds longer than on local disk.
> >
> > "close" on nfs is an operation that requires a round-trip to the server
> > and waiting for the disk to commit any writes made before the close, so
> > if you've got to do a lot of those it can take time.  Fooling with the
> > journaling on the exported filesystem may help.
> 
> Are there tools to measure latencies on NFS?  Given a network dump, desired 
> output would be histograms of latencies by file operation?  (Or maybe I can 
> catch the information on the client, VFS side instead of NFS?

You can catch the info on the client. See the 'nfs-iostat' tool in the
latest nfs-utils git tree:

  http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=summary

Trond


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[Steve Dickson]

Trond Myklebust wrote:
> On Wed, 2008-06-25 at 09:02 +0200, Adrian von Bidder wrote:
>> On Tuesday 24 June 2008 22.29:31 J. Bruce Fields wrote:
>>> On Tue, Jun 24, 2008 at 12:17:24PM +0200, Adrian von Bidder wrote:
>>>> Starting KDE, opening documents, sometimes also closing oo.org and
>>>> saving documents takes several seconds longer than on local disk.
>>> "close" on nfs is an operation that requires a round-trip to the server
>>> and waiting for the disk to commit any writes made before the close, so
>>> if you've got to do a lot of those it can take time.  Fooling with the
>>> journaling on the exported filesystem may help.
>> Are there tools to measure latencies on NFS?  Given a network dump, desired 
>> output would be histograms of latencies by file operation?  (Or maybe I can 
>> catch the information on the client, VFS side instead of NFS?
> 
> You can catch the info on the client. See the 'nfs-iostat' tool in the
> latest nfs-utils git tree:
One side note on the current state of the nfs-iostat tool... There is some talk 
about changing the name from 'nfs-iostat' to 'nfsiostat' so it will map better 
with the current command names that are in nfs-utils (i.e. no other commands
use have a '-' in their names). So at this point, even though the script is in the tree, 
its not being installed (via a make install) which allows us to make these types
of changes. 

But please do not let this caveat stop you from trying this script and
mountstats script. We definitely need feedback on how well they do or
don't work (and as always... patches are welcomed! ;-) ) Once things harden 
up (via any feedback), I'll added the install code to make these scripts be 
installed (probably in /usr/sbin) which means they will be ready for 
prime time... 

steved.



-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[J. Bruce Fields]

On Wed, Jun 25, 2008 at 09:02:42AM +0200, Adrian von Bidder wrote:
> On Tuesday 24 June 2008 22.29:31 J. Bruce Fields wrote:
> > On Tue, Jun 24, 2008 at 12:17:24PM +0200, Adrian von Bidder wrote:
> 
> > > Starting KDE, opening documents, sometimes also closing oo.org and
> > > saving documents takes several seconds longer than on local disk.
> >
> > "close" on nfs is an operation that requires a round-trip to the server
> > and waiting for the disk to commit any writes made before the close, so
> > if you've got to do a lot of those it can take time.  Fooling with the
> > journaling on the exported filesystem may help.
> 
> Are there tools to measure latencies on NFS?  Given a network dump, desired 
> output would be histograms of latencies by file operation?  (Or maybe I can 
> catch the information on the client, VFS side instead of NFS?
> 
> At this time, I really need to collect more data on where the problem is 
> since all I'm doing right now is fooling around based on assumptions... :-(
> 
> OTOH I'd suspect KDE/oo.org startup to be mostly reads of those config 
> files, so the problem shouldn't be close latencies.  Assumptions again.
> 
> > > TODO today: play around with NFSv4 on the shaky assumption that nfsv3
> > > is actually working but net latency is killing my performance.
> >
> > Delegations *might* help if the problem is really open latency.
> 
> First tries showed
>  * There are no acl on my files now

NFSv4 uses an entirely different type of ACL, for which you need
different client-side tools; see

	http://www.citi.umich.edu/projects/nfsv4/linux/nfs4-acl-tools/

>  * user id mapping seems funny: some users map to nobody, others map 
> correctly.  Huh?

And whereas v2/v3 require only uid's and gid's to agree, v4 (if you're
using auth_sys) requires uid's, gid's, *and* user and group names to
agree.

--b.

>  * Performance seems to be ok (timing desktop applications is always 
> difficult, and so far I'm working against on the production server with 
> varying load anyway...)
> 
> Haven't investigated these yet...


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[Adrian von Bidder]

[-- Attachment #1.1: Type: text/plain, Size: 2860 bytes --]

On Wednesday 25 June 2008 18.56:58 J. Bruce Fields wrote:
> On Wed, Jun 25, 2008 at 09:02:42AM +0200, Adrian von Bidder wrote:
> > On Tuesday 24 June 2008 22.29:31 J. Bruce Fields wrote:
> > > On Tue, Jun 24, 2008 at 12:17:24PM +0200, Adrian von Bidder wrote:
> > > > Starting KDE, opening documents, sometimes also closing oo.org and
> > > > saving documents takes several seconds longer than on local disk.
> > >
> > > "close" on nfs is an operation that requires a round-trip to the
> > > server and waiting for the disk to commit any writes made before the
> > > close, so if you've got to do a lot of those it can take time. 
> > > Fooling with the journaling on the exported filesystem may help.
> >
> > Are there tools to measure latencies on NFS?  Given a network dump,
> > desired output would be histograms of latencies by file operation?  (Or
> > maybe I can catch the information on the client, VFS side instead of
> > NFS?
> >
> > At this time, I really need to collect more data on where the problem
> > is since all I'm doing right now is fooling around based on
> > assumptions... :-(
> >
> > OTOH I'd suspect KDE/oo.org startup to be mostly reads of those config
> > files, so the problem shouldn't be close latencies.  Assumptions again.
> >
> > > > TODO today: play around with NFSv4 on the shaky assumption that
> > > > nfsv3 is actually working but net latency is killing my
> > > > performance.
> > >
> > > Delegations *might* help if the problem is really open latency.
> >
> > First tries showed
> >  * There are no acl on my files now
>
> NFSv4 uses an entirely different type of ACL, for which you need
> different client-side tools; see

I'm absolutely confused on the state of nfs4 acls.  There are so many old 
mailing list posts around that it's difficult to see what information 
applies to the currentimplementation.  So: are POSIX acls in the server 
filesystem somehow mapped to nfsv4 acls on the client side?  I understand 
there understands an RFC proposal on how to do this, and 
http://wiki.linux-nfs.org/wiki/index.php/ACLs is very interesting but 
doesn't really cover the current state of the implementation.

> 	http://www.citi.umich.edu/projects/nfsv4/linux/nfs4-acl-tools/

Hmm.  Not packaged in Debian yet?????? ;-)  So work to do for me if I end up 
using nfs4

>
> >  * user id mapping seems funny: some users map to nobody, others map
> > correctly.  Huh?
>
> And whereas v2/v3 require only uid's and gid's to agree, v4 (if you're
> using auth_sys) requires uid's, gid's, *and* user and group names to
> agree.

I got that fixed.  "localdomain" in idmapd.conf...

cheers
-- vbi


-- 
> Maybe that question would be a good starting point: What's the use for
> a gender field there?
Stalking.
        -- Miriam Ruiz, Marco d'Itri (im that order)

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 388 bytes --]

[-- Attachment #2: Type: text/plain, Size: 247 bytes --]

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[J. Bruce Fields]

On Thu, Jun 26, 2008 at 08:19:29AM +0200, Adrian von Bidder wrote:
> On Wednesday 25 June 2008 18.56:58 J. Bruce Fields wrote:
> > On Wed, Jun 25, 2008 at 09:02:42AM +0200, Adrian von Bidder wrote:
> > > First tries showed
> > >  * There are no acl on my files now
> >
> > NFSv4 uses an entirely different type of ACL, for which you need
> > different client-side tools; see
> 
> I'm absolutely confused on the state of nfs4 acls.  There are so many old 
> mailing list posts around that it's difficult to see what information 
> applies to the currentimplementation.  So: are POSIX acls in the server 
> filesystem somehow mapped to nfsv4 acls on the client side?  I understand 
> there understands an RFC proposal on how to do this, and 
> http://wiki.linux-nfs.org/wiki/index.php/ACLs is very interesting but 
> doesn't really cover the current state of the implementation.

I've attempted to update that wiki page to fix that problem; could you
take a look and tell me whether the changes help?

> > 	http://www.citi.umich.edu/projects/nfsv4/linux/nfs4-acl-tools/
> 
> Hmm.  Not packaged in Debian yet?????? ;-)  So work to do for me if I end up 
> using nfs4

Yes.  Volunteers to package those things welcomed....

(Or maybe we should get them into nfs-utils at some point.  I don't
know.)

--b.

> > >  * user id mapping seems funny: some users map to nobody, others map
> > > correctly.  Huh?
> >
> > And whereas v2/v3 require only uid's and gid's to agree, v4 (if you're
> > using auth_sys) requires uid's, gid's, *and* user and group names to
> > agree.
> 
> I got that fixed.  "localdomain" in idmapd.conf...
> 
> cheers
> -- vbi
> 
> 
> -- 
> > Maybe that question would be a good starting point: What's the use for
> > a gender field there?
> Stalking.
>         -- Miriam Ruiz, Marco d'Itri (im that order)



-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[Adrian von Bidder]

[-- Attachment #1.1: Type: text/plain, Size: 2098 bytes --]

On Thursday 26 June 2008 19.03:42 you wrote:
> On Thu, Jun 26, 2008 at 08:19:29AM +0200, Adrian von Bidder wrote:
> > On Wednesday 25 June 2008 18.56:58 J. Bruce Fields wrote:
> > > On Wed, Jun 25, 2008 at 09:02:42AM +0200, Adrian von Bidder wrote:
> > > > First tries showed
> > > >  * There are no acl on my files now
> > >
> > > NFSv4 uses an entirely different type of ACL, for which you need
> > > different client-side tools; see
> >
> > I'm absolutely confused on the state of nfs4 acls.  There are so many
> > old mailing list posts around that it's difficult to see what
> > information applies to the currentimplementation.  So: are POSIX acls
> > in the server filesystem somehow mapped to nfsv4 acls on the client
> > side?  I understand there understands an RFC proposal on how to do
> > this, and
> > http://wiki.linux-nfs.org/wiki/index.php/ACLs is very interesting but
> > doesn't really cover the current state of the implementation.
>
> I've attempted to update that wiki page to fix that problem; could you
> take a look and tell me whether the changes help?

The summary is: "POSIX ACLs on the server side turn (more or less) into 
NFSv4 ACL on the client.", correct?  So our current ACL based system of 
permissions should work except that we can't modify permissions on the 
client side until I've deployed nfs4 acl tools ...

>
> > > 	http://www.citi.umich.edu/projects/nfsv4/linux/nfs4-acl-tools/
> >
> > Hmm.  Not packaged in Debian yet?????? ;-)  So work to do for me if I
> > end up using nfs4
>
> Yes.  Volunteers to package those things welcomed....

... which means you've possibly got your Debian packager, and I might even 
be able to do that on company time.

Ok, I guess I know in which direction to put my work now, thanks a lot.  
(nfs-iostat will be the first thing I'll try, though, to get some more 
data.)

cheers
-- vbi


-- 
Why on earth should we teach children
that they are not allowed to share the toys.
        -- Patrick Harvie, Member of the Scottish Parliament
           Speaking at Debconf7



[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 388 bytes --]

[-- Attachment #2: Type: text/plain, Size: 247 bytes --]

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php

[-- Attachment #3: Type: text/plain, Size: 362 bytes --]

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[Krishna Kumar2]

linux-nfs-owner@vger.kernel.org wrote on 06/25/2008 07:29:16 PM:

> But please do not let this caveat stop you from trying this script and
> mountstats script. We definitely need feedback on how well they do or
> don't work (and as always... patches are welcomed! ;-) ) Once things
harden
> up (via any feedback), I'll added the install code to make these scripts
be
> installed (probably in /usr/sbin) which means they will be ready for
> prime time...

Hi Steve,

When I ran this util, it worked the first time:

[root@localhost ~]# /tmp/nfs-iostat

localhost:/local mounted on /nfs:

   op/s         rpc bklog
   1.27            0.01
read:             ops/s            Kb/s           Kb/op         retrans
avg RTT (ms)    avg exe (ms)
                  0.619          39.772          64.273        0 (0.0%)
16.889          16.927
write:            ops/s            Kb/s           Kb/op         retrans
avg RTT (ms)    avg exe (ms)
                  0.626          39.112          62.523        0 (0.0%)
26.357         4039.241

8.124.111.71:/mnt mounted on /mnt:

   op/s         rpc bklog
   3.00            0.00
read:             ops/s            Kb/s           Kb/op         retrans
avg RTT (ms)    avg exe (ms)
                  0.000           0.000           0.000        0 (0.0%)
0.000           0.000
write:            ops/s            Kb/s           Kb/op         retrans
avg RTT (ms)    avg exe (ms)
                  0.000           0.000           0.000        0 (0.0%)
0.000           0.000

(KK: for some reason, the real NFS mount gives all zero. Also, for
loopback, the read/write
BW is very poor: less than 40 Kb/s).

But from the next time I ran, and all subsequent runs (and even after a
reboot),
I get:

[root@localhost init.d]# /tmp/nfs-iostat
localhost:/local mounted on /nfs:

   op/s         rpc bklog
   0.04            0.00
read:             ops/s            Kb/s           Kb/op         retrans
avg RTT (ms)    avg exe (ms)
                  0.000           0.000           0.000        0 (0.0%)
0.000           0.000
write:            ops/s            Kb/s           Kb/op         retrans
avg RTT (ms)    avg exe (ms)
                  0.000           0.000           0.000        0 (0.0%)
0.000           0.000

8.124.111.71:/mnt mounted on /mnt:

   op/s         rpc bklog
   6.00            0.00
read:             ops/s            Kb/s           Kb/op         retrans
avg RTT (ms)    avg exe (ms)
                  0.000           0.000           0.000        0 (0.0%)
0.000           0.000
write:            ops/s            Kb/s           Kb/op         retrans
avg RTT (ms)    avg exe (ms)
                  0.000           0.000           0.000        0 (0.0%)
0.000           0.000

(KK: every filesystem gives zero for all parameters).

What could be wrong?

Thanks,

- KK


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[Chuck Lever]

On Fri, Jun 27, 2008 at 3:30 AM, Krishna Kumar2 <krkumar2@in.ibm.com> wrote:
> linux-nfs-owner@vger.kernel.org wrote on 06/25/2008 07:29:16 PM:
>
>> But please do not let this caveat stop you from trying this script and
>> mountstats script. We definitely need feedback on how well they do or
>> don't work (and as always... patches are welcomed! ;-) ) Once things
> harden
>> up (via any feedback), I'll added the install code to make these scripts
> be
>> installed (probably in /usr/sbin) which means they will be ready for
>> prime time...
>
> Hi Steve,
>
> When I ran this util, it worked the first time:
>
> [root@localhost ~]# /tmp/nfs-iostat
>
> localhost:/local mounted on /nfs:
>
>   op/s         rpc bklog
>   1.27            0.01
> read:             ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                  0.619          39.772          64.273        0 (0.0%)
> 16.889          16.927
> write:            ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                  0.626          39.112          62.523        0 (0.0%)
> 26.357         4039.241
>
> 8.124.111.71:/mnt mounted on /mnt:
>
>   op/s         rpc bklog
>   3.00            0.00
> read:             ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                  0.000           0.000           0.000        0 (0.0%)
> 0.000           0.000
> write:            ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                  0.000           0.000           0.000        0 (0.0%)
> 0.000           0.000
>
> (KK: for some reason, the real NFS mount gives all zero. Also, for
> loopback, the read/write
> BW is very poor: less than 40 Kb/s).

As an aside, Jim Rees pointed out to me that that should be KB/s, not
Kb/s.  I will post a patch to address that.

> But from the next time I ran, and all subsequent runs (and even after a
> reboot), I get:
>
> [root@localhost init.d]# /tmp/nfs-iostat
> localhost:/local mounted on /nfs:
>
>   op/s         rpc bklog
>   0.04            0.00
> read:             ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                  0.000           0.000           0.000        0 (0.0%)
> 0.000           0.000
> write:            ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                  0.000           0.000           0.000        0 (0.0%)
> 0.000           0.000
>
> 8.124.111.71:/mnt mounted on /mnt:
>
>   op/s         rpc bklog
>   6.00            0.00
> read:             ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                  0.000           0.000           0.000        0 (0.0%)
> 0.000           0.000
> write:            ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                  0.000           0.000           0.000        0 (0.0%)
> 0.000           0.000
>
> (KK: every filesystem gives zero for all parameters).
>
> What could be wrong?

At a guess, it's because the client is now caching your file data?

Can you post a copy of your mount command line, and your
/proc/self/mountstats file after a test run (but before you unmount)?

--
Chuck Lever

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[J. Bruce Fields]

On Fri, Jun 27, 2008 at 08:24:49AM +0200, Adrian von Bidder wrote:
> On Thursday 26 June 2008 19.03:42 you wrote:
> > On Thu, Jun 26, 2008 at 08:19:29AM +0200, Adrian von Bidder wrote:
> > > On Wednesday 25 June 2008 18.56:58 J. Bruce Fields wrote:
> > > > On Wed, Jun 25, 2008 at 09:02:42AM +0200, Adrian von Bidder wrote:
> > > > > First tries showed
> > > > >  * There are no acl on my files now
> > > >
> > > > NFSv4 uses an entirely different type of ACL, for which you need
> > > > different client-side tools; see
> > >
> > > I'm absolutely confused on the state of nfs4 acls.  There are so many
> > > old mailing list posts around that it's difficult to see what
> > > information applies to the currentimplementation.  So: are POSIX acls
> > > in the server filesystem somehow mapped to nfsv4 acls on the client
> > > side?  I understand there understands an RFC proposal on how to do
> > > this, and
> > > http://wiki.linux-nfs.org/wiki/index.php/ACLs is very interesting but
> > > doesn't really cover the current state of the implementation.
> >
> > I've attempted to update that wiki page to fix that problem; could you
> > take a look and tell me whether the changes help?
> 
> The summary is: "POSIX ACLs on the server side turn (more or less) into 
> NFSv4 ACL on the client.", correct?

Yes (but note that that translation happens on the server).

> So our current ACL based system of 
> permissions should work except that we can't modify permissions on the 
> client side until I've deployed nfs4 acl tools ...

ACL's will definitely still be enforced correctly, yes--enforcement of
permissions is entirely left up to the exported filesystem.

> > > > 	http://www.citi.umich.edu/projects/nfsv4/linux/nfs4-acl-tools/
> > >
> > > Hmm.  Not packaged in Debian yet?????? ;-)  So work to do for me if I
> > > end up using nfs4
> >
> > Yes.  Volunteers to package those things welcomed....
> 
> ... which means you've possibly got your Debian packager, and I might even 
> be able to do that on company time.

If you get a chance to work on that, that would be great.

> Ok, I guess I know in which direction to put my work now, thanks a lot.  
> (nfs-iostat will be the first thing I'll try, though, to get some more 
> data.)

Yes, that's probably best.  (NFSv4 will be somewhat of a long shot, and
in any case it'll help to first understand better where the time's
going.)

--b.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[Olga Kornievskaia]

Krishna Kumar2 wrote:
> linux-nfs-owner@vger.kernel.org wrote on 06/25/2008 07:29:16 PM:
>
>   
>> But please do not let this caveat stop you from trying this script and
>> mountstats script. We definitely need feedback on how well they do or
>> don't work (and as always... patches are welcomed! ;-) ) Once things
>>     
> harden
>   
>> up (via any feedback), I'll added the install code to make these scripts
>>     
> be
>   
>> installed (probably in /usr/sbin) which means they will be ready for
>> prime time...
>>     
>
> Hi Steve,
>
> When I ran this util, it worked the first time:
>
> [root@localhost ~]# /tmp/nfs-iostat
>
> localhost:/local mounted on /nfs:
>
>    op/s         rpc bklog
>    1.27            0.01
> read:             ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                   0.619          39.772          64.273        0 (0.0%)
> 16.889          16.927
> write:            ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                   0.626          39.112          62.523        0 (0.0%)
> 26.357         4039.241
>
> 8.124.111.71:/mnt mounted on /mnt:
>
>    op/s         rpc bklog
>    3.00            0.00
> read:             ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                   0.000           0.000           0.000        0 (0.0%)
> 0.000           0.000
> write:            ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                   0.000           0.000           0.000        0 (0.0%)
> 0.000           0.000
>
> (KK: for some reason, the real NFS mount gives all zero. Also, for
> loopback, the read/write
> BW is very poor: less than 40 Kb/s).
>
> But from the next time I ran, and all subsequent runs (and even after a
> reboot),
> I get:
>
> [root@localhost init.d]# /tmp/nfs-iostat
> localhost:/local mounted on /nfs:
>
>    op/s         rpc bklog
>    0.04            0.00
> read:             ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                   0.000           0.000           0.000        0 (0.0%)
> 0.000           0.000
> write:            ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                   0.000           0.000           0.000        0 (0.0%)
> 0.000           0.000
>
> 8.124.111.71:/mnt mounted on /mnt:
>
>    op/s         rpc bklog
>    6.00            0.00
> read:             ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                   0.000           0.000           0.000        0 (0.0%)
> 0.000           0.000
> write:            ops/s            Kb/s           Kb/op         retrans
> avg RTT (ms)    avg exe (ms)
>                   0.000           0.000           0.000        0 (0.0%)
> 0.000           0.000
>
> (KK: every filesystem gives zero for all parameters).
>
> What could be wrong?
>   
Are you running nfs-iostat just once? While doing your testing, you 
should do "nfs-iostat 2" so that every 2secs nfs-iostat displays 
information about the on-going test (you can choose any other polling 
interval value).
> Thanks,
>
> - KK
>
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://sourceforge.net/services/buy/index.php
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
> _______________________________________________
> Please note that nfs@lists.sourceforge.net is being discontinued.
> Please subscribe to linux-nfs@vger.kernel.org instead.
>     http://vger.kernel.org/vger-lists.html#linux-nfs
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>   

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[Trond Myklebust]

On Fri, 2008-06-27 at 09:44 -0400, Chuck Lever wrote:

> >
> > (KK: for some reason, the real NFS mount gives all zero. Also, for
> > loopback, the read/write
> > BW is very poor: less than 40 Kb/s).
> 
> As an aside, Jim Rees pointed out to me that that should be KB/s, not
> Kb/s.  I will post a patch to address that.

Strictly speaking, neither of the above are correct: the SI abbreviation
for kilo is 'k'. IOW it should be 'kB/s'

Cheers
  Trond


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] NFS performance debugging

[Krishna Kumar2]

chucklever@gmail.com wrote on 06/27/2008 07:14:38 PM:

> At a guess, it's because the client is now caching your file data?

Today it is working fine, so maybe something was wrong earlier. However,
nfs-iostat shows very low BW:

   op/s         rpc bklog
  42.60            0.00
read:             ops/s            Kb/s           Kb/op         retrans
avg RTT (ms)     avg exe (ms)
                 42.542         2734.324         64.273        0 (0.0%)
18.480   18.518
write:            ops/s            Kb/s           Kb/op         retrans
avg RTT (ms)     avg exe (ms)
                  0.000           0.000           0.000        0 (0.0%)
0.000    0.000

A cat of the file (after fresh mount of /local and /nfs) on the NFS
mountpoint to
/dev/null gives around 29.4 MB/s (vs 39.1 MB/s on /local).

> Can you post a copy of your mount command line, and your
> /proc/self/mountstats file after a test run (but before you unmount)?

mount:
      mount -o
rw,bg,hard,nointr,proto=tcp,vers=3,rsize=65536,wsize=65536,timeo=600,noatime
 localhost:/local /nfs

mountstats file:

device rootfs mounted on / with fstype rootfs
device /dev/root mounted on / with fstype ext3
device /dev mounted on /dev with fstype tmpfs
device /proc mounted on /proc with fstype proc
device /sys mounted on /sys with fstype sysfs
device none mounted on /selinux with fstype selinuxfs
device /proc/bus/usb mounted on /proc/bus/usb with fstype usbfs
device devpts mounted on /dev/pts with fstype devpts
device tmpfs mounted on /dev/shm with fstype tmpfs
device none mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
device sunrpc mounted on /var/lib/nfs/rpc_pipefs with fstype rpc_pipefs
device /etc/auto.misc mounted on /misc with fstype autofs
device -hosts mounted on /net with fstype autofs
device nfsd mounted on /proc/fs/nfsd with fstype nfsd
device /dev/sda6 mounted on /local with fstype ext3
device localhost:/local mounted on /nfs with fstype nfs statvers=1.0
        opts:
rw,vers=3,rsize=65536,wsize=65536,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys
        age:    79
        caps:   caps=0x9,wtmult=4096,dtsize=4096,bsize=0,namelen=255
        sec:    flavor=1,pseudoflavor=1
        events: 0 0 0 0 1 3 4 0 0 609 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0
        bytes:  595722240 0 0 0 596639744 0 145664 0
        RPC iostats version: 1.0  p/v: 100003/3 (nfs)
        xprt:   tcp 855 0 2 0 60 9117 9117 0 72672 0
        per-op statistics
                NULL: 0 0 0 0 0 0 0 0
             GETATTR: 2 2 0 264 224 0 0 0
             SETATTR: 0 0 0 0 0 0 0 0
              LOOKUP: 3 3 0 448 684 0 1 1
              ACCESS: 4 4 0 568 480 0 0 0
            READLINK: 0 0 0 0 0 0 0 0
                READ: 9104 9104 0 1383808 597805056 15 168246 168587
               WRITE: 0 0 0 0 0 0 0 0
              CREATE: 0 0 0 0 0 0 0 0
               MKDIR: 0 0 0 0 0 0 0 0
             SYMLINK: 0 0 0 0 0 0 0 0
               MKNOD: 0 0 0 0 0 0 0 0
              REMOVE: 0 0 0 0 0 0 0 0
               RMDIR: 0 0 0 0 0 0 0 0
              RENAME: 0 0 0 0 0 0 0 0
                LINK: 0 0 0 0 0 0 0 0
             READDIR: 0 0 0 0 0 0 0 0
         READDIRPLUS: 0 0 0 0 0 0 0 0
              FSSTAT: 0 0 0 0 0 0 0 0
              FSINFO: 1 1 0 132 80 0 0 0
            PATHCONF: 0 0 0 0 0 0 0 0
              COMMIT: 0 0 0 0 0 0 0 0

And:

[root@localhost nfs]# nfsstat -rc
Client rpc stats:
calls      retrans    authrefrsh
713710     19         0

Thanks,

- KK


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs