From: "J. Bruce Fields" <bfields@fieldses.org>
To: Peng Yu <pengyu.ut@gmail.com>
Cc: linux-nfs@vger.kernel.org
Subject: Re: 30 time speed difference between using NFS and without using NFS
Date: Tue, 31 Aug 2010 21:15:34 -0400 [thread overview]
Message-ID: <20100901011533.GA27635@fieldses.org> (raw)
In-Reply-To: <AANLkTikGx4SeY1RBQ4_SbcAwd02bCTtQ_0Sz+Pzs+sbU@mail.gmail.com>
On Tue, Aug 31, 2010 at 04:27:35PM -0500, Peng Yu wrote:
> On Tue, Aug 31, 2010 at 11:23 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
> > On Mon, Aug 30, 2010 at 04:39:54PM -0500, Peng Yu wrote:
> >> Please see below for the answers of you questions.
> >>
> >> $ find . -type d|wc
> >> 1491 1491 53372
> >>
> >> Search the directory as a local directory
> >>
> >> $ time find . -name 'data.frame' -type d
> >> ./library/base/data.frame
> >> ./library/base/data.frame/data.frame
> >>
> >> real 0m0.044s
> >> user 0m0.012s
> >> sys 0m0.028s
> >>
> >> Search the directory as a NFS directory. The same command runs three 3
> >> times in series.
> >>
> >> $ time find . -name 'data.frame' -type d
> >> ./library/base/data.frame
> >> ./library/base/data.frame/data.frame
> >>
> >> real 0m2.205s
> >> user 0m0.040s
> >> sys 0m0.430s
> >> $ time find . -name 'data.frame' -type d
> >> ./library/base/data.frame
> >> ./library/base/data.frame/data.frame
> >>
> >> real 0m1.203s
> >> user 0m0.060s
> >> sys 0m0.120s
> >> $ time find . -name 'data.frame' -type d
> >> ./library/base/data.frame
> >> ./library/base/data.frame/data.frame
> >>
> >> real 0m1.227s
> >> user 0m0.040s
> >> sys 0m0.200s
> >>
> >> Ping the NFS server gives me ttl=64 time=0.156 ms.
> >
> > So if readding each of those directories required only a single round
> > trip, and if network round trip time were the dominating factor, the
> > whole thing would only take a second. So there may be more rpc's (is
> > find also stat'ing every directory entry?), and/or you may be
> > bottlenecked by somethign else (e.g. seek time on the server).
> >
> > You might try using /proc/self/mounstats on the client to figure out
> > what rpc's are sent during the find and the average time they take.
>
> Hi,
>
> I don't really know how to read /proc/self/mounstats
There's also a script in nfs-utils, under
tools/nfs-iostat/nfs-iostat.py, that may help.
> But here is the corresponding record in it on the client machine
> before and after I run the find command once. Are you able to tell
> what is wrong?
There's not necessarily even anything wrong. NFS and local access tend
to perform differently, so if the difference here is a problem then you
need to figure out why and then decide what to do about it.
--b.
>
> #######before
>
> device xx.xx.x.xx:/pearson/data mounted on /pearson/data with fstype
> nfs statvers=1.0
> opts: rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.4.20,mountvers=3,mountport=0,mountproto=tcp
> age: 167462
> caps: caps=0x3fcf,wtmult=4096,dtsize=4096,bsize=0,namlen=255
> sec: flavor=1,pseudoflavor=1
> events: 97266 2937027 45 379 43366 10794 3030879 707 3 239 553 71267
> 53078 256 331 248 0 3941 0 1 209 0 0 0 0
> bytes: 3620495167 2489192 0 0 29243366 1919994 7293 553
> RPC iostats version: 1.0 p/v: 100003/3 (nfs)
> xprt: tcp 870 1 6 0 11 144892 144890 2 323286 727
> per-op statistics
> NULL: 0 0 0 0 0 0 0 0
> GETATTR: 97268 97268 0 17511024 10894016 300 27490 34980
> SETATTR: 428 428 0 92916 61632 0 1000 1030
> LOOKUP: 11128 11128 0 2158020 1509208 30 5010 5640
> ACCESS: 28926 28926 0 5210656 3471000 110 8000 9950
> .........
>
>
> #######After
>
> device xx.xx.x.xx:/pearson/data mounted on /pearson/data with fstype
> nfs statvers=1.0
> opts: rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.4.20,mountvers=3,mountport=0,mountproto=tcp
> age: 167478
> caps: caps=0x3fcf,wtmult=4096,dtsize=4096,bsize=0,namlen=255
> sec: flavor=1,pseudoflavor=1
> events: 101739 2943313 45 379 46349 10794 3043133 707 3 239 553 71267
> 56060 256 331 248 0 3941 0 1 209 0 0 0 0
> bytes: 3620495167 2489192 0 0 29243366 1919994 7293 553
> RPC iostats version: 1.0 p/v: 100003/3 (nfs)
> xprt: tcp 870 1 6 0 3 150855 150853 2 341175 727
> per-op statistics
> NULL: 0 0 0 0 0 0 0 0
> GETATTR: 101741 101741 0 18298272 11394992 310 28740 36660
> SETATTR: 428 428 0 92916 61632 0 1000 1030
> LOOKUP: 11128 11128 0 2158020 1509208 30 5010 5640
> ACCESS: 30416 30416 0 5478856 3649800 110 8290 10360
> .......
>
>
> --
> Regards,
> Peng
prev parent reply other threads:[~2010-09-01 1:16 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-08-27 23:08 30 time speed difference between using NFS and without using NFS Peng Yu
2010-08-30 17:09 ` J. Bruce Fields
2010-08-30 21:39 ` Peng Yu
2010-08-31 16:23 ` J. Bruce Fields
2010-08-31 21:27 ` Peng Yu
2010-09-01 1:15 ` J. Bruce Fields [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100901011533.GA27635@fieldses.org \
--to=bfields@fieldses.org \
--cc=linux-nfs@vger.kernel.org \
--cc=pengyu.ut@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.