From: "J. Bruce Fields" <bfields@fieldses.org>
To: Peng Yu <pengyu.ut@gmail.com>
Cc: linux-nfs@vger.kernel.org
Subject: Re: 30 time speed difference between using NFS and without using NFS
Date: Tue, 31 Aug 2010 21:15:34 -0400 [thread overview]
Message-ID: <20100901011533.GA27635@fieldses.org> (raw)
In-Reply-To: <AANLkTikGx4SeY1RBQ4_SbcAwd02bCTtQ_0Sz+Pzs+sbU@mail.gmail.com>
On Tue, Aug 31, 2010 at 04:27:35PM -0500, Peng Yu wrote:
> On Tue, Aug 31, 2010 at 11:23 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
> > On Mon, Aug 30, 2010 at 04:39:54PM -0500, Peng Yu wrote:
> >> Please see below for the answers of you questions.
> >>
> >> $ find . -type d|wc
> >> 1491 1491 53372
> >>
> >> Search the directory as a local directory
> >>
> >> $ time find . -name 'data.frame' -type d
> >> ./library/base/data.frame
> >> ./library/base/data.frame/data.frame
> >>
> >> real 0m0.044s
> >> user 0m0.012s
> >> sys 0m0.028s
> >>
> >> Search the directory as a NFS directory. The same command runs three 3
> >> times in series.
> >>
> >> $ time find . -name 'data.frame' -type d
> >> ./library/base/data.frame
> >> ./library/base/data.frame/data.frame
> >>
> >> real 0m2.205s
> >> user 0m0.040s
> >> sys 0m0.430s
> >> $ time find . -name 'data.frame' -type d
> >> ./library/base/data.frame
> >> ./library/base/data.frame/data.frame
> >>
> >> real 0m1.203s
> >> user 0m0.060s
> >> sys 0m0.120s
> >> $ time find . -name 'data.frame' -type d
> >> ./library/base/data.frame
> >> ./library/base/data.frame/data.frame
> >>
> >> real 0m1.227s
> >> user 0m0.040s
> >> sys 0m0.200s
> >>
> >> Ping the NFS server gives me ttl=64 time=0.156 ms.
> >
> > So if readding each of those directories required only a single round
> > trip, and if network round trip time were the dominating factor, the
> > whole thing would only take a second. So there may be more rpc's (is
> > find also stat'ing every directory entry?), and/or you may be
> > bottlenecked by somethign else (e.g. seek time on the server).
> >
> > You might try using /proc/self/mounstats on the client to figure out
> > what rpc's are sent during the find and the average time they take.
>
> Hi,
>
> I don't really know how to read /proc/self/mounstats
There's also a script in nfs-utils, under
tools/nfs-iostat/nfs-iostat.py, that may help.
> But here is the corresponding record in it on the client machine
> before and after I run the find command once. Are you able to tell
> what is wrong?
There's not necessarily even anything wrong. NFS and local access tend
to perform differently, so if the difference here is a problem then you
need to figure out why and then decide what to do about it.
--b.
>
> #######before
>
> device xx.xx.x.xx:/pearson/data mounted on /pearson/data with fstype
> nfs statvers=1.0
> opts: rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.4.20,mountvers=3,mountport=0,mountproto=tcp
> age: 167462
> caps: caps=0x3fcf,wtmult=4096,dtsize=4096,bsize=0,namlen=255
> sec: flavor=1,pseudoflavor=1
> events: 97266 2937027 45 379 43366 10794 3030879 707 3 239 553 71267
> 53078 256 331 248 0 3941 0 1 209 0 0 0 0
> bytes: 3620495167 2489192 0 0 29243366 1919994 7293 553
> RPC iostats version: 1.0 p/v: 100003/3 (nfs)
> xprt: tcp 870 1 6 0 11 144892 144890 2 323286 727
> per-op statistics
> NULL: 0 0 0 0 0 0 0 0
> GETATTR: 97268 97268 0 17511024 10894016 300 27490 34980
> SETATTR: 428 428 0 92916 61632 0 1000 1030
> LOOKUP: 11128 11128 0 2158020 1509208 30 5010 5640
> ACCESS: 28926 28926 0 5210656 3471000 110 8000 9950
> .........
>
>
> #######After
>
> device xx.xx.x.xx:/pearson/data mounted on /pearson/data with fstype
> nfs statvers=1.0
> opts: rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.4.20,mountvers=3,mountport=0,mountproto=tcp
> age: 167478
> caps: caps=0x3fcf,wtmult=4096,dtsize=4096,bsize=0,namlen=255
> sec: flavor=1,pseudoflavor=1
> events: 101739 2943313 45 379 46349 10794 3043133 707 3 239 553 71267
> 56060 256 331 248 0 3941 0 1 209 0 0 0 0
> bytes: 3620495167 2489192 0 0 29243366 1919994 7293 553
> RPC iostats version: 1.0 p/v: 100003/3 (nfs)
> xprt: tcp 870 1 6 0 3 150855 150853 2 341175 727
> per-op statistics
> NULL: 0 0 0 0 0 0 0 0
> GETATTR: 101741 101741 0 18298272 11394992 310 28740 36660
> SETATTR: 428 428 0 92916 61632 0 1000 1030
> LOOKUP: 11128 11128 0 2158020 1509208 30 5010 5640
> ACCESS: 30416 30416 0 5478856 3649800 110 8290 10360
> .......
>
>
> --
> Regards,
> Peng
prev parent reply other threads:[~2010-09-01 1:16 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-08-27 23:08 30 time speed difference between using NFS and without using NFS Peng Yu
2010-08-30 17:09 ` J. Bruce Fields
2010-08-30 21:39 ` Peng Yu
2010-08-31 16:23 ` J. Bruce Fields
2010-08-31 21:27 ` Peng Yu
2010-09-01 1:15 ` J. Bruce Fields [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100901011533.GA27635@fieldses.org \
--to=bfields@fieldses.org \
--cc=linux-nfs@vger.kernel.org \
--cc=pengyu.ut@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox