* 30 time speed difference between using NFS and without using NFS @ 2010-08-27 23:08 Peng Yu 2010-08-30 17:09 ` J. Bruce Fields 0 siblings, 1 reply; 6+ messages in thread From: Peng Yu @ 2010-08-27 23:08 UTC (permalink / raw) To: linux-nfs Hi, I use the following command to search for some directory. find -L some_dir -type d -name some_name If I'm on the machine that has some_dir locally, the run time is real 0m0.199s user 0m0.048s sys 0m0.140s If I'm on another machine that sees the same directory by NFS (NFS3). the run time is real 0m6.509s user 0m0.090s sys 0m1.380s There are 30 time speed difference. Is this normal? Is there any NFS parameter that I should tune to make the latter faster? -- Regards, Peng ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 30 time speed difference between using NFS and without using NFS 2010-08-27 23:08 30 time speed difference between using NFS and without using NFS Peng Yu @ 2010-08-30 17:09 ` J. Bruce Fields 2010-08-30 21:39 ` Peng Yu 0 siblings, 1 reply; 6+ messages in thread From: J. Bruce Fields @ 2010-08-30 17:09 UTC (permalink / raw) To: Peng Yu; +Cc: linux-nfs On Fri, Aug 27, 2010 at 06:08:25PM -0500, Peng Yu wrote: > Hi, > > I use the following command to search for some directory. > > find -L some_dir -type d -name some_name > > If I'm on the machine that has some_dir locally, the run time is > real 0m0.199s > user 0m0.048s > sys 0m0.140s > > If I'm on another machine that sees the same directory by NFS (NFS3). > the run time is > real 0m6.509s > user 0m0.090s > sys 0m1.380s > > There are 30 time speed difference. Is this normal? Is there any NFS > parameter that I should tune to make the latter faster? How many directories are there in some_dir? What's the round trip time (e.g. as reported by ping) to the server? If you repeate the find immediately, is it faster? This can indeed by much slower if it requires revalidating the cached attributes of each file. --b. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 30 time speed difference between using NFS and without using NFS 2010-08-30 17:09 ` J. Bruce Fields @ 2010-08-30 21:39 ` Peng Yu 2010-08-31 16:23 ` J. Bruce Fields 0 siblings, 1 reply; 6+ messages in thread From: Peng Yu @ 2010-08-30 21:39 UTC (permalink / raw) To: J. Bruce Fields; +Cc: linux-nfs On Mon, Aug 30, 2010 at 12:09 PM, J. Bruce Fields <bfields@fieldses.org> wrote: > On Fri, Aug 27, 2010 at 06:08:25PM -0500, Peng Yu wrote: >> Hi, >> >> I use the following command to search for some directory. >> >> find -L some_dir -type d -name some_name >> >> If I'm on the machine that has some_dir locally, the run time is >> real 0m0.199s >> user 0m0.048s >> sys 0m0.140s >> >> If I'm on another machine that sees the same directory by NFS (NFS3). >> the run time is >> real 0m6.509s >> user 0m0.090s >> sys 0m1.380s >> >> There are 30 time speed difference. Is this normal? Is there any NFS >> parameter that I should tune to make the latter faster? > > How many directories are there in some_dir? What's the round trip time > (e.g. as reported by ping) to the server? If you repeate the find > immediately, is it faster? > > This can indeed by much slower if it requires revalidating the cached > attributes of each file. Please see below for the answers of you questions. $ find . -type d|wc 1491 1491 53372 Search the directory as a local directory $ time find . -name 'data.frame' -type d ./library/base/data.frame ./library/base/data.frame/data.frame real 0m0.044s user 0m0.012s sys 0m0.028s Search the directory as a NFS directory. The same command runs three 3 times in series. $ time find . -name 'data.frame' -type d ./library/base/data.frame ./library/base/data.frame/data.frame real 0m2.205s user 0m0.040s sys 0m0.430s $ time find . -name 'data.frame' -type d ./library/base/data.frame ./library/base/data.frame/data.frame real 0m1.203s user 0m0.060s sys 0m0.120s $ time find . -name 'data.frame' -type d ./library/base/data.frame ./library/base/data.frame/data.frame real 0m1.227s user 0m0.040s sys 0m0.200s Ping the NFS server gives me ttl=64 time=0.156 ms. Please let me know if there is more information needed. -- Regards, Peng ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 30 time speed difference between using NFS and without using NFS 2010-08-30 21:39 ` Peng Yu @ 2010-08-31 16:23 ` J. Bruce Fields 2010-08-31 21:27 ` Peng Yu 0 siblings, 1 reply; 6+ messages in thread From: J. Bruce Fields @ 2010-08-31 16:23 UTC (permalink / raw) To: Peng Yu; +Cc: linux-nfs On Mon, Aug 30, 2010 at 04:39:54PM -0500, Peng Yu wrote: > Please see below for the answers of you questions. > > $ find . -type d|wc > 1491 1491 53372 > > Search the directory as a local directory > > $ time find . -name 'data.frame' -type d > ./library/base/data.frame > ./library/base/data.frame/data.frame > > real 0m0.044s > user 0m0.012s > sys 0m0.028s > > Search the directory as a NFS directory. The same command runs three 3 > times in series. > > $ time find . -name 'data.frame' -type d > ./library/base/data.frame > ./library/base/data.frame/data.frame > > real 0m2.205s > user 0m0.040s > sys 0m0.430s > $ time find . -name 'data.frame' -type d > ./library/base/data.frame > ./library/base/data.frame/data.frame > > real 0m1.203s > user 0m0.060s > sys 0m0.120s > $ time find . -name 'data.frame' -type d > ./library/base/data.frame > ./library/base/data.frame/data.frame > > real 0m1.227s > user 0m0.040s > sys 0m0.200s > > Ping the NFS server gives me ttl=64 time=0.156 ms. So if readding each of those directories required only a single round trip, and if network round trip time were the dominating factor, the whole thing would only take a second. So there may be more rpc's (is find also stat'ing every directory entry?), and/or you may be bottlenecked by somethign else (e.g. seek time on the server). You might try using /proc/self/mounstats on the client to figure out what rpc's are sent during the find and the average time they take. --b. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 30 time speed difference between using NFS and without using NFS 2010-08-31 16:23 ` J. Bruce Fields @ 2010-08-31 21:27 ` Peng Yu 2010-09-01 1:15 ` J. Bruce Fields 0 siblings, 1 reply; 6+ messages in thread From: Peng Yu @ 2010-08-31 21:27 UTC (permalink / raw) To: J. Bruce Fields; +Cc: linux-nfs On Tue, Aug 31, 2010 at 11:23 AM, J. Bruce Fields <bfields@fieldses.org> wrote: > On Mon, Aug 30, 2010 at 04:39:54PM -0500, Peng Yu wrote: >> Please see below for the answers of you questions. >> >> $ find . -type d|wc >> 1491 1491 53372 >> >> Search the directory as a local directory >> >> $ time find . -name 'data.frame' -type d >> ./library/base/data.frame >> ./library/base/data.frame/data.frame >> >> real 0m0.044s >> user 0m0.012s >> sys 0m0.028s >> >> Search the directory as a NFS directory. The same command runs three 3 >> times in series. >> >> $ time find . -name 'data.frame' -type d >> ./library/base/data.frame >> ./library/base/data.frame/data.frame >> >> real 0m2.205s >> user 0m0.040s >> sys 0m0.430s >> $ time find . -name 'data.frame' -type d >> ./library/base/data.frame >> ./library/base/data.frame/data.frame >> >> real 0m1.203s >> user 0m0.060s >> sys 0m0.120s >> $ time find . -name 'data.frame' -type d >> ./library/base/data.frame >> ./library/base/data.frame/data.frame >> >> real 0m1.227s >> user 0m0.040s >> sys 0m0.200s >> >> Ping the NFS server gives me ttl=64 time=0.156 ms. > > So if readding each of those directories required only a single round > trip, and if network round trip time were the dominating factor, the > whole thing would only take a second. So there may be more rpc's (is > find also stat'ing every directory entry?), and/or you may be > bottlenecked by somethign else (e.g. seek time on the server). > > You might try using /proc/self/mounstats on the client to figure out > what rpc's are sent during the find and the average time they take. Hi, I don't really know how to read /proc/self/mounstats But here is the corresponding record in it on the client machine before and after I run the find command once. Are you able to tell what is wrong? #######before device xx.xx.x.xx:/pearson/data mounted on /pearson/data with fstype nfs statvers=1.0 opts: rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.4.20,mountvers=3,mountport=0,mountproto=tcp age: 167462 caps: caps=0x3fcf,wtmult=4096,dtsize=4096,bsize=0,namlen=255 sec: flavor=1,pseudoflavor=1 events: 97266 2937027 45 379 43366 10794 3030879 707 3 239 553 71267 53078 256 331 248 0 3941 0 1 209 0 0 0 0 bytes: 3620495167 2489192 0 0 29243366 1919994 7293 553 RPC iostats version: 1.0 p/v: 100003/3 (nfs) xprt: tcp 870 1 6 0 11 144892 144890 2 323286 727 per-op statistics NULL: 0 0 0 0 0 0 0 0 GETATTR: 97268 97268 0 17511024 10894016 300 27490 34980 SETATTR: 428 428 0 92916 61632 0 1000 1030 LOOKUP: 11128 11128 0 2158020 1509208 30 5010 5640 ACCESS: 28926 28926 0 5210656 3471000 110 8000 9950 ......... #######After device xx.xx.x.xx:/pearson/data mounted on /pearson/data with fstype nfs statvers=1.0 opts: rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.4.20,mountvers=3,mountport=0,mountproto=tcp age: 167478 caps: caps=0x3fcf,wtmult=4096,dtsize=4096,bsize=0,namlen=255 sec: flavor=1,pseudoflavor=1 events: 101739 2943313 45 379 46349 10794 3043133 707 3 239 553 71267 56060 256 331 248 0 3941 0 1 209 0 0 0 0 bytes: 3620495167 2489192 0 0 29243366 1919994 7293 553 RPC iostats version: 1.0 p/v: 100003/3 (nfs) xprt: tcp 870 1 6 0 3 150855 150853 2 341175 727 per-op statistics NULL: 0 0 0 0 0 0 0 0 GETATTR: 101741 101741 0 18298272 11394992 310 28740 36660 SETATTR: 428 428 0 92916 61632 0 1000 1030 LOOKUP: 11128 11128 0 2158020 1509208 30 5010 5640 ACCESS: 30416 30416 0 5478856 3649800 110 8290 10360 ....... -- Regards, Peng ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 30 time speed difference between using NFS and without using NFS 2010-08-31 21:27 ` Peng Yu @ 2010-09-01 1:15 ` J. Bruce Fields 0 siblings, 0 replies; 6+ messages in thread From: J. Bruce Fields @ 2010-09-01 1:15 UTC (permalink / raw) To: Peng Yu; +Cc: linux-nfs On Tue, Aug 31, 2010 at 04:27:35PM -0500, Peng Yu wrote: > On Tue, Aug 31, 2010 at 11:23 AM, J. Bruce Fields <bfields@fieldses.org> wrote: > > On Mon, Aug 30, 2010 at 04:39:54PM -0500, Peng Yu wrote: > >> Please see below for the answers of you questions. > >> > >> $ find . -type d|wc > >> 1491 1491 53372 > >> > >> Search the directory as a local directory > >> > >> $ time find . -name 'data.frame' -type d > >> ./library/base/data.frame > >> ./library/base/data.frame/data.frame > >> > >> real 0m0.044s > >> user 0m0.012s > >> sys 0m0.028s > >> > >> Search the directory as a NFS directory. The same command runs three 3 > >> times in series. > >> > >> $ time find . -name 'data.frame' -type d > >> ./library/base/data.frame > >> ./library/base/data.frame/data.frame > >> > >> real 0m2.205s > >> user 0m0.040s > >> sys 0m0.430s > >> $ time find . -name 'data.frame' -type d > >> ./library/base/data.frame > >> ./library/base/data.frame/data.frame > >> > >> real 0m1.203s > >> user 0m0.060s > >> sys 0m0.120s > >> $ time find . -name 'data.frame' -type d > >> ./library/base/data.frame > >> ./library/base/data.frame/data.frame > >> > >> real 0m1.227s > >> user 0m0.040s > >> sys 0m0.200s > >> > >> Ping the NFS server gives me ttl=64 time=0.156 ms. > > > > So if readding each of those directories required only a single round > > trip, and if network round trip time were the dominating factor, the > > whole thing would only take a second. So there may be more rpc's (is > > find also stat'ing every directory entry?), and/or you may be > > bottlenecked by somethign else (e.g. seek time on the server). > > > > You might try using /proc/self/mounstats on the client to figure out > > what rpc's are sent during the find and the average time they take. > > Hi, > > I don't really know how to read /proc/self/mounstats There's also a script in nfs-utils, under tools/nfs-iostat/nfs-iostat.py, that may help. > But here is the corresponding record in it on the client machine > before and after I run the find command once. Are you able to tell > what is wrong? There's not necessarily even anything wrong. NFS and local access tend to perform differently, so if the difference here is a problem then you need to figure out why and then decide what to do about it. --b. > > #######before > > device xx.xx.x.xx:/pearson/data mounted on /pearson/data with fstype > nfs statvers=1.0 > opts: rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.4.20,mountvers=3,mountport=0,mountproto=tcp > age: 167462 > caps: caps=0x3fcf,wtmult=4096,dtsize=4096,bsize=0,namlen=255 > sec: flavor=1,pseudoflavor=1 > events: 97266 2937027 45 379 43366 10794 3030879 707 3 239 553 71267 > 53078 256 331 248 0 3941 0 1 209 0 0 0 0 > bytes: 3620495167 2489192 0 0 29243366 1919994 7293 553 > RPC iostats version: 1.0 p/v: 100003/3 (nfs) > xprt: tcp 870 1 6 0 11 144892 144890 2 323286 727 > per-op statistics > NULL: 0 0 0 0 0 0 0 0 > GETATTR: 97268 97268 0 17511024 10894016 300 27490 34980 > SETATTR: 428 428 0 92916 61632 0 1000 1030 > LOOKUP: 11128 11128 0 2158020 1509208 30 5010 5640 > ACCESS: 28926 28926 0 5210656 3471000 110 8000 9950 > ......... > > > #######After > > device xx.xx.x.xx:/pearson/data mounted on /pearson/data with fstype > nfs statvers=1.0 > opts: rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.4.20,mountvers=3,mountport=0,mountproto=tcp > age: 167478 > caps: caps=0x3fcf,wtmult=4096,dtsize=4096,bsize=0,namlen=255 > sec: flavor=1,pseudoflavor=1 > events: 101739 2943313 45 379 46349 10794 3043133 707 3 239 553 71267 > 56060 256 331 248 0 3941 0 1 209 0 0 0 0 > bytes: 3620495167 2489192 0 0 29243366 1919994 7293 553 > RPC iostats version: 1.0 p/v: 100003/3 (nfs) > xprt: tcp 870 1 6 0 3 150855 150853 2 341175 727 > per-op statistics > NULL: 0 0 0 0 0 0 0 0 > GETATTR: 101741 101741 0 18298272 11394992 310 28740 36660 > SETATTR: 428 428 0 92916 61632 0 1000 1030 > LOOKUP: 11128 11128 0 2158020 1509208 30 5010 5640 > ACCESS: 30416 30416 0 5478856 3649800 110 8290 10360 > ....... > > > -- > Regards, > Peng ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2010-09-01 1:16 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-08-27 23:08 30 time speed difference between using NFS and without using NFS Peng Yu 2010-08-30 17:09 ` J. Bruce Fields 2010-08-30 21:39 ` Peng Yu 2010-08-31 16:23 ` J. Bruce Fields 2010-08-31 21:27 ` Peng Yu 2010-09-01 1:15 ` J. Bruce Fields
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox