* 30 time speed difference between using NFS and without using NFS
@ 2010-08-27 23:08 Peng Yu
2010-08-30 17:09 ` J. Bruce Fields
0 siblings, 1 reply; 6+ messages in thread
From: Peng Yu @ 2010-08-27 23:08 UTC (permalink / raw)
To: linux-nfs
Hi,
I use the following command to search for some directory.
find -L some_dir -type d -name some_name
If I'm on the machine that has some_dir locally, the run time is
real 0m0.199s
user 0m0.048s
sys 0m0.140s
If I'm on another machine that sees the same directory by NFS (NFS3).
the run time is
real 0m6.509s
user 0m0.090s
sys 0m1.380s
There are 30 time speed difference. Is this normal? Is there any NFS
parameter that I should tune to make the latter faster?
--
Regards,
Peng
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 30 time speed difference between using NFS and without using NFS
2010-08-27 23:08 30 time speed difference between using NFS and without using NFS Peng Yu
@ 2010-08-30 17:09 ` J. Bruce Fields
2010-08-30 21:39 ` Peng Yu
0 siblings, 1 reply; 6+ messages in thread
From: J. Bruce Fields @ 2010-08-30 17:09 UTC (permalink / raw)
To: Peng Yu; +Cc: linux-nfs
On Fri, Aug 27, 2010 at 06:08:25PM -0500, Peng Yu wrote:
> Hi,
>
> I use the following command to search for some directory.
>
> find -L some_dir -type d -name some_name
>
> If I'm on the machine that has some_dir locally, the run time is
> real 0m0.199s
> user 0m0.048s
> sys 0m0.140s
>
> If I'm on another machine that sees the same directory by NFS (NFS3).
> the run time is
> real 0m6.509s
> user 0m0.090s
> sys 0m1.380s
>
> There are 30 time speed difference. Is this normal? Is there any NFS
> parameter that I should tune to make the latter faster?
How many directories are there in some_dir? What's the round trip time
(e.g. as reported by ping) to the server? If you repeate the find
immediately, is it faster?
This can indeed by much slower if it requires revalidating the cached
attributes of each file.
--b.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 30 time speed difference between using NFS and without using NFS
2010-08-30 17:09 ` J. Bruce Fields
@ 2010-08-30 21:39 ` Peng Yu
2010-08-31 16:23 ` J. Bruce Fields
0 siblings, 1 reply; 6+ messages in thread
From: Peng Yu @ 2010-08-30 21:39 UTC (permalink / raw)
To: J. Bruce Fields; +Cc: linux-nfs
On Mon, Aug 30, 2010 at 12:09 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> On Fri, Aug 27, 2010 at 06:08:25PM -0500, Peng Yu wrote:
>> Hi,
>>
>> I use the following command to search for some directory.
>>
>> find -L some_dir -type d -name some_name
>>
>> If I'm on the machine that has some_dir locally, the run time is
>> real 0m0.199s
>> user 0m0.048s
>> sys 0m0.140s
>>
>> If I'm on another machine that sees the same directory by NFS (NFS3).
>> the run time is
>> real 0m6.509s
>> user 0m0.090s
>> sys 0m1.380s
>>
>> There are 30 time speed difference. Is this normal? Is there any NFS
>> parameter that I should tune to make the latter faster?
>
> How many directories are there in some_dir? What's the round trip time
> (e.g. as reported by ping) to the server? If you repeate the find
> immediately, is it faster?
>
> This can indeed by much slower if it requires revalidating the cached
> attributes of each file.
Please see below for the answers of you questions.
$ find . -type d|wc
1491 1491 53372
Search the directory as a local directory
$ time find . -name 'data.frame' -type d
./library/base/data.frame
./library/base/data.frame/data.frame
real 0m0.044s
user 0m0.012s
sys 0m0.028s
Search the directory as a NFS directory. The same command runs three 3
times in series.
$ time find . -name 'data.frame' -type d
./library/base/data.frame
./library/base/data.frame/data.frame
real 0m2.205s
user 0m0.040s
sys 0m0.430s
$ time find . -name 'data.frame' -type d
./library/base/data.frame
./library/base/data.frame/data.frame
real 0m1.203s
user 0m0.060s
sys 0m0.120s
$ time find . -name 'data.frame' -type d
./library/base/data.frame
./library/base/data.frame/data.frame
real 0m1.227s
user 0m0.040s
sys 0m0.200s
Ping the NFS server gives me ttl=64 time=0.156 ms.
Please let me know if there is more information needed.
--
Regards,
Peng
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 30 time speed difference between using NFS and without using NFS
2010-08-30 21:39 ` Peng Yu
@ 2010-08-31 16:23 ` J. Bruce Fields
2010-08-31 21:27 ` Peng Yu
0 siblings, 1 reply; 6+ messages in thread
From: J. Bruce Fields @ 2010-08-31 16:23 UTC (permalink / raw)
To: Peng Yu; +Cc: linux-nfs
On Mon, Aug 30, 2010 at 04:39:54PM -0500, Peng Yu wrote:
> Please see below for the answers of you questions.
>
> $ find . -type d|wc
> 1491 1491 53372
>
> Search the directory as a local directory
>
> $ time find . -name 'data.frame' -type d
> ./library/base/data.frame
> ./library/base/data.frame/data.frame
>
> real 0m0.044s
> user 0m0.012s
> sys 0m0.028s
>
> Search the directory as a NFS directory. The same command runs three 3
> times in series.
>
> $ time find . -name 'data.frame' -type d
> ./library/base/data.frame
> ./library/base/data.frame/data.frame
>
> real 0m2.205s
> user 0m0.040s
> sys 0m0.430s
> $ time find . -name 'data.frame' -type d
> ./library/base/data.frame
> ./library/base/data.frame/data.frame
>
> real 0m1.203s
> user 0m0.060s
> sys 0m0.120s
> $ time find . -name 'data.frame' -type d
> ./library/base/data.frame
> ./library/base/data.frame/data.frame
>
> real 0m1.227s
> user 0m0.040s
> sys 0m0.200s
>
> Ping the NFS server gives me ttl=64 time=0.156 ms.
So if readding each of those directories required only a single round
trip, and if network round trip time were the dominating factor, the
whole thing would only take a second. So there may be more rpc's (is
find also stat'ing every directory entry?), and/or you may be
bottlenecked by somethign else (e.g. seek time on the server).
You might try using /proc/self/mounstats on the client to figure out
what rpc's are sent during the find and the average time they take.
--b.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 30 time speed difference between using NFS and without using NFS
2010-08-31 16:23 ` J. Bruce Fields
@ 2010-08-31 21:27 ` Peng Yu
2010-09-01 1:15 ` J. Bruce Fields
0 siblings, 1 reply; 6+ messages in thread
From: Peng Yu @ 2010-08-31 21:27 UTC (permalink / raw)
To: J. Bruce Fields; +Cc: linux-nfs
On Tue, Aug 31, 2010 at 11:23 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
> On Mon, Aug 30, 2010 at 04:39:54PM -0500, Peng Yu wrote:
>> Please see below for the answers of you questions.
>>
>> $ find . -type d|wc
>> 1491 1491 53372
>>
>> Search the directory as a local directory
>>
>> $ time find . -name 'data.frame' -type d
>> ./library/base/data.frame
>> ./library/base/data.frame/data.frame
>>
>> real 0m0.044s
>> user 0m0.012s
>> sys 0m0.028s
>>
>> Search the directory as a NFS directory. The same command runs three 3
>> times in series.
>>
>> $ time find . -name 'data.frame' -type d
>> ./library/base/data.frame
>> ./library/base/data.frame/data.frame
>>
>> real 0m2.205s
>> user 0m0.040s
>> sys 0m0.430s
>> $ time find . -name 'data.frame' -type d
>> ./library/base/data.frame
>> ./library/base/data.frame/data.frame
>>
>> real 0m1.203s
>> user 0m0.060s
>> sys 0m0.120s
>> $ time find . -name 'data.frame' -type d
>> ./library/base/data.frame
>> ./library/base/data.frame/data.frame
>>
>> real 0m1.227s
>> user 0m0.040s
>> sys 0m0.200s
>>
>> Ping the NFS server gives me ttl=64 time=0.156 ms.
>
> So if readding each of those directories required only a single round
> trip, and if network round trip time were the dominating factor, the
> whole thing would only take a second. So there may be more rpc's (is
> find also stat'ing every directory entry?), and/or you may be
> bottlenecked by somethign else (e.g. seek time on the server).
>
> You might try using /proc/self/mounstats on the client to figure out
> what rpc's are sent during the find and the average time they take.
Hi,
I don't really know how to read /proc/self/mounstats
But here is the corresponding record in it on the client machine
before and after I run the find command once. Are you able to tell
what is wrong?
#######before
device xx.xx.x.xx:/pearson/data mounted on /pearson/data with fstype
nfs statvers=1.0
opts: rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.4.20,mountvers=3,mountport=0,mountproto=tcp
age: 167462
caps: caps=0x3fcf,wtmult=4096,dtsize=4096,bsize=0,namlen=255
sec: flavor=1,pseudoflavor=1
events: 97266 2937027 45 379 43366 10794 3030879 707 3 239 553 71267
53078 256 331 248 0 3941 0 1 209 0 0 0 0
bytes: 3620495167 2489192 0 0 29243366 1919994 7293 553
RPC iostats version: 1.0 p/v: 100003/3 (nfs)
xprt: tcp 870 1 6 0 11 144892 144890 2 323286 727
per-op statistics
NULL: 0 0 0 0 0 0 0 0
GETATTR: 97268 97268 0 17511024 10894016 300 27490 34980
SETATTR: 428 428 0 92916 61632 0 1000 1030
LOOKUP: 11128 11128 0 2158020 1509208 30 5010 5640
ACCESS: 28926 28926 0 5210656 3471000 110 8000 9950
.........
#######After
device xx.xx.x.xx:/pearson/data mounted on /pearson/data with fstype
nfs statvers=1.0
opts: rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.4.20,mountvers=3,mountport=0,mountproto=tcp
age: 167478
caps: caps=0x3fcf,wtmult=4096,dtsize=4096,bsize=0,namlen=255
sec: flavor=1,pseudoflavor=1
events: 101739 2943313 45 379 46349 10794 3043133 707 3 239 553 71267
56060 256 331 248 0 3941 0 1 209 0 0 0 0
bytes: 3620495167 2489192 0 0 29243366 1919994 7293 553
RPC iostats version: 1.0 p/v: 100003/3 (nfs)
xprt: tcp 870 1 6 0 3 150855 150853 2 341175 727
per-op statistics
NULL: 0 0 0 0 0 0 0 0
GETATTR: 101741 101741 0 18298272 11394992 310 28740 36660
SETATTR: 428 428 0 92916 61632 0 1000 1030
LOOKUP: 11128 11128 0 2158020 1509208 30 5010 5640
ACCESS: 30416 30416 0 5478856 3649800 110 8290 10360
.......
--
Regards,
Peng
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: 30 time speed difference between using NFS and without using NFS
2010-08-31 21:27 ` Peng Yu
@ 2010-09-01 1:15 ` J. Bruce Fields
0 siblings, 0 replies; 6+ messages in thread
From: J. Bruce Fields @ 2010-09-01 1:15 UTC (permalink / raw)
To: Peng Yu; +Cc: linux-nfs
On Tue, Aug 31, 2010 at 04:27:35PM -0500, Peng Yu wrote:
> On Tue, Aug 31, 2010 at 11:23 AM, J. Bruce Fields <bfields@fieldses.org> wrote:
> > On Mon, Aug 30, 2010 at 04:39:54PM -0500, Peng Yu wrote:
> >> Please see below for the answers of you questions.
> >>
> >> $ find . -type d|wc
> >> 1491 1491 53372
> >>
> >> Search the directory as a local directory
> >>
> >> $ time find . -name 'data.frame' -type d
> >> ./library/base/data.frame
> >> ./library/base/data.frame/data.frame
> >>
> >> real 0m0.044s
> >> user 0m0.012s
> >> sys 0m0.028s
> >>
> >> Search the directory as a NFS directory. The same command runs three 3
> >> times in series.
> >>
> >> $ time find . -name 'data.frame' -type d
> >> ./library/base/data.frame
> >> ./library/base/data.frame/data.frame
> >>
> >> real 0m2.205s
> >> user 0m0.040s
> >> sys 0m0.430s
> >> $ time find . -name 'data.frame' -type d
> >> ./library/base/data.frame
> >> ./library/base/data.frame/data.frame
> >>
> >> real 0m1.203s
> >> user 0m0.060s
> >> sys 0m0.120s
> >> $ time find . -name 'data.frame' -type d
> >> ./library/base/data.frame
> >> ./library/base/data.frame/data.frame
> >>
> >> real 0m1.227s
> >> user 0m0.040s
> >> sys 0m0.200s
> >>
> >> Ping the NFS server gives me ttl=64 time=0.156 ms.
> >
> > So if readding each of those directories required only a single round
> > trip, and if network round trip time were the dominating factor, the
> > whole thing would only take a second. So there may be more rpc's (is
> > find also stat'ing every directory entry?), and/or you may be
> > bottlenecked by somethign else (e.g. seek time on the server).
> >
> > You might try using /proc/self/mounstats on the client to figure out
> > what rpc's are sent during the find and the average time they take.
>
> Hi,
>
> I don't really know how to read /proc/self/mounstats
There's also a script in nfs-utils, under
tools/nfs-iostat/nfs-iostat.py, that may help.
> But here is the corresponding record in it on the client machine
> before and after I run the find command once. Are you able to tell
> what is wrong?
There's not necessarily even anything wrong. NFS and local access tend
to perform differently, so if the difference here is a problem then you
need to figure out why and then decide what to do about it.
--b.
>
> #######before
>
> device xx.xx.x.xx:/pearson/data mounted on /pearson/data with fstype
> nfs statvers=1.0
> opts: rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.4.20,mountvers=3,mountport=0,mountproto=tcp
> age: 167462
> caps: caps=0x3fcf,wtmult=4096,dtsize=4096,bsize=0,namlen=255
> sec: flavor=1,pseudoflavor=1
> events: 97266 2937027 45 379 43366 10794 3030879 707 3 239 553 71267
> 53078 256 331 248 0 3941 0 1 209 0 0 0 0
> bytes: 3620495167 2489192 0 0 29243366 1919994 7293 553
> RPC iostats version: 1.0 p/v: 100003/3 (nfs)
> xprt: tcp 870 1 6 0 11 144892 144890 2 323286 727
> per-op statistics
> NULL: 0 0 0 0 0 0 0 0
> GETATTR: 97268 97268 0 17511024 10894016 300 27490 34980
> SETATTR: 428 428 0 92916 61632 0 1000 1030
> LOOKUP: 11128 11128 0 2158020 1509208 30 5010 5640
> ACCESS: 28926 28926 0 5210656 3471000 110 8000 9950
> .........
>
>
> #######After
>
> device xx.xx.x.xx:/pearson/data mounted on /pearson/data with fstype
> nfs statvers=1.0
> opts: rw,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.4.20,mountvers=3,mountport=0,mountproto=tcp
> age: 167478
> caps: caps=0x3fcf,wtmult=4096,dtsize=4096,bsize=0,namlen=255
> sec: flavor=1,pseudoflavor=1
> events: 101739 2943313 45 379 46349 10794 3043133 707 3 239 553 71267
> 56060 256 331 248 0 3941 0 1 209 0 0 0 0
> bytes: 3620495167 2489192 0 0 29243366 1919994 7293 553
> RPC iostats version: 1.0 p/v: 100003/3 (nfs)
> xprt: tcp 870 1 6 0 3 150855 150853 2 341175 727
> per-op statistics
> NULL: 0 0 0 0 0 0 0 0
> GETATTR: 101741 101741 0 18298272 11394992 310 28740 36660
> SETATTR: 428 428 0 92916 61632 0 1000 1030
> LOOKUP: 11128 11128 0 2158020 1509208 30 5010 5640
> ACCESS: 30416 30416 0 5478856 3649800 110 8290 10360
> .......
>
>
> --
> Regards,
> Peng
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2010-09-01 1:16 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-08-27 23:08 30 time speed difference between using NFS and without using NFS Peng Yu
2010-08-30 17:09 ` J. Bruce Fields
2010-08-30 21:39 ` Peng Yu
2010-08-31 16:23 ` J. Bruce Fields
2010-08-31 21:27 ` Peng Yu
2010-09-01 1:15 ` J. Bruce Fields
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox