2.4 vs. 2.6 nfs client performance

2.4 vs. 2.6 nfs client performance

[Jason Holmes]

Hi,

I'm running some NFS performance tests to determine the best way to go
to configure a few fileservers for some linux clusters I run.  Right now
I'm getting together a suite of test programs representative of the
typical applications we see run on our clusters (scientific applications
such as Ansys or Abaqus, custom MPI code, etc.) for use as a benchmark
suite that I can disperse across 16 machines or so to create a decent
load on the fileservers.  I've just got started with a program called
Gaussian03 (molecular modelling code that does a lot of I/O) and I'm
already seeing some odd performance differences between the 2.4.22
client and the 2.6.0-test9-mm1 client (both against a 2.6.0-test9-mm2
server):

async mounts
------------
2.4.22:           110.87user 43.69system 4:04.58elapsed 63%CPU
2.6.0-test9-mm1:  111.88user 315.57system 9:51.87elapsed 72%CPU

sync mounts
-----------
2.4.22:           109.99user 45.49system 32:04.44elapsed 8%CPU
2.6.0-test9-mm1:  112.33user 197.76system 1:08:13elapsed 7%CPU

Note that the 1:08:13 in the sync 2.6.0-test9-mm1 is 1 *hour*, 8
minutes, not 1 *minute*.  In both cases the 2.6 client came in at about
twice the time.  A local run not using NFS finishes in 2:13.49.

The nfsstats output for two async runs looks like:

-- 2.4.22 --
Client nfs v3:
null       getattr    setattr    lookup     access     readlink
0       0% 880     0% 0       0% 160     0% 4745    0% 0       0%
read       write      create     mkdir      symlink    mknod
780     0% 178588 32% 24      0% 0       0% 0       0% 0       0%
remove     rmdir      rename     link       readdir    readdirplus
24      0% 0       0% 0       0% 0       0% 18      0% 0       0%
fsstat     fsinfo     pathconf   commit
3       0% 3       0% 0       0% 369185 66%

Client rpc stats:
calls      retrans    authrefrsh
554410     2099       0


-- 2.6.0-test9-mm1 --

Client nfs v3:
null       getattr    setattr    lookup     access     readlink
0       0% 740     0% 0       0% 126     0% 249     0% 0       0%
read       write      create     mkdir      symlink    mknod
855     0% 178574 14% 24      0% 0       0% 0       0% 0       0%
remove     rmdir      rename     link       readdir    readdirplus
24      0% 0       0% 0       0% 0       0% 18      0% 0       0%
fsstat     fsinfo     pathconf   commit
0       0% 3       0% 0       0% 1023777 85%

Client rpc stats:
calls      retrans    authrefrsh
1204390    5          0

The 2.6 client has over twice the number of RPC calls and almost 3 times
the number of commits whereas the 2.4 client has alot more accesses. 
Mounts were done with rsize=32768,wsize=32768.  The NFS filesystem is
ext3.  The two machines are connected via gigabit ethernet and the
traffic between them never goes above 20-30 MB/s.

Can someone clue me in as to why this may be happening and if it's a
"bug" or not?

Thanks,

--
Jason Holmes


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Trond Myklebust]

>>>>> " " == Jason Holmes <jholmes@psu.edu> writes:

     > Can someone clue me in as to why this may be happening and if
     > it's a "bug" or not?

Large numbers of 'commit' operations are a typical sign of the memory
management going wild. Have you compared this to the standard
2.6.0-test9 kernel?

What sort of load did you otherwise have on that machine?

Cheers,
 Trond


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Jason Holmes]

Trond Myklebust wrote:
> 
> >>>>> " " == Jason Holmes <jholmes@psu.edu> writes:
> 
>      > Can someone clue me in as to why this may be happening and if
>      > it's a "bug" or not?
> 
> Large numbers of 'commit' operations are a typical sign of the memory
> management going wild. Have you compared this to the standard
> 2.6.0-test9 kernel?
> 
> What sort of load did you otherwise have on that machine?

The machines (both client and server) are idle other than the testing
I'm doing and I'm doing only one test at a time at this point.

I did not do plain 2.6.0-test9 yet, but here is 2.6.0-test6 and
2.6.0-test9-bk17:

-- 2.6.0-test6 --

Client nfs v3:
null       getattr    setattr    lookup     access     readlink
0       0% 749     0% 0       0% 130     0% 251     0% 0       0%
read       write      create     mkdir      symlink    mknod
833     0% 178577 14% 24      0% 0       0% 0       0% 0       0%
remove     rmdir      rename     link       readdir    readdirplus
24      0% 0       0% 0       0% 0       0% 18      0% 1       0%
fsstat     fsinfo     pathconf   commit
0       0% 3       0% 0       0% 1024801 85%

Client rpc stats:
calls      retrans    authrefrsh
1205411    7          0

-- 2.6.0-test9-bk17 --

Client nfs v3:
null       getattr    setattr    lookup     access     readlink  
0       0% 744     0% 0       0% 129     0% 252     0% 0       0%
read       write      create     mkdir      symlink    mknod     
873     0% 178574 14% 24      0% 0       0% 0       0% 0       0% 
remove     rmdir      rename     link       readdir    readdirplus
24      0% 0       0% 0       0% 0       0% 18      0% 0       0%
fsstat     fsinfo     pathconf   commit     
0       0% 3       0% 0       0% 1023451 84%

Client rpc stats:
calls      retrans    authrefrsh
1204092    3          0

Thanks,

--
Jason Holmes


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Trond Myklebust]

>>>>> " " == Jason Holmes <jholmes@psu.edu> writes:

     > Client nfs v3: null getattr setattr lookup access readlink 0 0%
     > 744 0% 0 0% 129 0% 252 0% 0 0% read write create mkdir symlink
     > mknod 873 0% 178574 14% 24 0% 0 0% 0 0% 0 0% remove rmdir
     > rename link readdir readdirplus 24 0% 0 0% 0 0% 0 0% 18 0% 0 0%
     > fsstat fsinfo pathconf commit 0 0% 3 0% 0 0% 1023451 84%

     > Client rpc stats: calls retrans authrefrsh 1204092 3 0

Something is seriously screwed up here: it should be impossible to get
more commit requests than there are write requests.

Basically, commit is only sent if at least one unstable write has been
sent, and is waiting on the commit list. The commit list itself is
emptied before the RPC call gets sent. If the commit fails, it
resends all pending writes.

IOW: it looks like you are seeing some form of list corruption. I
suspect that the problem here might be something scribbling over the
nfs_inode. Are you running with slab debugging, and stack overflow
enabled?

Cheers,
  Trond


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Jason Holmes]

Trond Myklebust wrote:
> 
> >>>>> " " == Jason Holmes <jholmes@psu.edu> writes:
> 
>      > Client nfs v3: null getattr setattr lookup access readlink 0 0%
>      > 744 0% 0 0% 129 0% 252 0% 0 0% read write create mkdir symlink
>      > mknod 873 0% 178574 14% 24 0% 0 0% 0 0% 0 0% remove rmdir
>      > rename link readdir readdirplus 24 0% 0 0% 0 0% 0 0% 18 0% 0 0%
>      > fsstat fsinfo pathconf commit 0 0% 3 0% 0 0% 1023451 84%
> 
>      > Client rpc stats: calls retrans authrefrsh 1204092 3 0
> 
> Something is seriously screwed up here: it should be impossible to get
> more commit requests than there are write requests.
> 
> Basically, commit is only sent if at least one unstable write has been
> sent, and is waiting on the commit list. The commit list itself is
> emptied before the RPC call gets sent. If the commit fails, it
> resends all pending writes.
> 
> IOW: it looks like you are seeing some form of list corruption. I
> suspect that the problem here might be something scribbling over the
> nfs_inode. Are you running with slab debugging, and stack overflow
> enabled?

I wasn't, but I'm recompiling test9-bk17 with all of the debugging
options turned on right now.  Is there anything in particular you want
me to do?

Thanks,

--
Jason Holmes


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Eric Whiting]

Jason Holmes wrote:
>
<snip>
> I'm
> already seeing some odd performance differences between the 2.4.22
> client and the 2.6.0-test9-mm1 client (both against a 2.6.0-test9-mm2
> server):

Similar observations here. Our users have reported NFS slowdowns of 10x on
2.6.0-test6-mm4 nfs clients compared to the same box and setup running 2.4.22. 
Something is goofy in 2.6.0-test6-mm4 that has a bad effect on NFS client
performance.

eric


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Trond Myklebust]

>>>>> " " == Jason Holmes <jholmes@psu.edu> writes:

     > I wasn't, but I'm recompiling test9-bk17 with all of the
     > debugging options turned on right now.  Is there anything in
     > particular you want me to do?

Come to think of it, could you try running with and without the patch
on

  http://www.fys.uio.no/~trondmy/src/Linux-2.6.x/2.6.0-test9/linux-2.6.0-01-fix_deadlock.dif

That fixes a race that is likely to show up when memory is
low. Normally it should cause a deadlock rather than corruption, but
it is conceivable that there are other side-effects...

Please make sure that you run with slab debugging enabled...

Cheers,
  Trond


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Jason Holmes]

Trond Myklebust wrote:
> 
> >>>>> " " == Jason Holmes <jholmes@psu.edu> writes:
> 
>      > I wasn't, but I'm recompiling test9-bk17 with all of the
>      > debugging options turned on right now.  Is there anything in
>      > particular you want me to do?
> 
> Come to think of it, could you try running with and without the patch
> on
> 
>   http://www.fys.uio.no/~trondmy/src/Linux-2.6.x/2.6.0-test9/linux-2.6.0-01-fix_deadlock.dif
> 
> That fixes a race that is likely to show up when memory is
> low. Normally it should cause a deadlock rather than corruption, but
> it is conceivable that there are other side-effects...
> 
> Please make sure that you run with slab debugging enabled...

With all debugging enabled, test9-bk17:

patched:   112.21user 259.86system 10:53.48elapsed 56%CPU
unpatched: 112.17user 252.35system 10:37.43elapsed 57%CPU

-- patched --

Client nfs v3:
null       getattr    setattr    lookup     access     readlink   
0       0% 752     0% 0       0% 134     0% 260     0% 0       0% 
read       write      create     mkdir      symlink    mknod      
873     0% 178571 14% 24      0% 0       0% 0       0% 0       0% 
remove     rmdir      rename     link       readdir    readdirplus
24      0% 0       0% 0       0% 0       0% 18      0% 0       0% 
fsstat     fsinfo     pathconf   commit     
0       0% 3       0% 0       0% 1044537 85%

Client rpc stats:
calls      retrans    authrefrsh
1225196    8          0

-- unpatched -- 

Client nfs v3:
null       getattr    setattr    lookup     access     readlink   
0       0% 753     0% 0       0% 134     0% 260     0% 0       0% 
read       write      create     mkdir      symlink    mknod      
873     0% 178575 14% 24      0% 0       0% 0       0% 0       0% 
remove     rmdir      rename     link       readdir    readdirplus
24      0% 0       0% 0       0% 0       0% 18      0% 0       0% 
fsstat     fsinfo     pathconf   commit     
0       0% 3       0% 0       0% 1050417 85%

Client rpc stats:
calls      retrans    authrefrsh
1231081    5          0

I didn't see anything to indicate that any debug code had triggered (I'm
assuming that if it did the kernel would give an OOPS?)

FWIW, I ran the test with nfsvers=2 (with the export still being async)
and it finished in 112.53user 45.02system 2:43.50elapsed 96%CPU.

Thanks,

--
Jason Holmes


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Trond Myklebust]

>>>>> " " == Jason Holmes <jholmes@psu.edu> writes:

     > FWIW, I ran the test with nfsvers=2 (with the export still
     > being async) and it finished in 112.53user 45.02system
     > 2:43.50elapsed 96%CPU.

For comparison, on my own machine, I get stats of the form:

Client rpc stats:
calls      retrans    authrefrsh
248291     6225       0
Client nfs v2:
null       getattr    setattr    root       lookup     readlink
0       0% 0       0% 0       0% 0       0% 0       0% 0       0%
read       wrcache    write      create     remove     rename
0       0% 0       0% 0       0% 0       0% 0       0% 0       0%
link       symlink    mkdir      rmdir      readdir    fsstat
0       0% 0       0% 0       0% 0       0% 0       0% 0       0%
 
Client nfs v3:
null       getattr    setattr    lookup     access     readlink
0       0% 77      0% 0       0% 60      0% 94      0% 0       0%
read       write      create     mkdir      symlink    mknod
0       0% 240378 96% 8       0% 0       0% 0       0% 0       0%
remove     rmdir      rename     link       readdir    readdirplus
33      0% 0       0% 0       0% 0       0% 0       0% 12      0%
fsstat     fsinfo     pathconf   commit
0       0% 1       0% 0       0% 7628    3%


Any hints on how I can reproduce your results?

Cheers,
 Trond


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Jason Holmes]

Trond Myklebust wrote:
> 
> >>>>> " " == Jason Holmes <jholmes@psu.edu> writes:
> 
>      > FWIW, I ran the test with nfsvers=2 (with the export still
>      > being async) and it finished in 112.53user 45.02system
>      > 2:43.50elapsed 96%CPU.
> 
> For comparison, on my own machine, I get stats of the form:
> 
> Client rpc stats:
> calls      retrans    authrefrsh
> 248291     6225       0
> Client nfs v2:
> null       getattr    setattr    root       lookup     readlink
> 0       0% 0       0% 0       0% 0       0% 0       0% 0       0%
> read       wrcache    write      create     remove     rename
> 0       0% 0       0% 0       0% 0       0% 0       0% 0       0%
> link       symlink    mkdir      rmdir      readdir    fsstat
> 0       0% 0       0% 0       0% 0       0% 0       0% 0       0%
> 
> Client nfs v3:
> null       getattr    setattr    lookup     access     readlink
> 0       0% 77      0% 0       0% 60      0% 94      0% 0       0%
> read       write      create     mkdir      symlink    mknod
> 0       0% 240378 96% 8       0% 0       0% 0       0% 0       0%
> remove     rmdir      rename     link       readdir    readdirplus
> 33      0% 0       0% 0       0% 0       0% 0       0% 12      0%
> fsstat     fsinfo     pathconf   commit
> 0       0% 1       0% 0       0% 7628    3%
> 
> Any hints on how I can reproduce your results?

Unfortunately Gaussian's licensing would prohibit me from passing it
along to reproduce this.  I'll try to reproduce this with another
program this morning and get back to you.

Thanks,

--
Jason Holmes


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Eric Whiting]

Trond Myklebust wrote:
> Any hints on how I can reproduce your results?

Trond, 

This is not the same test, nor the same results as Jason's, but here is some
more data from bonnie and dd tests.This shows some write performance difference
between 2.4 and 2.6
eric

SERVER: 2.4.21

CLIENT: 2.4.19 nfsv3
Bonnie 
Writing with putc()...  done:   9132 kB/s  33.4 %CPU
Rewriting...            done:   8713 kB/s   4.2 %CPU
Writing intelligently...done:   9116 kB/s   4.6 %CPU

$ rm file;time dd if=/dev/zero of=file bs=1 count=10000000
10000000+0 records in
10000000+0 records out

real    0m30.404s
user    0m3.930s
sys     0m26.250s



CLIENT: 2.6.0-test6-mm4
Writing with putc()...  done:    806 kB/s   5.4 %CPU
Rewriting...            done:   3321 kB/s   3.9 %CPU
Writing intelligently...done:   2715 kB/s   2.4 %CPU


$ rm file;time dd if=/dev/zero of=file bs=1 count=10000000
10000000+0 records in
10000000+0 records out

real    1m37.848s
user    0m17.355s
sys     1m8.947s


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Trond Myklebust]

     > This is not the same test, nor the same results as Jason's, but
     > here is some more data from bonnie and dd tests.This shows some
     > write performance difference between 2.4 and 2.6 eric

How about vs. 2.4.23?

Cheers,
  Trond


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Eric Whiting]

Trond Myklebust wrote:
> 
>      > This is not the same test, nor the same results as Jason's, but
>      > here is some more data from bonnie and dd tests.This shows some
>      > write performance difference between 2.4 and 2.6 eric
> 
> How about vs. 2.4.23?

2.4.23 looks good -- see below -- these are acceptable numbers for a 100M
network. I reran the 2.6.0-test6-mm4 benchmarks and it still is much slower than
the the 2.4.23.

Question for Jason: -- are you running the 4/4 vm layout in your kernel? My
2.6.0-test6-mm4 has the 4/4 vm enabled -- which can have an impact on
performance. I'll tetest with 2.6.0-test9 without the mm1 and 4/4 vm enabled.

eric


Linux host 2.4.23-rc1 #2 SMP Thu Nov 13 11:30:02 MST 2003 i686 athlon i386
GNU/Linux
/test> bonnie -s 100
Writing with putc()...  done:   9228 kB/s  52.0 %CPU
Rewriting...            done:   9435 kB/s   6.0 %CPU
Writing intelligently...done:   9235 kB/s   4.0 %CPU

/test> rm file;time dd if=/dev/zero of=file bs=1 count=10000000
10000000+0 records in
10000000+0 records out

real    0m20.086s
user    0m3.120s
sys     0m16.740s


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Eric Whiting]

Eric Whiting wrote:
> 
> Trond Myklebust wrote:
> >
> >      > This is not the same test, nor the same results as Jason's, but
> >      > here is some more data from bonnie and dd tests.This shows some
> >      > write performance difference between 2.4 and 2.6 eric
> >
> > How about vs. 2.4.23?
>
> I'll tetest with 2.6.0-test9 without the mm1 and 4/4 vm enabled.
> 

2.6.0-test9 (without mm1 4/4 vm) is even worse than 2.6.0-test6-mm4.

tcpdump shows very slow writes..


12:07:12.380001 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs]
(frag 33858:1480@0+)
12:07:12.380012 172.16.17.176 > 172.16.32.181: udp (frag 33858:1480@1480+)
12:07:12.380014 172.16.17.176 > 172.16.32.181: udp (frag 33858:1284@2960)
12:07:17.979092 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs]
(frag 33859:1480@0+)
12:07:17.979102 172.16.17.176 > 172.16.32.181: udp (frag 33859:1480@1480+)
12:07:17.979104 172.16.17.176 > 172.16.32.181: udp (frag 33859:1284@2960)
12:07:18.678981 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs]
(frag 33860:1480@0+)
12:07:18.678992 172.16.17.176 > 172.16.32.181: udp (frag 33860:1480@1480+)
12:07:18.678994 172.16.17.176 > 172.16.32.181: udp (frag 33860:1284@2960)
12:07:20.078755 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs]
(frag 33861:1480@0+)
12:07:20.078766 172.16.17.176 > 172.16.32.181: udp (frag 33861:1480@1480+)
12:07:20.078768 172.16.17.176 > 172.16.32.181: udp (frag 33861:1284@2960)
12:07:22.879310 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs]
(frag 33862:1480@0+)
12:07:22.879321 172.16.17.176 > 172.16.32.181: udp (frag 33862:1480@1480+)
12:07:22.879323 172.16.17.176 > 172.16.32.181: udp (frag 33862:1284@2960)
12:07:28.479388 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs]
(frag 33863:1480@0+)
12:07:28.479399 172.16.17.176 > 172.16.32.181: udp (frag 33863:1480@1480+)
12:07:28.479400 172.16.17.176 > 172.16.32.181: udp (frag 33863:1284@2960)
12:07:29.179275 172.16.17.176.1525633085 > 172.16.32.181.2049: 1472 write [|nfs]
(frag 33864:1480@0+)
12:07:29.179286 172.16.17.176 > 172.16.32.181: udp (frag 33864:1480@1480+)


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Jason Holmes]

Eric Whiting wrote:
> 
> Trond Myklebust wrote:
> >
> >      > This is not the same test, nor the same results as Jason's, but
> >      > here is some more data from bonnie and dd tests.This shows some
> >      > write performance difference between 2.4 and 2.6 eric
> >
> > How about vs. 2.4.23?
> 
> 2.4.23 looks good -- see below -- these are acceptable numbers for a 100M
> network. I reran the 2.6.0-test6-mm4 benchmarks and it still is much slower than
> the the 2.4.23.
> 
> Question for Jason: -- are you running the 4/4 vm layout in your kernel? My
> 2.6.0-test6-mm4 has the 4/4 vm enabled -- which can have an impact on
> performance. I'll tetest with 2.6.0-test9 without the mm1 and 4/4 vm enabled.

Nope.  I'm just doing normal 4GB highmem.

Thanks,

--
Jason Holmes


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Trond Myklebust]

>>>>> " " == Eric Whiting <ewhiting@amis.com> writes:

     > 2.6.0-test9 (without mm1 4/4 vm) is even worse than
     > 2.6.0-test6-mm4.

     > tcpdump shows very slow writes..


Were you filtering away replies from the server, or were they really
missing? If the latter, then that points towards a problem in the
socket layer or possibly a problem with the device driver for your
NIC.

Cheers,
  Trond


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Duc Vianney]

> Trond Myklebust/Jason Holmes/Eric Whiting wrote ...

I ran the iozone in the cluster mode to time read and write under NFSv3.
Under a server memory constraint scenario, I saw reads on 2.6.0-test9 about
8X slower than 2.4.22.
For writes, I saw improvement in 2.6.0-test9 irrespective of server memory
size.

The following data represents the throughput in KB/sec as seen by the NFS
server subject to various memory sizes.

   Server Memory  512MB   512MB      2048MB   2048MB
   Kernel         2.4.22  260-test9  2.4.22   260-test9
   Write          14514   29134      13670    29330
   Rewrite        15140   25943      14683    27046
   Read           24404    3685      72165    78445
   ReRead         24702    3705      73267    78375

NFS Server: 4-way 700MHz, 2.5GB RAM, e1000 switch box, stock 2.6.0-test9.
NFS Client: 2-way, 2GHz, 1GB RAM, e1000, stock kernel 2.4.22, NFSv3 mounted
with default parameters
Benchmark: iozone running in cluster mode from the server with test file
size set to 1GB.

rpc data for 2422 and 260test9 at 512MB and 2048MB server memory:

             512MB    512MB    ratio            2048MB   2048MB    ratio
Kernel        2422    260t9  2422/260t9           2422   260t9   2422/260t

Server nfs v3 stats:
getattr         3       15     0.20                 3        3     1.00
fsstat          0        0                          0        0
fsinfo          0        0                          0        0
read       262992    66409     3.96            262992    65545     4.01
write      262162    65806     3.98            262162    65646     3.99
access         20       20     1.00                20       20     1.00
remove          2        1     2.00                 2        1     2.00
create          1        1     1.00                 1        1     1.00
commit       2056     2132     0.96              2056     2102     0.98
                         0
server rp                0
calls      527239   134383     3.92            527239   133345     3.95


Client nfs v3 stats:
getattr         3       15     0.20                 3        3     1.00
setattr         0        0                          0        0
lookup          3        2     1.50                 3        3     1.00
access         20       20     1.00                20       20     1.00
read       262992    65547     4.01            262992    65560     4.01
write      262145    65537     4.00            262145    65537     4.00
create          1        1     1.00                 1        1     1.00
remove          1        1     1.00                 1        1     1.00
readdir         0        0                          0        0
fsstat          0        0                          0        0
fsinfo          0        0                          0        0
commit       2055     2052     1.00              2055     2052     1.00

client rpc stats:
calls      527220   133175     3.96            527220   133177     3.96
retrans      7509     2242     3.35              7509      620    12.11


I did open a defect to describe the above read problem at:

http://bugme.osdl.org/show_bug.cgi?id=1539


Regards ...Duc



                                                                                                                
                      Eric Whiting                                                                              
                      <ewhiting@amis.com>         To:       trond.myklebust@fys.uio.no, Jason Holmes            
                      Sent by:                     <jholmes@psu.edu>                                            
                      nfs-admin@lists.sour        cc:       nfs@lists.sourceforge.net                           
                      ceforge.net                 Subject:  Re: [NFS] 2.4 vs. 2.6 nfs client performance        
                                                                                                                
                                                                                                                
                      11/13/2003 12:59 PM                                                                       
                                                                                                                
                                                                                                                




Trond Myklebust wrote:
>
>      > This is not the same test, nor the same results as Jason's, but
>      > here is some more data from bonnie and dd tests.This shows some
>      > write performance difference between 2.4 and 2.6 eric
>
> How about vs. 2.4.23?

2.4.23 looks good -- see below -- these are acceptable numbers for a 100M
network. I reran the 2.6.0-test6-mm4 benchmarks and it still is much slower
than
the the 2.4.23.

Question for Jason: -- are you running the 4/4 vm layout in your kernel? My
2.6.0-test6-mm4 has the 4/4 vm enabled -- which can have an impact on
performance. I'll tetest with 2.6.0-test9 without the mm1 and 4/4 vm
enabled.

eric


Linux host 2.4.23-rc1 #2 SMP Thu Nov 13 11:30:02 MST 2003 i686 athlon i386
GNU/Linux
/test> bonnie -s 100
Writing with putc()...  done:   9228 kB/s  52.0 %CPU
Rewriting...            done:   9435 kB/s   6.0 %CPU
Writing intelligently...done:   9235 kB/s   4.0 %CPU

/test> rm file;time dd if=/dev/zero of=file bs=1 count=10000000
10000000+0 records in
10000000+0 records out

real    0m20.086s
user    0m3.120s
sys     0m16.740s


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs






-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Trond Myklebust]

>>>>> " " == Duc Vianney <dvianney@us.ibm.com> writes:

     > I ran the iozone in the cluster mode to time read and write
     > under NFSv3.  Under a server memory constraint scenario, I saw
     > reads on 2.6.0-test9 about 8X slower than 2.4.22.  For writes,
     > I saw improvement in 2.6.0-test9 irrespective of server memory
     > size.

This is consistent with what I see on my setups.

The increase in write speed you see is due to the fact that we've
managed to lift the restriction we have in 2.4.x that only 256 writes
writes may be cached before have to we do another COMMIT and flush
them to disk.

The slowdown in reads is something I see too. What's more, both I and
others are also seeing slowdowns on local filesystem reads. I
therefore suspect some bug in the new VFS readahead code, but I still
haven't found time to look into this.

Cheers,
  Trond


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Jason Holmes]

Jason Holmes wrote:
> 
> Trond Myklebust wrote:
> >
> > Any hints on how I can reproduce your results?
> 
> Unfortunately Gaussian's licensing would prohibit me from passing it
> along to reproduce this.  I'll try to reproduce this with another
> program this morning and get back to you.

OK, I haven't found any other program that behaves similar to Gaussian
on this, but I did do a few more runs with 2.6.0-test9-bk17 vs 2.4.22,
one set with nfs_debug on and one set with rpc_debug on.  I haven't
analyzed the files in depth (and doing so would really be beyond me at
this point), but on a "dumb" level, the nfs files are of similar length
in terms of lines, but the RPC file for 2.6.0-test9-bk17 is much larger:

[root@fs4 log]# wc -l kern.*
  16236 kern.nfs-2.4.22
  17188 kern.nfs-2.6.0-test9-bk17
  20565 kern.rpc-2.4.22
  32684 kern.rpc-2.6.0-test9-bk17

Note that there is probably some small amount of noise in these files,
but 98% of it should be from the Gaussian runs.  I put them up at:

  http://magicbus.cac.psu.edu/nfs

if you want to look at them.  One small observation is that 2.4.22
reaches much larger values of cwnd than test9-bk17, resulting in a lot
fewer calls to functions such as xprt_cwnd_limited (1861 vs. 2715). 
Whether or not this means anything, I don't know.

Trond, if none of this helps and you really want to debug this, I can
probably give you access to two machines to do so.

Thanks,

--
Jason Holmes


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Trond Myklebust]

>>>>> " " == Jason Holmes <jholmes@psu.edu> writes:

     > if you want to look at them.  One small observation is that
     > 2.4.22 reaches much larger values of cwnd than test9-bk17,
     > resulting in a lot fewer calls to functions such as
     > xprt_cwnd_limited (1861 vs. 2715). Whether or not this means
     > anything, I don't know.

It means that the UDP throughput is slightly more stable. However the
thing that surprises me there is that in both cases 'xprt_cwnd_limited
cong = 0' is the dominant case. That means basically that you only
have one outstanding request on the wire at any point in time (there
are only a few cases where cong=256 => 2 requests, and cong=512 => 3
requests).

IOW this all appears to be doing synchronous I/O only (both in the
case of 2.4.22 and 2.6.0).
Note: most of these RPC calls do indeed appear to be something other
than WRITE calls. Are they perhaps GETATTR calls? Have you for
instance set 'noac', 'actimeo' or something like that?


Could you please try using

echo "9" >/proc/sys/sunrpc/nfs_debug

rather than echo "1". That should give us the debugging info from
nfs/read.c and nfs/write.c in addition to what you've already got.


Cheers,
  Trond


-------------------------------------------------------
This SF.Net email sponsored by: ApacheCon 2003,
16-19 November in Las Vegas. Learn firsthand the latest
developments in Apache, PHP, Perl, XML, Java, MySQL,
WebDAV, and more! http://www.apachecon.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Jason Holmes]

Trond Myklebust wrote:
> 
> IOW this all appears to be doing synchronous I/O only (both in the
> case of 2.4.22 and 2.6.0).
> Note: most of these RPC calls do indeed appear to be something other
> than WRITE calls. Are they perhaps GETATTR calls? Have you for
> instance set 'noac', 'actimeo' or something like that?

This was my mistake - I had changed the export to sync for a test right
before the debug runs and forgot to change it back.

> Could you please try using
> 
> echo "9" >/proc/sys/sunrpc/nfs_debug

Files should be up there (http://magicbus.cac.psu.edu/nfs).  I do have
to admit that something odd is going on - 2.4.22 still is faster than
2.6.0-test9-bk17 and test9-bk17 is still showing the large amount of
commits, but both of them are running faster than they did before on the
sync mounts and I'm not sure what I might have changed to cause this.

Thanks,

--
Jason Holmes


-------------------------------------------------------
This SF. Net email is sponsored by: GoToMyPC
GoToMyPC is the fast, easy and secure way to access your computer from
any Web browser or wireless device. Click here to Try it Free!
https://www.gotomypc.com/tr/OSDN/AW/Q4_2003/t/g22lp?Target=mm/g22lp.tmpl
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Trond Myklebust]

>>>>> " " == Jason Holmes <jholmes@psu.edu> writes:

     > Files should be up there (http://magicbus.cac.psu.edu/nfs).  I
     > do have to admit that something odd is going on - 2.4.22 still
     > is faster than 2.6.0-test9-bk17 and test9-bk17 is still showing
     > the large amount of commits, but both of them are running
     > faster than they did before on the sync mounts and I'm not sure
     > what I might have changed to cause this.

Hmm... It looks like you are doing a succession of 4k commits. I'll
bet the read() code is involved in this...

Could you try the following patch and see if it improves things?

Cheers,
  Trond

--- linux-2.6.0-test9/fs/nfs/write.c.orig	2003-11-13 18:39:34.000000000 -0500
+++ linux-2.6.0-test9/fs/nfs/write.c	2003-11-14 16:04:58.000000000 -0500
@@ -1103,7 +1103,7 @@
 			error = nfs_flush_file(inode, file, idx_start, npages, how);
 #if defined(CONFIG_NFS_V3) || defined(CONFIG_NFS_V4)
 		if (error == 0)
-			error = nfs_commit_file(inode, file, idx_start, npages, how);
+			error = nfs_commit_file(inode, file, 0, 0, how);
 #endif
 	} while (error > 0);
 	return error;


-------------------------------------------------------
This SF. Net email is sponsored by: GoToMyPC
GoToMyPC is the fast, easy and secure way to access your computer from
any Web browser or wireless device. Click here to Try it Free!
https://www.gotomypc.com/tr/OSDN/AW/Q4_2003/t/g22lp?Target=mm/g22lp.tmpl
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4 vs. 2.6 nfs client performance

[Jason Holmes]

Trond Myklebust wrote:
> 
> >>>>> " " == Jason Holmes <jholmes@psu.edu> writes:
> 
>      > Files should be up there (http://magicbus.cac.psu.edu/nfs).  I
>      > do have to admit that something odd is going on - 2.4.22 still
>      > is faster than 2.6.0-test9-bk17 and test9-bk17 is still showing
>      > the large amount of commits, but both of them are running
>      > faster than they did before on the sync mounts and I'm not sure
>      > what I might have changed to cause this.
> 
> Hmm... It looks like you are doing a succession of 4k commits. I'll
> bet the read() code is involved in this...
> 
> Could you try the following patch and see if it improves things?

That made a huge difference:

Client nfs v3:
null       getattr    setattr    lookup     access     readlink   
0       0% 714     0% 0       0% 110     0% 204     0% 0       0% 
read       write      create     mkdir      symlink    mknod      
873     0% 178571 92% 24      0% 0       0% 0       0% 0       0% 
remove     rmdir      rename     link       readdir    readdirplus
24      0% 0       0% 0       0% 0       0% 18      0% 0       0% 
fsstat     fsinfo     pathconf   commit     
0       0% 3       0% 0       0% 13175   6%

Client rpc stats:
calls      retrans    authrefrsh
193716     13         0 

It also cut the job time in about half.  I put a level9 log of it up
with the other ones.  I'll have to rerun the 2.4 vs. this, but I think
that this change makes the 2.6 client faster than the 2.4 for this case.

Thanks,

--
Jason Holmes


-------------------------------------------------------
This SF. Net email is sponsored by: GoToMyPC
GoToMyPC is the fast, easy and secure way to access your computer from
any Web browser or wireless device. Click here to Try it Free!
https://www.gotomypc.com/tr/OSDN/AW/Q4_2003/t/g22lp?Target=mm/g22lp.tmpl
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs