Poor NFS performance, kernel 2.6.6.

Poor NFS performance, kernel 2.6.6.

[Harald Hannelius]

Hi there,

I'm running two servers, both Dual Opterons with Broadcom Gb NICs. I'm
using the bcm5700 driver and I see no errors whatsoever with 'ifconfig'.

I'm using nfs-utils-1.0.6 and the 2.6.6 kernel, compiled with both NFSv3
and NFSv4 support. The client and servers are identical except for the
server having a scsi-raid under /home and the client being and IDE-box.

Distribution slackware-9.1 on both computers.

/etc/exports;

   /home   193.167.32.175(ro,no_root_squash,async)

/etc/fstab;

  193.167.32.187:/home    /home   nfs     \
 defaults,noauto,rsize=32768,wsize=32768,nfsvers=3 0 0

(as you can see I have experimented with different rsize,wsize without any
noticeable effect, same goes for nfsvers 2 and 3. I even mounted the
filesystem as ext2 on the server and as ext3 with data=journal with no
effect)

The computers are connected through a HP 8-port Gb switch and are on the
same subnet.

rsync over ssh gives me roughly 200Mbps (37 GB dataset)
netcat over tcp with a 2GB file gives me 457 Mbps
netcat over udp with a 2GB file gives me 640 Mbps

But dd'ing the 2GB file over nfs as some NFS-HOWTO suggests takes over 5
minutes. That should equal to something around 49 Mbps, correct?

The netcat takes 25 secs to transfer 2GB file.

Is there some way I can look at the nfs-server what it's doing? Any
suggestions on solutions for this?

And what datarates over NFS could I expect on a setup like this?
Half-wirespeed maybe?

Thanks in advance,

  Harald


-- 
A: Top Posters
Q: What is the most annoying thing on mailing lists?

Harald H Hannelius | harald/a\arcada.fi      | GSM +358 50 594 1020


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Poor NFS performance, kernel 2.6.6.

[Harald Hannelius]

On Thu, 27 May 2004, Vincent ROQUETA wrote:

> Le Jeudi 27 Mai 2004 12:31, Harald Hannelius a =E9crit :

> > Is there some way I can look at the nfs-server what it's doing? Any
> > suggestions on solutions for this?
> >
> > And what datarates over NFS could I expect on a setup like this?
> > Half-wirespeed maybe?

> on NFSv3, 1 ethernet 1GB I reached  89% of the avalaible brandwith for NF=
Sv3,
> kernel 2.6.3, for file less than the total amount of ram (4Go), and 22% o=
f
> the brandwith for file over 6Go, because of the use of harddisk use in th=
e
> copy process (swap).
>
> What is  your memory configuration?

I took a look at "top" while reading the file on the client;

It looks like it's sitting there waiting for io. Something for the
kernel-people maybe?

 15:15:10  up 1 day, 19:31,  4 users,  load average: 7.55, 4.31, 3.54
93 processes: 92 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states:   0.0% user   1.2% system    0.0% nice  95.1% iowait   3.0%
idle
CPU1 states:   0.0% user   1.2% system    0.0% nice  90.1% iowait   8.1%
idle
Mem:  1035184k av, 1029660k used,    5524k free,       0k shrd,   18216k
buff
        52712k active,             929460k inactive
Swap:  987956k av,       0k used,  987956k free                  957788k
cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
 4473 root      16   0  4008 1084  3832 R     0.3  0.1   0:00   1 top
    8 root       5 -10     0    0     0 SW<   0.1  0.0   0:15   0
kblockd/0
    9 root       5 -10     0    0     0 SW<   0.1  0.0   0:14   1
kblockd/1
 4382 root      15   0     0    0     0 SW    0.1  0.0   0:01   0 nfsd
 4384 root      15   0     0    0     0 SW    0.1  0.0   0:01   0 nfsd
 4386 root      15   0     0    0     0 DW    0.1  0.0   0:01   1 nfsd
 4387 root      15   0     0    0     0 SW    0.1  0.0   0:01   0 nfsd
 4392 root      15   0     0    0     0 DW    0.1  0.0   0:01   0 nfsd
 4394 root      15   0     0    0     0 DW    0.1  0.0   0:00   1 nfsd
 4395 root      15   0     0    0     0 SW    0.1  0.0   0:00   1 nfsd
 4396 root      15   0     0    0     0 DW    0.1  0.0   0:01   0 nfsd
    1 root      16   0   488  240   464 S     0.0  0.0   0:05   1 init


--=20
A: Top Posters
Q: What is the most annoying thing on mailing lists?

Harald H Hannelius | harald/a\arcada.fi      | GSM +358 50 594 1020


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Poor NFS performance, kernel 2.6.6.

[Dexter Filmore]

How are the GB NICs attached? Onboard? Integrated into Southbridge? PCI?
PCI-X? 
If it's plain old PCI, you won't get more, PCI maxes out here. 


On Thu, 27 May 2004 13:31:57 +0300 (EEST)
Harald Hannelius <harald+nfs@arcada.fi> wrote:

> 
> Hi there,
> 
> I'm running two servers, both Dual Opterons with Broadcom Gb NICs. I'm
> using the bcm5700 driver and I see no errors whatsoever with 'ifconfig'.
> 
> I'm using nfs-utils-1.0.6 and the 2.6.6 kernel, compiled with both NFSv3
> and NFSv4 support. The client and servers are identical except for the
> server having a scsi-raid under /home and the client being and IDE-box.
> 
> Distribution slackware-9.1 on both computers.
> 
> /etc/exports;
> 
>    /home   193.167.32.175(ro,no_root_squash,async)
> 
> /etc/fstab;
> 
>   193.167.32.187:/home    /home   nfs     \
>  defaults,noauto,rsize=32768,wsize=32768,nfsvers=3 0 0
> 
> (as you can see I have experimented with different rsize,wsize without any
> noticeable effect, same goes for nfsvers 2 and 3. I even mounted the
> filesystem as ext2 on the server and as ext3 with data=journal with no
> effect)
> 
> The computers are connected through a HP 8-port Gb switch and are on the
> same subnet.
> 
> rsync over ssh gives me roughly 200Mbps (37 GB dataset)
> netcat over tcp with a 2GB file gives me 457 Mbps
> netcat over udp with a 2GB file gives me 640 Mbps
> 
> But dd'ing the 2GB file over nfs as some NFS-HOWTO suggests takes over 5
> minutes. That should equal to something around 49 Mbps, correct?
> 
> The netcat takes 25 secs to transfer 2GB file.
> 
> Is there some way I can look at the nfs-server what it's doing? Any
> suggestions on solutions for this?
> 
> And what datarates over NFS could I expect on a setup like this?
> Half-wirespeed maybe?
> 
> Thanks in advance,
> 
>   Harald
> 
> 
> -- 
> A: Top Posters
> Q: What is the most annoying thing on mailing lists?
> 
> Harald H Hannelius | harald/a\arcada.fi      | GSM +358 50 594 1020
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the market... Oracle 10g. 
> Take an Oracle 10g class now, and we'll give you the exam FREE.
> http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs
> 


-- 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS d--(+)@ s-:+ a- C+++(++++) UL+>++++ P+>++ L+++>++++ E-- W++ N o? K-
w--(---) !O M+ V- PS++(+) PE(-) Y+ PGP(-) t++(---)@ 5 X+(++) R+(++) tv--(+)@ 
b+(+++) DI+++ D G++ e* h>++ r%>* y?
------END GEEK CODE BLOCK------

http://www.againsttcpa.com - nothing fights like the opposition


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Poor NFS performance, kernel 2.6.6.

[Vincent ROQUETA]

Le Jeudi 27 Mai 2004 14:17, Harald Hannelius a =E9crit :
> On Thu, 27 May 2004, Vincent ROQUETA wrote:
> > Le Jeudi 27 Mai 2004 12:31, Harald Hannelius a =E9crit :
> > > Is there some way I can look at the nfs-server what it's doing? Any
> > > suggestions on solutions for this?
> > >
> > > And what datarates over NFS could I expect on a setup like this?
> > > Half-wirespeed maybe?
> >
> > on NFSv3, 1 ethernet 1GB I reached  89% of the avalaible brandwith for
> > NFSv3, kernel 2.6.3, for file less than the total amount of ram (4Go),
> > and 22% of the brandwith for file over 6Go, because of the use of
> > harddisk use in the copy process (swap).
> >
> > What is  your memory configuration?
>
> I took a look at "top" while reading the file on the client;
>
> It looks like it's sitting there waiting for io. Something for the
> kernel-people maybe?
>
>  15:15:10  up 1 day, 19:31,  4 users,  load average: 7.55, 4.31, 3.54
> 93 processes: 92 sleeping, 1 running, 0 zombie, 0 stopped
> CPU0 states:   0.0% user   1.2% system    0.0% nice  95.1% iowait   3.0%
> idle
> CPU1 states:   0.0% user   1.2% system    0.0% nice  90.1% iowait   8.1%
> idle
> Mem:  1035184k av, 1029660k used,    5524k free,       0k shrd,   18216k
> buff
>         52712k active,             929460k inactive
> Swap:  987956k av,       0k used,  987956k free                  957788k
> cached
>
>   PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
>  4473 root      16   0  4008 1084  3832 R     0.3  0.1   0:00   1 top
>     8 root       5 -10     0    0     0 SW<   0.1  0.0   0:15   0
> kblockd/0
>     9 root       5 -10     0    0     0 SW<   0.1  0.0   0:14   1
> kblockd/1
>  4382 root      15   0     0    0     0 SW    0.1  0.0   0:01   0 nfsd
>  4384 root      15   0     0    0     0 SW    0.1  0.0   0:01   0 nfsd
>  4386 root      15   0     0    0     0 DW    0.1  0.0   0:01   1 nfsd
>  4387 root      15   0     0    0     0 SW    0.1  0.0   0:01   0 nfsd
>  4392 root      15   0     0    0     0 DW    0.1  0.0   0:01   0 nfsd
>  4394 root      15   0     0    0     0 DW    0.1  0.0   0:00   1 nfsd
>  4395 root      15   0     0    0     0 SW    0.1  0.0   0:00   1 nfsd
>  4396 root      15   0     0    0     0 DW    0.1  0.0   0:01   0 nfsd
>     1 root      16   0   488  240   464 S     0.0  0.0   0:05   1 init

That's normal : waiting for IO -> probably waiting for reading something fr=
om=20
the network. I am trying to do the same experience.




-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Poor NFS performance, kernel 2.6.6.

[Harald Hannelius]

On Thu, 27 May 2004, Dexter Filmore wrote:

> How are the GB NICs attached? Onboard? Integrated into Southbridge? PCI?
> PCI-X?

It's a Tyan K8S PRo, dual tigon 3 NICs onboard.

eth0: Tigon3 [partno(BCM95704A7) rev 2003 PHY(5704)] (PCIX:100MHz:64-bit)
10/100/1000BaseT Ethernet 00:e0:81:28:cf:89

> If it's plain old PCI, you won't get more, PCI maxes out here.

Oh, and why do netcat transfer the 2GB file in 25 secs? With the same NIC
and network ofcourse.. As stated in my original message.

I tried linux-2.4.26 kernel instead. Now the 'time cat file > /dev/null'
takes 1 min 3s on the client which is more like I wanted it. That should
equal to something around 254Mbps..

Since I don't have any real reason for a 2.6 kernel right now (32bit
distro) I will rest my case here. Just thought someone would like to
know..

  Cheers, and many thanks

	Harald


-- 
A: Top Posters
Q: What is the most annoying thing on mailing lists?

Harald H Hannelius | harald/a\arcada.fi      | GSM +358 50 594 1020


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Poor NFS performance, kernel 2.6.6.

[Harald Hannelius]

[already posted this on the nfs-list too]

Hi there,

I'm running two servers, both Dual Opterons with Broadcom Gb NICs. I'm
using the bcm5700 driver and I see no errors whatsoever with 'ifconfig'.
These NICs are onboard PCIX:100MHz:64-bit.

I'm using nfs-utils-1.0.6 and the 2.6.6 kernel, compiled with both NFSv3
and NFSv4 support. The client and servers are identical except for the
server having a scsi-raid under /home and the client being and IDE-box.

Distribution slackware-9.1 on both computers.

/etc/exports;

   /home   193.167.32.175(ro,no_root_squash,async)

/etc/fstab;

  193.167.32.187:/home    /home   nfs     \
 defaults,noauto,rsize=32768,wsize=32768,nfsvers=3 0 0

(as you can see I have experimented with different rsize,wsize without any
noticeable effect, same goes for nfsvers 2 and 3. I even mounted the
filesystem as ext2 on the server and as ext3 with data=journal with no
effect)

The computers are connected through a HP 8-port Gb switch and are on the
same subnet.

rsync over ssh gives me roughly 200Mbps (37 GB dataset)
netcat over tcp with a 2GB file gives me 457 Mbps
netcat over udp with a 2GB file gives me 640 Mbps

But dd'ing the 2GB file over nfs as some NFS-HOWTO suggests takes over 5
minutes. That should equal to something around 49 Mbps, correct?

The netcat takes 25 secs to transfer 2GB file.

I installed kernel 2.4.26 and the netcat finished in 29seconds (550Mbps).
With kernel 2.4.26 the 'time cat largefile > /dev/null' over NFS took just
1min3s on the client. Which should equal to something like 254Mbps.


Is there some way I can look at the nfs-server what it's doing? Any
suggestions on solutions for this?

And what datarates over NFS could I expect on a setup like this?
Half-wirespeed maybe?

Thanks in advance,

  Harald



-- 
A: Top Posters
Q: What is the most annoying thing on mailing lists?

Harald H Hannelius | harald/a\arcada.fi      | GSM +358 50 594 1020

Re: Poor NFS performance, kernel 2.6.6.

[Bernd Schubert]

[-- Attachment #1: Type: text/plain, Size: 736 bytes --]

Hello,

>
> Distribution slackware-9.1 on both computers.
>
> /etc/exports;
>
>    /home   193.167.32.175(ro,no_root_squash,async)
>
> /etc/fstab;
>
>   193.167.32.187:/home    /home   nfs     \
>  defaults,noauto,rsize=32768,wsize=32768,nfsvers=3 0 0
>

if you read the list archive, you will see that Trond always suggests to use 
tcp in situations like this. Some weeks ago there was a discussion on LKML 
about 2.6., udp and nfs, the final statement was, that one always should use 
tcp with 2.6. kernel versions (I still don't understand why, but my knowledge 
about network protocol design and kernel internal networking is probably too 
much limited to understand the reason ;) ).

Hope it helps,
	Bernd 

[-- Attachment #2: signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Poor NFS performance, kernel 2.6.6.

[Trond Myklebust]

P=E5 to , 27/05/2004 klokka 14:07, skreiv Bernd Schubert:
> if you read the list archive, you will see that Trond always suggests to =
use=20
> tcp in situations like this. Some weeks ago there was a discussion on LKM=
L=20
> about 2.6., udp and nfs, the final statement was, that one always should =
use=20
> tcp with 2.6. kernel versions (I still don't understand why, but my knowl=
edge=20
> about network protocol design and kernel internal networking is probably =
too=20
> much limited to understand the reason ;) ).

It has all to do with the network protocols. It's nothing we can fix in
the kernel.

See the discussion in FAQ number B10 on http://nfs.sourceforge.net

see also Casper's discussion on the same issue on
   http://www.netsys.com/sunmgr/1997-09/msg00070.html

Cheers,
  Trond


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Poor NFS performance, kernel 2.6.6.

[Harald Hannelius]

On Thu, 27 May 2004, Bernd Schubert wrote:

> if you read the list archive, you will see that Trond always suggests to use
> tcp in situations like this. Some weeks ago there was a discussion on LKML

I did, same results.


-- 
A: Top Posters
Q: What is the most annoying thing on mailing lists?

Harald H Hannelius | harald/a\arcada.fi      | GSM +358 50 594 1020


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Poor NFS performance, kernel 2.6.6.

[Trond Myklebust]

P=E5 to , 27/05/2004 klokka 06:31, skreiv Harald Hannelius:

> But dd'ing the 2GB file over nfs as some NFS-HOWTO suggests takes over 5
> minutes. That should equal to something around 49 Mbps, correct?

So what is the speed when dding the file locally on the server? (that is
*not* the same as rsync)

What if you then do the same thing on the server via a loopback mount on
the server (i.e. mount localhost:/home /mnt)?

Finally, if you do "nfsstat", what sort of values are you getting for
"retrans"?

Cheers,
  Trond


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Poor NFS performance, kernel 2.6.6.

[Harald Hannelius]

On Thu, 27 May 2004, Trond Myklebust wrote:

> P=E5 to , 27/05/2004 klokka 06:31, skreiv Harald Hannelius:
>
> > But dd'ing the 2GB file over nfs as some NFS-HOWTO suggests takes over =
5
> > minutes. That should equal to something around 49 Mbps, correct?
>
> So what is the speed when dding the file locally on the server? (that is
> *not* the same as rsync)

:)

Kernel 2.4.26:

# time dd if=3D/dev/zero of=3Dlarge bs=3D1M count=3D2000
2000+0 records in
2000+0 records out

real    1m11.565s
user    0m0.010s
sys     0m10.520s

# time dd if=3Dlarge of=3D/dev/null
4096000+0 records in
4096000+0 records out

real    0m23.506s
user    0m1.360s
sys     0m6.820s


From=20now on all tests with kernel 2.6.6;

# time dd if=3Dlarge of=3D/dev/null
4096000+0 records in
4096000+0 records out

real    0m21.190s
user    0m1.622s
sys     0m5.192s

> What if you then do the same thing on the server via a loopback mount on
> the server (i.e. mount localhost:/home /mnt)?

While doing the dd;

 10:07:25  up 8 min,  2 users,  load average: 9.87, 6.16, 2.60

And timings for the dd-command;

 # time dd if=3Dlarge of=3D/dev/null
4096000+0 records in
4096000+0 records out

real    6m21.086s
user    0m1.712s
sys     0m5.454s

Just for the heck of it I rebooted into 2.4.26 and did  the same test over
nfs on localhost;

 # time dd if=3Dlarge of=3D/dev/null
4096000+0 records in
4096000+0 records out

real    1m11.875s
user    0m1.730s
sys     0m7.730s


> Finally, if you do "nfsstat", what sort of values are you getting for
> "retrans"?

(2.6.6)
# nfsstat
Server rpc stats:
calls      badcalls   badauth    badclnt    xdrcall
48018      0          0          0          0
Server nfs v2:
null       getattr    setattr    root       lookup     readlink
0       0% 0       0% 0       0% 0       0% 0       0% 0       0%
read       wrcache    write      create     remove     rename
0       0% 0       0% 0       0% 0       0% 0       0% 0       0%
link       symlink    mkdir      rmdir      readdir    fsstat
0       0% 0       0% 0       0% 0       0% 0       0% 0       0%

Server nfs v3:
null       getattr    setattr    lookup     access     readlink
0       0% 9       0% 0       0% 1       0% 10      0% 0       0%
read       write      create     mkdir      symlink    mknod
47995  99% 0       0% 0       0% 0       0% 0       0% 0       0%
remove     rmdir      rename     link       readdir    readdirplus
0       0% 0       0% 0       0% 0       0% 0       0% 2       0%
fsstat     fsinfo     pathconf   commit
0       0% 1       0% 0       0% 0       0%

Client rpc stats:
calls      retrans    authrefrsh
47656      609        0
Client nfs v2:
null       getattr    setattr    root       lookup     readlink
0       0% 0       0% 0       0% 0       0% 0       0% 0       0%
read       wrcache    write      create     remove     rename
0       0% 0       0% 0       0% 0       0% 0       0% 0       0%
link       symlink    mkdir      rmdir      readdir    fsstat
0       0% 0       0% 0       0% 0       0% 0       0% 0       0%

Client nfs v3:
null       getattr    setattr    lookup     access     readlink
0       0% 9       0% 0       0% 1       0% 10      0% 0       0%
read       write      create     mkdir      symlink    mknod
47633  99% 0       0% 0       0% 0       0% 0       0% 0       0%
remove     rmdir      rename     link       readdir    readdirplus
0       0% 0       0% 0       0% 0       0% 0       0% 2       0%
fsstat     fsinfo     pathconf   commit
0       0% 1       0% 0       0% 0       0%


And here's output from vmstat while doing the dd (2.6.6);

# vmstat 1
procs                      memory      swap          io     system
cpu
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy
wa id
 1 15      0   5452   1548 1009476    0    0  3350     4  910  1186  0  2
55 42
 0  8      0   6452   1556 1008312    0    0  6104     0 2583  4576  1  2
94  3
 0 12      0   4876   1552 1009948    0    0  5252     0 2395  3915  0  2
98  1
 0  8      0   5388   1536 1009488    0    0  5132     0 2404  4099  0  2
89 10
 0 12      0   6564   1536 1008264    0    0  4856     0 2310  3737  0  2
96  3
 0 17      0   4556   1544 1010364    0    0  4364    72 2157  3128  0  1
98  0
 0  7      0   5772   1552 1009132    0    0  4880     4 2325  3639  0  2
97  1
 0  5      0   6220   1536 1008808    0    0  5316     0 2377  4361  1  2
94  4
 0  2      0   5708   1528 1009156    0    0  6748     0 2756  4942  0  3
93  4
 0  6      0   6388   1536 1008468    0    0  4784     0 2269  3458  0  2
92  7
 0 16      0   5196   1504 1009724    0    0  5928     0 2563  4368  1  2
96  2
 0  3      0   6284   1508 1008632    0    0  5136     8 2384  4121  0  3
95  2
 0  2      0   6452   1488 1008380    0    0  5288     4 2400  4311  1  1
92  6
 0  6      0   4812   1492 1010144    0    0  5256     0 2428  3806  0  2
96  2
 0  7      0   5260   1472 1009756    0    0  5244     0 2379  4180  0  2
98  1
 0  8      0   5580   1456 1009500    0    0  5332     0 2418  3858  0  2
98  1
 0  6      0   5004   1464 1009968    0    0  4524     8 2238  3263  0  1
95  3


>
> Cheers,
>   Trond
>

--=20
A: Top Posters
Q: What is the most annoying thing on mailing lists?

Harald H Hannelius | harald/a\arcada.fi      | GSM +358 50 594 1020


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Poor NFS performance, kernel 2.6.6.

[Olaf Kirch]

[2.6 over loopback nfs]
> real    6m21.086s
> user    0m1.712s
> sys     0m5.454s

[2.4 over loopback nfs]
> real    1m11.875s
> user    0m1.730s
> sys     0m7.730s

This roughly matches some performance issues I've been seeing on a 2.6.5
kernel. Are you using reiser or ext3?

In general, dd to a local file is much different from what NFS does,
because nfsd flushes data to disk.

The NFS client will usually issue a COMMIT call roughly every 1 Meg,
so the following should mimic the behavior of nfsd a little better,
when run on your file system locally

	iozone -s 1g -r 1m -o -i 0

Higher numbers indicate better performance.

Could you run this on 2.4 and 2.6, please?

Olaf
-- 
Olaf Kirch     |  The Hardware Gods hate me.
okir@suse.de   |
---------------+ 


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Poor NFS performance, kernel 2.6.6.

[Phy Prabab]

Hello,

I have been watching this thread with interest.  I
have a 2.6.6-mm5 kernel NFS server in production and
have noticed some significant slow down to that server
(in comparison to an old crusty 2.4.22 kernel and
NetApp).  I too would like to understand the issue
here so I have run iozone as indicated to give another
point of interest.  Note, that during one of the tests
(nfsv2,udp,8192, the server crawled to a halt, load
sky rocketed to 30 with zero load (100% available
reported by top), at this point the server refused
access via ssh/rsh and refused to export any volumes,
but ping-able.  At terminal, everything behavied
normal.

Prior to the lock up, here are the numbers I was
getting:

Client 2.6.6-m5

2.6.6-mm5:

        File size set to 1048576 KB
        Record Size 1024 KB
        SYNC Mode. 
        Command line used:
/var/tmp/iozone/src/current/iozone -s 1g -r 1m -o -i 0
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                      
     
UDP, NFSv2, sync, r/w 8192
         1048576    1024   39889   38745              
                                                      
                      
UDP, NFSv3, sync, r/w 32768
         1048576    1024   12226   10749
TCP, NFSv3, sync, r/w 32768
         1048576    1024   11909   12208              
                                                      
                      

2.4.22
UDP, NFSv3, sync, r/w 32768
         1048576    1024   11259   10761
UDP, NFSv3, sync, r/w 8192
         1048576    1024    6136    6266              
                                                      
        


I cc'ed the linux-kernel mailing list as well.

Cheers,
Phy


--- Olaf Kirch <okir@suse.de> wrote:
> [2.6 over loopback nfs]
> > real    6m21.086s
> > user    0m1.712s
> > sys     0m5.454s
> 
> [2.4 over loopback nfs]
> > real    1m11.875s
> > user    0m1.730s
> > sys     0m7.730s
> 
> This roughly matches some performance issues I've
> been seeing on a 2.6.5
> kernel. Are you using reiser or ext3?
> 
> In general, dd to a local file is much different
> from what NFS does,
> because nfsd flushes data to disk.
> 
> The NFS client will usually issue a COMMIT call
> roughly every 1 Meg,
> so the following should mimic the behavior of nfsd a
> little better,
> when run on your file system locally
> 
> 	iozone -s 1g -r 1m -o -i 0
> 
> Higher numbers indicate better performance.
> 
> Could you run this on 2.4 and 2.6, please?
> 
> Olaf
> -- 
> Olaf Kirch     |  The Hardware Gods hate me.
> okir@suse.de   |
> ---------------+ 
> 
> 
>
-------------------------------------------------------
> This SF.Net email is sponsored by: Oracle 10g
> Get certified on the hottest thing ever to hit the
> market... Oracle 10g. 
> Take an Oracle 10g class now, and we'll give you the
> exam FREE.
>
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs



	
		
__________________________________
Do you Yahoo!?
Friends.  Fun.  Try the all-new Yahoo! Messenger.
http://messenger.yahoo.com/ 


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Poor NFS performance, kernel 2.6.6.

[Harald Hannelius]

On Fri, 28 May 2004, Olaf Kirch wrote:

> [2.6 over loopback nfs]
> > real    6m21.086s
> > user    0m1.712s
> > sys     0m5.454s
>
> [2.4 over loopback nfs]
> > real    1m11.875s
> > user    0m1.730s
> > sys     0m7.730s
>
> This roughly matches some performance issues I've been seeing on a 2.6.5
> kernel. Are you using reiser or ext3?

ext3.

> In general, dd to a local file is much different from what NFS does,
> because nfsd flushes data to disk.

I guess (as a total don't-know) that nfsd reads and writes files in a
different way than local processes do. I read somewhere something about
read-ahead but that was in 2.6.0-pre-something..

> The NFS client will usually issue a COMMIT call roughly every 1 Meg,
> so the following should mimic the behavior of nfsd a little better,
> when run on your file system locally
>
> 	iozone -s 1g -r 1m -o -i 0
>
> Higher numbers indicate better performance.
>
> Could you run this on 2.4 and 2.6, please?

Oh sorry, the server went into production to day so I can't run the tests
on 2.6 without rebooting first. Not a good idea right now :)

But here are (useless without 2.6 ditto) results with kernel 2.4.26 run
locally on the fileserver;

 http://people.arcada.fi/~harald/nfs/iozone-1MBcommit.txt
 http://people.arcada.fi/~harald/nfs/iozone-1MBcommit2.txt



-- 
A: Top Posters
Q: What is the most annoying thing on mailing lists?

Harald H Hannelius | harald/a\arcada.fi      | GSM +358 50 594 1020


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Poor NFS performance, kernel 2.6.6.

[Trond Myklebust]

P=E5 fr , 28/05/2004 klokka 14:03, skreiv Harald Hannelius:

> I guess (as a total don't-know) that nfsd reads and writes files in a
> different way than local processes do. I read somewhere something about
> read-ahead but that was in 2.6.0-pre-something..

Hmm... The current readahead code does a load of crap in order to
determine whether or not the user is doing linear or random I/O. It may
be that the nfsd server with all its multi-threading and out-of-sequence
I/O is triggering the random I/O readahead code (which of course aims to
limit readahead).

Cheers
  Trond


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Poor NFS performance, kernel 2.6.6.

[Yusuf Goolamabbas]

> > I guess (as a total don't-know) that nfsd reads and writes files in a
> > different way than local processes do. I read somewhere something about
> > read-ahead but that was in 2.6.0-pre-something..
> 
> Hmm... The current readahead code does a load of crap in order to
> determine whether or not the user is doing linear or random I/O. It may
> be that the nfsd server with all its multi-threading and out-of-sequence
> I/O is triggering the random I/O readahead code (which of course aims to
> limit readahead).

Any thoughts on this patch posted to lkml 

http://marc.theaimsgroup.com/?l=linux-kernel&m=108603045809565&w=2

Regards, /yg


-------------------------------------------------------
This SF.Net email is sponsored by: Oracle 10g
Get certified on the hottest thing ever to hit the market... Oracle 10g. 
Take an Oracle 10g class now, and we'll give you the exam FREE.
http://ads.osdn.com/?ad_id=3149&alloc_id=8166&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Poor NFS performance, kernel 2.6.6.

[Trond Myklebust]

P=E5 m=E5 , 31/05/2004 klokka 17:16, skreiv Yusuf Goolamabbas:
> > > I guess (as a total don't-know) that nfsd reads and writes files in a
> > > different way than local processes do. I read somewhere something abo=
ut
> > > read-ahead but that was in 2.6.0-pre-something..
> >=20
> > Hmm... The current readahead code does a load of crap in order to
> > determine whether or not the user is doing linear or random I/O. It may
> > be that the nfsd server with all its multi-threading and out-of-sequenc=
e
> > I/O is triggering the random I/O readahead code (which of course aims t=
o
> > limit readahead).
>=20
> Any thoughts on this patch posted to lkml=20
>=20
> http://marc.theaimsgroup.com/?l=3Dlinux-kernel&m=3D108603045809565&w=3D2

May or may not fix a bug in the NFSd code. Does nothing to change the
basic readahead code in the VFS layer in order to fix the issues I
raised above.

Cheers,
 Trond


-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.

Re: Poor NFS performance, kernel 2.6.6.

[Jeffrey Layton]

I haven't been following this thread closely, but figured I'd chime in
with my own experience with this. Here's a rather unscientific test,
dd'ing to a file on an NFS-mounted filesystem. Mount options are:

udp,soft,intr,nfsvers=3

(I'm using UDP as I'm trying to set up a HA-NFS server, and the clients
seem to recover much faster when using UDP as a transport).

With a 2.4 kernel server:

% time dd if=/dev/zero of=./testfile bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes transferred in 11.806235 seconds (8881544 bytes/sec)
dd if=/dev/zero of=./testfile bs=100M count=1  0.00s user 0.28s system
2% cpu 12.147 total

With a 2.6 kernel server:

% time dd if=/dev/zero of=./testfile bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes transferred in 66.997572 seconds (1565096 bytes/sec)
dd if=/dev/zero of=./testfile bs=100M count=1  0.00s user 0.29s system
0% cpu 1:07.06 total


The machines are different hardware, but local write performance is
pretty comparable (in fact the 2.6 box is a faster machine, and is
currently less utilized than the 2.4 kernel machine). Both are using
reiserfs as the underlying filesystem.

Write performance in this cursory test was 10x worse! Clearly, there's
some sort of problem with NFS on 2.6. I'll be happy to send in what info
I can. My 2.4 machine is currently a production box, but I can run an
instrumented kernel, etc. on the 2.6 box in the near future if anyone
here can guide me on what I can do to help.

-- Jeff




-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.

Re: Poor NFS performance, kernel 2.6.6.

[Greg Banks]

On Wed, Jun 02, 2004 at 08:33:53AM -0400, Jeffrey Layton wrote:
> The machines are different hardware, but local write performance is
> pretty comparable (in fact the 2.6 box is a faster machine, and is
> currently less utilized than the 2.4 kernel machine). Both are using
> reiserfs as the underlying filesystem.

What is the speed on each server for a local dd from /dev/zero to disk?
What actual kernel versions are you running?

> Write performance in this cursory test was 10x worse!

This completely fails to correlate with my experience.  I'm using XFS
on 2.6.4 and I see single thread streaming write performance increase
by about 25% going from 2.4 to 2.6.  Other metrics show improvements
as good or better, except the scalability of UDP reads tops out lower.

Greg.
-- 
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
I don't speak for SGI.


-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.

Re: Poor NFS performance, kernel 2.6.6.

[Vincent ROQUETA]

Le Mercredi 02 Juin 2004 14:33, Jeffrey Layton a =C3=A9crit :
> I haven't been following this thread closely, but figured I'd chime in
> with my own experience with this. Here's a rather unscientific test,
> dd'ing to a file on an NFS-mounted filesystem. Mount options are:
>
> udp,soft,intr,nfsvers=3D3
>
> (I'm using UDP as I'm trying to set up a HA-NFS server, and the clients
> seem to recover much faster when using UDP as a transport).
>
> With a 2.4 kernel server:
>
> % time dd if=3D/dev/zero of=3D./testfile bs=3D100M count=3D1
> 1+0 records in
> 1+0 records out
> 104857600 bytes transferred in 11.806235 seconds (8881544 bytes/sec)
> dd if=3D/dev/zero of=3D./testfile bs=3D100M count=3D1  0.00s user 0.28s s=
ystem
> 2% cpu 12.147 total
>
> With a 2.6 kernel server:
>
> % time dd if=3D/dev/zero of=3D./testfile bs=3D100M count=3D1
> 1+0 records in
> 1+0 records out
> 104857600 bytes transferred in 66.997572 seconds (1565096 bytes/sec)
> dd if=3D/dev/zero of=3D./testfile bs=3D100M count=3D1  0.00s user 0.29s s=
ystem
> 0% cpu 1:07.06 total
>
>
> The machines are different hardware, but local write performance is
> pretty comparable (in fact the 2.6 box is a faster machine, and is
> currently less utilized than the 2.4 kernel machine). Both are using
> reiserfs as the underlying filesystem.
>
> Write performance in this cursory test was 10x worse! Clearly, there's
> some sort of problem with NFS on 2.6. I'll be happy to send in what info
> I can. My 2.4 machine is currently a production box, but I can run an
> instrumented kernel, etc. on the 2.6 box in the near future if anyone
> here can guide me on what I can do to help.
>
> -- Jeff
>

What appends with linux 2.6.6?
Is NFSv4 enabled?
Can you use bonnie / IOzone and repport activity?
 (-> http://nfsv4.bullopensource.org/tools/sondage.php for performances too=
ls=20
)
=20


-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.

Re: Poor NFS performance, kernel 2.6.6.

[Jeffrey Layton]

On Wed, 2004-06-02 at 09:00, Greg Banks wrote:
> On Wed, Jun 02, 2004 at 08:33:53AM -0400, Jeffrey Layton wrote:
> > The machines are different hardware, but local write performance is
> > pretty comparable (in fact the 2.6 box is a faster machine, and is
> > currently less utilized than the 2.4 kernel machine). Both are using
> > reiserfs as the underlying filesystem.
> 
> What is the speed on each server for a local dd from /dev/zero to disk?
> What actual kernel versions are you running?
> 

I need to amend my statement above. I'm having a problem mounting the
filesystem from the 2.6 kernel as v3. I keep getting a message saying
that v3 isn't supported, even though I can mount using v3 from the 2.4
box without an issue. Another 2.6 box doesn't seem to have this issue,
so I'm unsure of what the deal is. Rebuilding my client's kernel to see
if I can clear up that problem. Once I get that redone, I'll redo this
test and see if the poor performance remains...


In any case...

Less than .2 secs to do a local dd to disk:

laytonj@angstrom:/services/NFS/home% time dd if=/dev/zero of=./testfile
bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes transferred in 0.200756 seconds (522313514 bytes/sec)
dd if=/dev/zero of=./testfile bs=100M count=1  0.00s user 0.20s system
107% cpu 0.186 total

It's a stock 2.6.6 kernel (the 'a' is an internal versioning scheme I
use when I have multiple kernel builds of same version):

laytonj@angstrom:/services/NFS/home% uname -a
Linux angstrom 2.6.6a #1 Mon May 17 14:29:24 EDT 2004 i686 GNU/Linux





-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.

Re: Poor NFS performance, kernel 2.6.6.

[Jeffrey Layton]

Ok, it looks like the problem here was that I already had the filesystem
mounted on another directory as a NFSv2 filesystem. When I unmounted
this filesystem, and remounted it as v3 only, I started getting
performance roughly on par with the 2.4 kernels. Sorry for the false
alarm!
Cheers,
-- Jeff



-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.