2.4.20rc1 NFS V3 UDP client problems

2.4.20rc1 NFS V3 UDP client problems

[Eric Whiting]

I'm seeing some 2.4.20rc1 client problems.

25 seconds to write a 1G file
13 minutes to read that file back

CLIENT 2.4.20rc1 NFSV3 UDP HIGHMEM SUPPORT ON
gigE card with jumbo frames enabled 
Tested with both netgear and intel -- same result

SERVER
Netapps 825 ontap 6.3
gigE card with jumbo frames enabled

private point-to-point connection.

Is this a HIGHMEM problem? Or is it related to jumbo frames? I'll test
with both disabled.

I'm seeing network errors on the client -- dropped and overruns are both
the same.

Anything I should try?
 
eric


# time dd if=/dev/zero of=file1 bs=1G count=1
1+0 records in
1+0 records out

real    0m25.881s
user    0m0.000s
sys     0m4.220s


(run a program to calloc 1G of RAM and clear some buffer cache)


# time dd if=file1  of=/dev/null

2097152+0 records in
2097152+0 records out

real    13m7.445s
user    0m0.370s
sys     0m2.160s

 # ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:02:B3:AC:52:F5
          inet addr:10.0.0.2  Bcast:10.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:1534585 errors:10240 dropped:10240 overruns:10240
frame:0
          TX packets:5332748 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:2069590604 (1973.7 Mb)  TX bytes:587855320 (560.6 Mb)
          Interrupt:12 Base address:0xc000 Memory:ea020000-ea040000


-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing 
your web site with SSL, click here to get a FREE TRIAL of a Thawte 
Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.20rc1 NFS V3 UDP client problems

[Eric Whiting]

I disabled jumbo frames and I'm seeing 70M/s reads (with a clean
cache)..

There must be a UDP or NFS client problem with the 9000 mtu packets.

eric



Eric Whiting wrote:
> 
> I'm seeing some 2.4.20rc1 client problems.
> 
> 25 seconds to write a 1G file
> 13 minutes to read that file back
> 
> CLIENT 2.4.20rc1 NFSV3 UDP HIGHMEM SUPPORT ON
> gigE card with jumbo frames enabled
> Tested with both netgear and intel -- same result
> 
> SERVER
> Netapps 825 ontap 6.3
> gigE card with jumbo frames enabled
> 
> private point-to-point connection.
> 
> Is this a HIGHMEM problem? Or is it related to jumbo frames? I'll test
> with both disabled.
> 
> I'm seeing network errors on the client -- dropped and overruns are both
> the same.
> 
> Anything I should try?
> 
> eric
> 
> # time dd if=/dev/zero of=file1 bs=1G count=1
> 1+0 records in
> 1+0 records out
> 
> real    0m25.881s
> user    0m0.000s
> sys     0m4.220s
> 
> (run a program to calloc 1G of RAM and clear some buffer cache)
> 
> # time dd if=file1  of=/dev/null
> 
> 2097152+0 records in
> 2097152+0 records out
> 
> real    13m7.445s
> user    0m0.370s
> sys     0m2.160s
> 
>  # ifconfig eth0
> eth0      Link encap:Ethernet  HWaddr 00:02:B3:AC:52:F5
>           inet addr:10.0.0.2  Bcast:10.255.255.255  Mask:255.0.0.0
>           UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
>           RX packets:1534585 errors:10240 dropped:10240 overruns:10240
> frame:0
>           TX packets:5332748 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:100
>           RX bytes:2069590604 (1973.7 Mb)  TX bytes:587855320 (560.6 Mb)
>           Interrupt:12 Base address:0xc000 Memory:ea020000-ea040000
> 
> -------------------------------------------------------
> This sf.net email is sponsored by: To learn the basics of securing
> your web site with SSL, click here to get a FREE TRIAL of a Thawte
> Server Certificate: http://www.gothawte.com/rd524.html
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs


-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing 
your web site with SSL, click here to get a FREE TRIAL of a Thawte 
Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.20rc1 NFS V3 UDP client problems

[Trond Myklebust]

>>>>> " " == Eric Whiting <ewhiting@amis.com> writes:

     > I disabled jumbo frames and I'm seeing 70M/s reads (with a
     > clean cache)..

     > There must be a UDP or NFS client problem with the 9000 mtu
     > packets.

NFS neither knows nor cares what the MTU is: all it knows is that it
has a socket. I've suspected a bug in the UDP layer for a long time as
you are far from being the first person to report this problem when
enabling jumbo frames.

Cheers,
  Trond


-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing 
your web site with SSL, click here to get a FREE TRIAL of a Thawte 
Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.20rc1 NFS V3 UDP client problems

[canon]

We had to set our MTUs to 8960 for things to work.
We've never followed through to see what the issue
was though.

We are using SysKonnect cards and 2.4.18 kernel.  Note, 
we haven't recently verified that the problem still
exists.

--Shane



-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing 
your web site with SSL, click here to get a FREE TRIAL of a Thawte 
Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.20rc1 NFS V3 UDP client problems

[Eric Whiting]

I tried various MTU values headed gradually towards 9000. I found that a
MTUs between 1500 and 7500 give me the expected performance. MTU values
of anything much more than that causes problems given my setup.

eric


canon@pookie.nersc.gov wrote:
> 
> We had to set our MTUs to 8960 for things to work.
> We've never followed through to see what the issue
> was though.
> 
> We are using SysKonnect cards and 2.4.18 kernel.  Note,
> we haven't recently verified that the problem still
> exists.
> 
> --Shane


-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing 
your web site with SSL, click here to get a FREE TRIAL of a Thawte 
Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.20rc1 NFS V3 UDP client problems

[Benjamin LaHaise]

On Thu, Nov 14, 2002 at 04:13:42PM -0700, Eric Whiting wrote:
> I tried various MTU values headed gradually towards 9000. I found that a
> MTUs between 1500 and 7500 give me the expected performance. MTU values
> of anything much more than that causes problems given my setup.

Which NIC?  Some chipsets (like ns83820) only have an 8KB TX fifo and end 
up doing two passes over the data for larger packet sizes.  Also, running 
gige with a standard 32bit/33MHz bus is pretty close to the limits.

		-ben


-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing 
your web site with SSL, click here to get a FREE TRIAL of a Thawte 
Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.20rc1 NFS V3 UDP client problems

[Eric Whiting]

Both of these NICs seemed to have the same behavior.
<6>eth0: Intel(R) PRO/1000 Network Connection
<6>eth1: NetGear GA620 Gigabit Ethernet at 0xea060000, irq 11

Yes -- I'm running on a 32bit/33Mhz bus right now. I do have a Sun 280R
with a 64bit/66Mhz card in a 66Mhz slot that doesn't even seem to
support jumbo frames at all.

eric



Benjamin LaHaise wrote:
> 
> On Thu, Nov 14, 2002 at 04:13:42PM -0700, Eric Whiting wrote:
> > I tried various MTU values headed gradually towards 9000. I found that a
> > MTUs between 1500 and 7500 give me the expected performance. MTU values
> > of anything much more than that causes problems given my setup.
> 
> Which NIC?  Some chipsets (like ns83820) only have an 8KB TX fifo and end
> up doing two passes over the data for larger packet sizes.  Also, running
> gige with a standard 32bit/33MHz bus is pretty close to the limits.
> 
>                 -ben


-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing 
your web site with SSL, click here to get a FREE TRIAL of a Thawte 
Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.20rc1 NFS V3 UDP client problems

[Benjamin LaHaise]

On Thu, Nov 14, 2002 at 04:23:07PM -0700, Eric Whiting wrote:
> Both of these NICs seemed to have the same behavior.
> <6>eth0: Intel(R) PRO/1000 Network Connection
> <6>eth1: NetGear GA620 Gigabit Ethernet at 0xea060000, irq 11
> 
> Yes -- I'm running on a 32bit/33Mhz bus right now. I do have a Sun 280R
> with a 64bit/66Mhz card in a 66Mhz slot that doesn't even seem to
> support jumbo frames at all.

I'd guess you're hitting the PCI bandwidth limitations then, as the larger 
packets permit greater throughput, which in turn drives up bus utilizations.  
If you've got any disk controllers on the same bus, that would definately 
explain the problem (or any other devices on the bus that are in use).

		-ben


-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing 
your web site with SSL, click here to get a FREE TRIAL of a Thawte 
Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: 2.4.20rc1 NFS V3 UDP client problems -- (not a problem now)

[Eric Whiting]

Ben, 

I found a 64Bit/66Mhz PCI bus and the jumbo frames are working fine.
Same 
kernel, same network cards, same Netapp 825 filer -- simply a different
motherboard.

Here are the NFS client read/write speeds for my setup.  The write
speeds appear to be somewhat limited by the netapps 825 disk/cpu speed.
I wish I had a netapps 9xx to test.

        TCP      UDP
READ    99MB/s  104MB/s
WRITE   36MB/s   49MB/s

UDP looks good with my current setup -- but with the gigE running jumbo
frames in the 32 bit PCI bus the UDP was sick on the reads -- 1MB/s.

Read testing is:
while(true);do mount blaa;dd if=path of=/dev/null bs=x count=y;umount
blaa;done
where bs/count equal a 500M file -- to use the netapps cache.

Write testing is:
dd if=/dev/zero of=file bs=1M

Network rates monitored using netapps syststat and the time results.

eric


Benjamin LaHaise wrote:
> I'd guess you're hitting the PCI bandwidth limitations then, as the larger
> packets permit greater throughput, which in turn drives up bus utilizations.
> If you've got any disk controllers on the same bus, that would definately
> explain the problem (or any other devices on the bus that are in use).
> 
>                 -ben


-------------------------------------------------------
This sf.net email is sponsored by: To learn the basics of securing 
your web site with SSL, click here to get a FREE TRIAL of a Thawte 
Server Certificate: http://www.gothawte.com/rd524.html
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs