Throughput @100Mbs on link of ~10ms latency

Throughput @100Mbs on link of ~10ms latency

[Mark Henson]

Hi,

Can someone give me a pointer to a FAQ on how to tune a 2.4 machine to
achieve high throughput (approx i/f speed 100Mbits/sec) on a link with the
following characteristics:



Latency		Throughput

9-10ms		3.8 MByte/s
3-4ms		7-8MByte/s

I have implemented:

echo "4096 87380 4194304" > /proc/sys/net/ipv4/tcp_rmem
echo "4096 65536 4194304" > /proc/sys/net/ipv4/tcp_wmem

from http://www-didc.lbl.gov/tcp-wan.html

this lifted the performance from ~1MByte/s to the 3.8 above.

When the receiving machine is freebsd I get 10.05 MBytes/s which
is interesting - but when sending from BSD I get the same rate.

cheers
Mark


[root@tsaturn ncftp]# lsmod
Module                  Size  Used by
autofs                 11264   1  (autoclean)
3c59x                  25344   1  (autoclean)
e100                   44240   1  (autoclean)
ipchains               38976   0  (unused)

Re: Throughput @100Mbs on link of ~10ms latency

[Ben Greear]

Mark Henson wrote:
> 
> Hi,
> 
> Can someone give me a pointer to a FAQ on how to tune a 2.4 machine to
> achieve high throughput (approx i/f speed 100Mbits/sec) on a link with the
> following characteristics:
> 
> Latency         Throughput
> 
> 9-10ms          3.8 MByte/s
> 3-4ms           7-8MByte/s
> 
> I have implemented:
> 
> echo "4096 87380 4194304" > /proc/sys/net/ipv4/tcp_rmem
> echo "4096 65536 4194304" > /proc/sys/net/ipv4/tcp_wmem
> 
> from http://www-didc.lbl.gov/tcp-wan.html
> 
> this lifted the performance from ~1MByte/s to the 3.8 above.
> 
> When the receiving machine is freebsd I get 10.05 MBytes/s which
> is interesting - but when sending from BSD I get the same rate.
> 

Some things I change to increase performance (it seems to help...)
(I'm cutting & pasting from a perl script, so you'll need to interpret
 it a bit..)

# NOTE:  For faster systems I usually bump the *mem* settings up to 4MB.

my $netdev_max_backlog = 4096; # Maximum number of packets, queued on the INPUT side, when
                               # the interface receives pkts faster than it can process them.

my $wmem_max = 512000;  # Write memory buffer.  This is probably fine for any setup,
                        # and could be smaller (256000) for < 5Mbps connections.

my $wmem_default = 512000;  # Write memory buffer.  This is probably fine for any setup,
                            # and could be smaller (256000) for < 5Mbps connections.

my $rmem_max = 1024000;  # Receive memory (packet) buffer.  If you are running lots of very fast traffic,
                         # you may want to make this larger, up to 4096000 or so.  If you are
                         # at around 20Mbps of traffic per connection or smaller, then 1024000
                         # is plenty.  For < 5Mbps of traffic, 512000 should be fine.

my $rmem_default = 1024000;  # Receive memory (packet) buffer.  If you are running lots of very fast traffic,
                             # you may want to make this larger, up to 4096000 or so.  If you are
                             # at around 20Mbps of traffic per connection or smaller, then 1024000
                             # is plenty.  For < 5Mbps of traffic, 512000 should be fine.

printAndExec("echo $wmem_max > /proc/sys/net/core/wmem_max");
printAndExec("echo $wmem_default > /proc/sys/net/core/wmem_default");
printAndExec("echo $rmem_max > /proc/sys/net/core/rmem_max");
printAndExec("echo $rmem_default > /proc/sys/net/core/rmem_default");
printAndExec("echo $netdev_max_backlog > /proc/sys/net/core/netdev_max_backlog");


And of course, make sure you can get the performance with a known fast network
(and near zero latency) first!!

The e100 has some interesting options that seem to make it handle high packet
counts better, as well as giving it bigger descriptor lists, but I haven't
really benchmarked it..

Ben

> cheers
> Mark
> 
> [root@tsaturn ncftp]# lsmod
> Module                  Size  Used by
> autofs                 11264   1  (autoclean)
> 3c59x                  25344   1  (autoclean)
> e100                   44240   1  (autoclean)
> ipchains               38976   0  (unused)
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: Throughput @100Mbs on link of ~10ms latency

[Mark Henson]

On Thu, 4 Oct 2001, Ben Greear wrote:

> printAndExec("echo $rmem_default > /proc/sys/net/core/rmem_default");
> printAndExec("echo $netdev_max_backlog > /proc/sys/net/core/netdev_max_backlog");
>

... thanks -

tried these parameters and saw no change in the throughput.  I am
thinking that the problem is at the receiving end rather than the transmit
end because with the FreeBSD machine at 1 end I got a much higher
throughput when sending to that machine.  (about 10 MBytes /sec)

I was suspicious of another problem with the FreeBSD machine which is why
I changed it out. (network occasionally locked up).  The distance between
the local ethernet switches is about 1200km of fibre.  Speed of light for
that distance is ~ 4ms.  Light in fibre I believe to be about 0.6 giving
around 6.5ms for light in this loop - so 9.5 ms RTT seems pretty good

The machines (Compaq Deskpro ~800MHz) are able to reliably produce
~10MBytes/sec using ncftp which is hardly calibrated but is atleast
indicative of the rate I should be getting.  Calculating transfer time of
26/27 secs of 106348240 gives me net of about 32Mbps

anyway thanks for your help

cheers
Mark


one machine is:

eth1      Link encap:Ethernet  HWaddr 00:04:76:B8:8B:DC
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11757991 errors:0 dropped:0 overruns:1 frame:0
          TX packets:11699364 errors:0 dropped:0 overruns:0 carrier:49
          collisions:274 txqueuelen:100
          Interrupt:5 Base address:0x1000

eth1: 3Com PCI 3cSOHO100-TX Hurricane at 0x1000,  00:04:76:b8:8b:dc, IRQ 5
   product code 4d4d rev 00.12 date 06-29-01
   8K byte-wide RAM 5:3 Rx:Tx split, autoselect/Autonegotiate interface.
   MII transceiver found at address 24, status 7849.
   Enabling bus-master transmits and whole-frame receives.
 eth1: scatter/gather disabled. h/w checksums enabled
 eth1: using NWAY device table, not 8

the second is:

eth1      Link encap:Ethernet  HWaddr 00:01:03:39:ED:5F
          inet addr:192.168.1.3  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9567005 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9652682 errors:0 dropped:0 overruns:0 carrier:607
          collisions:0 txqueuelen:100
          Interrupt:5 Base address:0x1000

eth1: 3Com PCI 3c980 10/100 Base-TX NIC(Python-T) at 0x1000,
00:01:03:39:ed:5f, IRQ 5
  product code 4b50 rev 00.14 date 03-12-01
  8K byte-wide RAM 5:3 Rx:Tx split, autoselect/Autonegotiate interface.
  MII transceiver found at address 24, status 7809.
  Enabling bus-master transmits and whole-frame receives.

>
> And of course, make sure you can get the performance with a known fast network
> (and near zero latency) first!!
>
> The e100 has some interesting options that seem to make it handle high packet
> counts better, as well as giving it bigger descriptor lists, but I haven't
> really benchmarked it..
>
> Ben
>
>