Network speed Linux-2.6.10

Network speed Linux-2.6.10

[linux-os]

Conditions:

Intel NIC e100 device driver. Two identical machines.
Private network, no other devices. Connected using a Netgear switch.
Test data is the same thing sent from memory on one machine
to a discard server on another, using TCP/IP SOCK_STREAM.

If I set both machines to auto-negotiation OFF and half duplex,
I get about 9 to 9.5 megabytes/second across the private wire
network.

If I set one machine to full duplex and the other to half-duplex
I get 10 to 11 megabytes/second transfer across the network,
regardless of direction.

If I set both machines to auto-negotiation OFF and full duplex,
I get 300 to 400 kilobytes/second regardless of the direction.

I thought the problem must be the switch so I substituted a
cross-over wire. The problem gets worse, maybe 50 to 100
kilobytes (random bursts) per second.

`lspci`

00:00.0 Host bridge: Intel Corp. 82865G/PE/P DRAM Controller/Host-Hub Interface (rev 02)
00:01.0 PCI bridge: Intel Corp. 82865G/PE/P PCI to AGP Controller (rev 02)
00:1d.0 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #1 (rev 02)
00:1d.1 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #2 (rev 02)
00:1d.2 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #3 (rev 02)
00:1d.3 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #4 (rev 02)
00:1d.7 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02)
00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB/ER Hub interface to PCI Bridge (rev c2)
00:1f.0 ISA bridge: Intel Corp. 82801EB/ER (ICH5/ICH5R) LPC Bridge (rev 02)
00:1f.1 IDE interface: Intel Corp. 82801EB/ER (ICH5/ICH5R) Ultra ATA 100 Storage Controller (rev 02)
00:1f.2 IDE interface: Intel Corp. 82801EB (ICH5) Serial ATA 150 Storage Controller (rev 02)
00:1f.3 SMBus: Intel Corp. 82801EB/ER (ICH5/ICH5R) SMBus Controller (rev 02)
00:1f.5 Multimedia audio controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) AC'97 Audio Controller (rev 02)
01:00.0 VGA compatible controller: nVidia Corporation NV17 [GeForce4 MX 440] (rev a3)
02:00.0 SCSI storage controller: Adaptec AHA-2940U/UW/D / AIC-7881U (rev 01)
02:01.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone] (rev 30)
02:04.0 Class 02b0: Analogic Corp: Unknown device 8004 (rev 07)
02:07.0 FireWire (IEEE 1394): Lucent Microelectronics FW323 (rev 61)
02:08.0 Ethernet controller: Intel Corp. 82562EZ 10/100 Ethernet Controller (rev 01)

proc/interrupts

            CPU0
   0:  507789786    IO-APIC-edge  timer
   1:     459692    IO-APIC-edge  i8042
   7:          0    IO-APIC-edge  parport0
   8:          1    IO-APIC-edge  rtc
   9:          0   IO-APIC-level  acpi
  12:         66    IO-APIC-edge  i8042
  14:     381384    IO-APIC-edge  ide0
  16:          0   IO-APIC-level  uhci_hcd, uhci_hcd
  17:          0   IO-APIC-level  Intel ICH5
  18:          0   IO-APIC-level  libata, uhci_hcd
  19:          0   IO-APIC-level  uhci_hcd
  20:    5748440   IO-APIC-level  eth0
  21:     278400   IO-APIC-level  aic7xxx
  23:          0   IO-APIC-level  ehci_hcd
NMI:          0 
LOC:  507803974 
ERR:          0
MIS:          0


These are identical machines/motherboards. Anybody have a clue
why full-duplex sucks and mismatching FD/HD actually makes
things faster? Is it possible that full-duplex is actually
half-duplex of something strange like that?


Ethtool statistics:

NIC statistics:
      rx_packets: 1732112
      tx_packets: 3612310
      rx_bytes: 162571267
      tx_bytes: 4119843621
      rx_errors: 1094
      tx_errors: 0
      rx_dropped: 0
      tx_dropped: 0
      multicast: 0
      collisions: 50986
      rx_length_errors: 933
      rx_over_errors: 0
      rx_crc_errors: 84
      rx_frame_errors: 77
      rx_fifo_errors: 0
      rx_missed_errors: 0
      tx_aborted_errors: 0
      tx_carrier_errors: 0
      tx_fifo_errors: 0
      tx_heartbeat_errors: 0
      tx_window_errors: 0
      tx_deferred: 298989
      tx_single_collisions: 34534
      tx_multi_collisions: 6779
      tx_flow_control_pause: 122901
      rx_flow_control_pause: 0
      rx_flow_control_unsupported: 0
      tx_tco_packets: 0
      rx_tco_packets: 0

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.

Re: Network speed Linux-2.6.10

[Ben Greear]

linux-os wrote:
> 
> Conditions:
> 
> Intel NIC e100 device driver. Two identical machines.
> Private network, no other devices. Connected using a Netgear switch.
> Test data is the same thing sent from memory on one machine
> to a discard server on another, using TCP/IP SOCK_STREAM.
> 
> If I set both machines to auto-negotiation OFF and half duplex,
> I get about 9 to 9.5 megabytes/second across the private wire
> network.
> 
> If I set one machine to full duplex and the other to half-duplex
> I get 10 to 11 megabytes/second transfer across the network,
> regardless of direction.

That is asking for all sorts of trouble.

> If I set both machines to auto-negotiation OFF and full duplex,
> I get 300 to 400 kilobytes/second regardless of the direction.

Check for errors in the NICs counters (/proc/net/dev/) in this case.
It appears it is not actually set to full-duplex, or maybe it's
10Mbps-FD.  Use ethtool to see the actual settings.

What happens if you just don't muck with the NIC and let it auto-negotiate
on it's own?

Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Network speed Linux-2.6.10

[Lee Revell]

On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
> What happens if you just don't muck with the NIC and let it auto-negotiate
> on it's own?

This can be asking for trouble too (auto negotiation is often buggy).
What if you hard set them both to 100/full?

Lee

Re: Network speed Linux-2.6.10

[linux-os]

On Tue, 1 Mar 2005, Lee Revell wrote:

> On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
>> What happens if you just don't muck with the NIC and let it auto-negotiate
>> on it's own?
>
> This can be asking for trouble too (auto negotiation is often buggy).
> What if you hard set them both to 100/full?
>
> Lee
>

As previously stated, the through-put is awful.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.

Re: Network speed Linux-2.6.10

[Miquel van Smoorenburg]

In article <1109708691.14272.8.camel@mindpipe>,
Lee Revell  <rlrevell@joe-job.com> wrote:
>On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
>> What happens if you just don't muck with the NIC and let it auto-negotiate
>> on it's own?
>
>This can be asking for trouble too (auto negotiation is often buggy).
>What if you hard set them both to 100/full?

If you do that you also need to force the switchports to full duplex.
Lots of switches default to half-duplex without auto-negotiation.

Mike.

Re: Network speed Linux-2.6.10

[Lee Revell]

On Tue, 2005-03-01 at 20:30 +0000, Miquel van Smoorenburg wrote:
> In article <1109708691.14272.8.camel@mindpipe>,
> Lee Revell  <rlrevell@joe-job.com> wrote:
> >On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
> >> What happens if you just don't muck with the NIC and let it auto-negotiate
> >> on it's own?
> >
> >This can be asking for trouble too (auto negotiation is often buggy).
> >What if you hard set them both to 100/full?
> 
> If you do that you also need to force the switchports to full duplex.
> Lots of switches default to half-duplex without auto-negotiation.
> 

Yup, that was exactly what we ended up doing.

Lee

Re: Network speed Linux-2.6.10

[Ben Greear]

Lee Revell wrote:
> On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
> 
>>What happens if you just don't muck with the NIC and let it auto-negotiate
>>on it's own?
> 
> 
> This can be asking for trouble too (auto negotiation is often buggy).
> What if you hard set them both to 100/full?

I have not noticed any buggy autonegotiation with the e100 driver in several
years...

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Network speed Linux-2.6.10

[Lee Revell]

On Tue, 2005-03-01 at 12:30 -0800, Ben Greear wrote:
> Lee Revell wrote:
> > On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
> > 
> >>What happens if you just don't muck with the NIC and let it auto-negotiate
> >>on it's own?
> > 
> > 
> > This can be asking for trouble too (auto negotiation is often buggy).
> > What if you hard set them both to 100/full?
> 
> I have not noticed any buggy autonegotiation with the e100 driver in several
> years...
> 

Sorry, I misread the post.  He tried this.

I was under the impression this was due to inconsistent implementation
of autonegotiation in hardware.  When I was an ISP sysadmin we had this
problem with various devices (Cisco switches, Linux and BSD/OS
machines).  A device would get power cycled and one side would come up
100/full, the other 100/half.   We ended up hard setting everything.

Lee

Re: Network speed Linux-2.6.10

[venom]

not the e100 driver, but some switch, (e.g. some matrix) has a buggy 
autonegotiation.

On Tue, 1 Mar 2005, Ben Greear wrote:

> Lee Revell wrote:
>> On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
>> 
>>> What happens if you just don't muck with the NIC and let it auto-negotiate
>>> on it's own?
>> 
>> 
>> This can be asking for trouble too (auto negotiation is often buggy).
>> What if you hard set them both to 100/full?
>
> I have not noticed any buggy autonegotiation with the e100 driver in several
> years...
>
> Ben
>
> -- 
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Network speed Linux-2.6.10

[venom]

I wonder if switch ports are configured as 100FDX auto=off or
100HDX auto=off.
from the report I saw it seems that switch ports are 100HDX auto=off 
instead of 100FDX auto=off.

On Tue, 1 Mar 2005, Lee Revell wrote:

> On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
>> What happens if you just don't muck with the NIC and let it auto-negotiate
>> on it's own?
>
> This can be asking for trouble too (auto negotiation is often buggy).
> What if you hard set them both to 100/full?
>
> Lee
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Network speed Linux-2.6.10

[Horst von Brand]

Lee Revell <rlrevell@joe-job.com> said:
> On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:

[...]

> > What happens if you just don't muck with the NIC and let it auto-negotiate
> > on it's own?

> This can be asking for trouble too (auto negotiation is often buggy).

I'be seen much more broken networks than buggy autonegotiation. If they
negotiate something funny, check and fix the network.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: Network speed Linux-2.6.10

[linux-os]

On Wed, 2 Mar 2005, Horst von Brand wrote:

> Lee Revell <rlrevell@joe-job.com> said:
>> On Tue, 2005-03-01 at 12:20 -0800, Ben Greear wrote:
>
> [...]
>
>>> What happens if you just don't muck with the NIC and let it auto-negotiate
>>> on it's own?
>
>> This can be asking for trouble too (auto negotiation is often buggy).
>
> I'be seen much more broken networks than buggy autonegotiation. If they
> negotiate something funny, check and fix the network.
> --

In this case it's going to negotiate with the exact same device
in the other box when connected with a X-over cable, and with
a Netgear switch on my desk when not. In both cases, I have
complete control of the "network".

FYI I just upgraded to Linux-2.6.11 I'm going to repeat my
experiment(s) later today after I put the same kernel on
my other machine.

> Dr. Horst H. von Brand                   User #22616 counter.li.org
> Departamento de Informatica                     Fono: +56 32 654431
> Universidad Tecnica Federico Santa Maria              +56 32 654239
> Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.11 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.

Re: Network speed Linux-2.6.10

[linux-os]

On Tue, 1 Mar 2005, Ben Greear wrote:

> linux-os wrote:
>> 
>> Conditions:
>> 
>> Intel NIC e100 device driver. Two identical machines.
>> Private network, no other devices. Connected using a Netgear switch.
>> Test data is the same thing sent from memory on one machine
>> to a discard server on another, using TCP/IP SOCK_STREAM.
>> 
>> If I set both machines to auto-negotiation OFF and half duplex,
>> I get about 9 to 9.5 megabytes/second across the private wire
>> network.
>> 
>> If I set one machine to full duplex and the other to half-duplex
>> I get 10 to 11 megabytes/second transfer across the network,
>> regardless of direction.
>
> That is asking for all sorts of trouble.
>
>> If I set both machines to auto-negotiation OFF and full duplex,
>> I get 300 to 400 kilobytes/second regardless of the direction.
>
> Check for errors in the NICs counters (/proc/net/dev/) in this case.
> It appears it is not actually set to full-duplex, or maybe it's
> 10Mbps-FD.  Use ethtool to see the actual settings.
>

I supplied the actual settings.

> What happens if you just don't muck with the NIC and let it auto-negotiate
> on it's own?
>

It goes to half-duplex and runs 9 to 9.5 megabytes/second as stated
above.

That's why I think 1/2 duplex is __really__ full-duplex.

> Ben
>
>
> -- 
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.

Re: Network speed Linux-2.6.10

[Ben Greear]

linux-os wrote:
> On Tue, 1 Mar 2005, Ben Greear wrote:

> I supplied the actual settings.
> 
>> What happens if you just don't muck with the NIC and let it 
>> auto-negotiate
>> on it's own?
>>
> 
> It goes to half-duplex and runs 9 to 9.5 megabytes/second as stated
> above.
> 
> That's why I think 1/2 duplex is __really__ full-duplex.

half-duplex will run near line speed in one direction.  Try sending in
both directions at the same time and you'll get closer to 40% of the
link utilization...  Also, if you see any collisions you are in half-duplex
mode.

You could try connecting the NICs back-to-back with a cross-over
cable to take your switch out of the loop?

Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Network speed Linux-2.6.10

[Paul Dickson]

On Tue, 1 Mar 2005 14:29:24 -0500 (EST), linux-os wrote:

> Intel NIC e100 device driver. Two identical machines.
> Private network, no other devices. Connected using a Netgear switch.
> Test data is the same thing sent from memory on one machine
> to a discard server on another, using TCP/IP SOCK_STREAM.
> 
> If I set both machines to auto-negotiation OFF and half duplex,
> I get about 9 to 9.5 megabytes/second across the private wire
> network.
> 
> If I set one machine to full duplex and the other to half-duplex
> I get 10 to 11 megabytes/second transfer across the network,
> regardless of direction.
> 
> If I set both machines to auto-negotiation OFF and full duplex,
> I get 300 to 400 kilobytes/second regardless of the direction.

Might this be related to the broken BicTCP implementations in the 2.6.6+
kernels?  A fix was added around 2.6.11-rc3 or 4.

	-Paul

Re: Network speed Linux-2.6.10

[Baruch Even]

Paul Dickson wrote:
> On Tue, 1 Mar 2005 14:29:24 -0500 (EST), linux-os wrote:
>>Intel NIC e100 device driver. Two identical machines.
>>Private network, no other devices. Connected using a Netgear switch.
>>Test data is the same thing sent from memory on one machine
>>to a discard server on another, using TCP/IP SOCK_STREAM.
>>
>>If I set both machines to auto-negotiation OFF and half duplex,
>>I get about 9 to 9.5 megabytes/second across the private wire
>>network.
>>
>>If I set one machine to full duplex and the other to half-duplex
>>I get 10 to 11 megabytes/second transfer across the network,
>>regardless of direction.
>>
>>If I set both machines to auto-negotiation OFF and full duplex,
>>I get 300 to 400 kilobytes/second regardless of the direction.
> 
> Might this be related to the broken BicTCP implementations in the 2.6.6+
> kernels?  A fix was added around 2.6.11-rc3 or 4.

Unlikely, the problem with BIC would have shown itself only at high 
speeds over long latency links, not over a lan connection.

Baruch

Re: Network speed Linux-2.6.10

[Paul Dickson]

On Wed, 02 Mar 2005 01:02:50 +0000, Baruch Even wrote:

> > Might this be related to the broken BicTCP implementations in the 2.6.6+
> > kernels?  A fix was added around 2.6.11-rc3 or 4.
> 
> Unlikely, the problem with BIC would have shown itself only at high 
> speeds over long latency links, not over a lan connection.

I only mentioned the possibility because I saw the same profile given by
the PDF (the link was mentioned in the patch) while downloading gnoppix
via my cable modem.  The oscillations of speed varied from 40K to 500+K.
The average ended up around 270K.  (I was using wget for the download).

	-Paul

Re: Network speed Linux-2.6.10

[Baruch Even]

Paul Dickson wrote:
> On Wed, 02 Mar 2005 01:02:50 +0000, Baruch Even wrote:
> 
>>>Might this be related to the broken BicTCP implementations in the 2.6.6+
>>>kernels?  A fix was added around 2.6.11-rc3 or 4.
>>
>>Unlikely, the problem with BIC would have shown itself only at high 
>>speeds over long latency links, not over a lan connection.
> 
> I only mentioned the possibility because I saw the same profile given by
> the PDF (the link was mentioned in the patch) while downloading gnoppix
> via my cable modem.  The oscillations of speed varied from 40K to 500+K.
> The average ended up around 270K.  (I was using wget for the download).

If it is indeed BIC than we have a bug where it doesn't shut itself off 
for low latencies. Since we don't test this case extensively here (we 
work to improve high-speed and just make sure we don't ruin slower 
speeds) I can't say it's impossible, try turning BIC off and see if it 
helps.

Due to the scenario that the OP gave it is more likely something to do 
with auto-detection somewhere along the way or a driver bug. It is also 
possible that I'm mistaken and it is BIC, never hurts to check.

Baruch