BCM5704 performance questions.

BCM5704 performance questions.

[Ben Greear]

Hello!

I have a 4-port NIC by silicom-usa.com that uses the BCM5704 (rev 10) chipset.

It's running in a PCI-X bus (100 or 133Mhz).  CPUs are dual xeon 2.8Ghz,
1MB cache, 1GB RAM, etc).  Kernel is 2.6.11 + my hacks (no hacks to tg3, minor
hacks to e1000 and other parts of the networking stacks).

I am trying to bridge as much traffic as possible across two interfaces,
using a proprietary kernel module.
The network traffic is 1514 byte packets, generated by a modified version of
pktgen running on another machine with similar hardware (Intel NICs).

With the BCM NIC I can get about 600Mbps in one direction and about 800Mbps
in the other..with a great deal of dropped packets.  With the Intel 4-port
NIC (same machine, different PCI slot, and also from Silicom-usa.com)
I can get 900+Mbps in both directions with virtually no drops.

So:

*  Is the BCM5704 chipset/driver really that much slower?

*  Is there some information on tuning the tg3 somewhere?
    (I didn't see a Documentation/networking/tg3.txt file, for instance)

*  Is there a way to verify the bus speed that the NIC is running at?
    (ethtool -d ethX gives lots of meaningless (to me) hex)

Please let me know if more information would be useful.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: BCM5704 performance questions.

[Michael Chan]

On Thu, 2005-06-09 at 17:38 -0700, Ben Greear wrote:

> 
> *  Is the BCM5704 chipset/driver really that much slower?
> 

Unfortunately, the 5704 requires the "ONE_DMA" workaround which will
limit throughput in a PCIX 100/133 bus. If you comment out the line that
sets the DMA_RWCTRL_ONE_DMA flag in tg3.c, you should see improved
performance. However, you may run into some DMA issues on certain
systems.

> *  Is there some information on tuning the tg3 somewhere?
>     (I didn't see a Documentation/networking/tg3.txt file, for instance)
> 
> *  Is there a way to verify the bus speed that the NIC is running at?
>     (ethtool -d ethX gives lots of meaningless (to me) hex)
> 

tg3 probing string for each device will tell you the bus type, width,
and speed.

Re: BCM5704 performance questions.

[Ben Greear]

Michael Chan wrote:
> On Thu, 2005-06-09 at 17:38 -0700, Ben Greear wrote:
> 
>>*  Is the BCM5704 chipset/driver really that much slower?
>>
> Unfortunately, the 5704 requires the "ONE_DMA" workaround which will
> limit throughput in a PCIX 100/133 bus. If you comment out the line that
> sets the DMA_RWCTRL_ONE_DMA flag in tg3.c, you should see improved
> performance. However, you may run into some DMA issues on certain
> systems.

Is there any way I can tell which systems are affected?  It won't be
an option for me to purposefully ship possibly busted drivers/hardware,
but if I can be certain that my systems are immune, I will try this
modification.

>>*  Is there some information on tuning the tg3 somewhere?
>>    (I didn't see a Documentation/networking/tg3.txt file, for instance)
>>
>>*  Is there a way to verify the bus speed that the NIC is running at?
>>    (ethtool -d ethX gives lots of meaningless (to me) hex)
>>
> 
> 
> tg3 probing string for each device will tell you the bus type, width,
> and speed.

Thanks, I found it.

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: BCM5704 performance questions.

[Michael Chan]

On Thu, 2005-06-09 at 18:24 -0700, Ben Greear wrote:
> Michael Chan wrote:
> > 
> > Unfortunately, the 5704 requires the "ONE_DMA" workaround which will
> > limit throughput in a PCIX 100/133 bus. If you comment out the line that
> > sets the DMA_RWCTRL_ONE_DMA flag in tg3.c, you should see improved
> > performance. However, you may run into some DMA issues on certain
> > systems.
> 
> Is there any way I can tell which systems are affected?  It won't be
> an option for me to purposefully ship possibly busted drivers/hardware,
> but if I can be certain that my systems are immune, I will try this
> modification.
> 
I mentioned this so that you could verify that the slow performance was
indeed caused by ONE_DMA. Even if your system is affected, it's a very
subtle problem that won't show up right away and should allow you to get
some performance numbers.

Unfortunately, if indeed it is ONE_DMA, there is no easy way for us to
tell which system is affected. And the recommendation is to turn it on
for all 5704 in PCIX 100/133.

Re: BCM5704 performance questions.

[Ben Greear]

Michael Chan wrote:
> On Thu, 2005-06-09 at 18:24 -0700, Ben Greear wrote:
> 
>>Michael Chan wrote:
>>
>>>Unfortunately, the 5704 requires the "ONE_DMA" workaround which will
>>>limit throughput in a PCIX 100/133 bus. If you comment out the line that
>>>sets the DMA_RWCTRL_ONE_DMA flag in tg3.c, you should see improved
>>>performance. However, you may run into some DMA issues on certain
>>>systems.
>>
>>Is there any way I can tell which systems are affected?  It won't be
>>an option for me to purposefully ship possibly busted drivers/hardware,
>>but if I can be certain that my systems are immune, I will try this
>>modification.
>>
> 
> I mentioned this so that you could verify that the slow performance was
> indeed caused by ONE_DMA. Even if your system is affected, it's a very
> subtle problem that won't show up right away and should allow you to get
> some performance numbers.

I commented out the code and ran the pktgen test again.  It may be a small
bit better, but not much:  770Mbps in one direction, 750Mbps in the other.

Have you done any tests with 2 tg3 NICs in a single machine to see if they
can run at or near line speed (full duplex)?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: BCM5704 performance questions.

[Michael Chan]

On Fri, 2005-06-10 at 14:09 -0700, Ben Greear wrote:

> I commented out the code and ran the pktgen test again.  It may be a small
> bit better, but not much:  770Mbps in one direction, 750Mbps in the other.
> 

The latest tg3 driver will print out the dma_rwctrl at probe time. Can
you check the value to make sure ONE_DMA is disabled? Bit 14 sets
ONE_DMA.

> Have you done any tests with 2 tg3 NICs in a single machine to see if they
> can run at or near line speed (full duplex)?
> 

Yes, we have years ago but not with the tg3 driver. We set up the 5704
to bridge (or route, I don't remember) one port to the other. It cannot
bridge at line-rate with the ONE_DMA workaround enabled.

Re: BCM5704 performance questions.

[Ben Greear]

Michael Chan wrote:
> On Fri, 2005-06-10 at 14:09 -0700, Ben Greear wrote:

>>I commented out the code and ran the pktgen test again.  It may be a small
>>bit better, but not much:  770Mbps in one direction, 750Mbps in the other.
>>

> The latest tg3 driver will print out the dma_rwctrl at probe time. Can
> you check the value to make sure ONE_DMA is disabled? Bit 14 sets
> ONE_DMA.

I don't see any printout in /var/log/messages that looks like it
relates to the dma_rwctrl.  I'm using the driver in 2.6.11..maybe
it is not recent enough to print this information out?

>>Have you done any tests with 2 tg3 NICs in a single machine to see if they
>>can run at or near line speed (full duplex)?
>>

> Yes, we have years ago but not with the tg3 driver. We set up the 5704
> to bridge (or route, I don't remember) one port to the other. It cannot
> bridge at line-rate with the ONE_DMA workaround enabled.

Ok.  Do you have other chipsets/NICs that can handle faster GigE speeds?

If you'd like to send me some multi-port NICs to play with I'll be happy
to report my findings :)

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: BCM5704 performance questions.

[David S. Miller]

From: Ben Greear <greearb@candelatech.com>
Date: Fri, 10 Jun 2005 15:35:35 -0700

> Michael Chan wrote:
> > On Fri, 2005-06-10 at 14:09 -0700, Ben Greear wrote:
> 
> >>I commented out the code and ran the pktgen test again.  It may be a small
> >>bit better, but not much:  770Mbps in one direction, 750Mbps in the other.
> >>
> 
> > The latest tg3 driver will print out the dma_rwctrl at probe time. Can
> > you check the value to make sure ONE_DMA is disabled? Bit 14 sets
> > ONE_DMA.
> 
> I don't see any printout in /var/log/messages that looks like it
> relates to the dma_rwctrl.  I'm using the driver in 2.6.11..maybe
> it is not recent enough to print this information out?

Yes, your driver is too old.  The one in Linus's current GIT
tree has this, along with many other enhancements.

Re: BCM5704 performance questions.

[Rick Jones]

> Have you done any tests with 2 tg3 NICs in a single machine to see if they
> can run at or near line speed (full duplex)?

It isn't just a question of two tg3 NICs in the same box is it?  You are running 
two NICs on the same bus right?  And unless my dimm memory is mistaken, four 
ports on a card with 5704s means two 5704's a bridge chip right?  So, it would 
be two tg3 NICs going through the same bridge chip, not just the same bus or 
same system.  I'd be worrying about DMA latencies on the system and the bridge 
chip, and perhaps the efficiency of the PCI-X bus usage (not sure - is there 
anything in your system's chipset to extract that sort of information?)

What happens when you turn pktgen around/insideout and source packets from the 
bridging system to each of the (two other?) systems?

Since you are bridging, does having CKO enabled really matter?  Mightn't that 
allow the firmware on the 5704(s) to run a triffle faster?  Or does bridging 
already not request CKO (I suppose it might).

Are your interface interrupts distributed across the CPUs?

rick jones

Re: BCM5704 performance questions.

[Ben Greear]

Rick Jones wrote:
> 
>> Have you done any tests with 2 tg3 NICs in a single machine to see if 
>> they
>> can run at or near line speed (full duplex)?
> 
> 
> It isn't just a question of two tg3 NICs in the same box is it?  You are 
> running two NICs on the same bus right?  And unless my dimm memory is 
> mistaken, four ports on a card with 5704s means two 5704's a bridge chip 
> right?  So, it would be two tg3 NICs going through the same bridge chip, 
> not just the same bus or same system.  I'd be worrying about DMA 
> latencies on the system and the bridge chip, and perhaps the efficiency 
> of the PCI-X bus usage (not sure - is there anything in your system's 
> chipset to extract that sort of information?)

There will be a bridge chip, and indeed I see better performance when I just
use a 2-port Intel NIC as opposed to a 4 port, even if I am only actively
using 2 of the 4 ports on the 4-port NIC.  For the tg3 hardware I only
have a 4-port NIC.  I do assume that a 2-port tg3 NIC w/out a bridge chip
would be faster..but probably not too much.

> What happens when you turn pktgen around/insideout and source packets 
> from the bridging system to each of the (two other?) systems?

I looped two ports on the same NIC together for the pktgen tests, so there
is only a single machine in question.  With Intel I can source/sink about
960Mbps on two ports simultaneously in this configuration.  With the tg3
NIC I can only do about 750Mbps.

And, the tg3 is in the faster PCI-X slot (133Mhz v/s 100Mhz).  So, to me
it appears that the tg3 hardware and/or driver can only handle about 80%
of the performance that the intel e1000 can produce.  It's possible I have
a particularly sub-optimal configuration for tg3, or maybe a poorly designed
NIC, which is why I'd like to know what others see...

> Since you are bridging, does having CKO enabled really matter?  Mightn't 
> that allow the firmware on the 5704(s) to run a triffle faster?  Or does 
> bridging already not request CKO (I suppose it might).

CKO == IP checksum offload?

Since Dave doesn't want to debug my bridge setup (and I don't blame him),
I am going to try to focus my testing/debug reports on the pktgen tests.
If/when pktgen shows better performance with tg3, I can verify that
I see the same speedups with my proprietary bridging module.  I've no idea
if CKO would help or hinder pktgen, nor have I tried to enable or disable it.

> Are your interface interrupts distributed across the CPUs?

I'm using FC2, basically a default install.  It does seem to have
an irq balance daemon running.  But, I'm not specifically binding IRQs
or anything like that.  pktgen tx is running as a single thread, so the rx
code could run mostly on the other CPU if locking allows...

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: BCM5704 performance questions.

[Rick Jones]

> There will be a bridge chip, and indeed I see better performance when I just 
> use a 2-port Intel NIC as opposed to a 4 port, even if I am only actively 
> using 2 of the 4 ports on the 4-port NIC.  For the tg3 hardware I only have a
> 4-port NIC.  I do assume that a 2-port tg3 NIC w/out a bridge chip would be
> faster..but probably not too much.

I have been taught by several wise old engineers that the proper spelling of 
assume is ass-u-me :)

Bridge chips can in theory do all sorts of nasty things to performance.

> CKO == IP checksum offload?


Yes.


> Since Dave doesn't want to debug my bridge setup (and I don't blame him), I
> am going to try to focus my testing/debug reports on the pktgen tests. 
> If/when pktgen shows better performance with tg3, I can verify that I see the
> same speedups with my proprietary bridging module.  I've no idea if CKO would
> help or hinder pktgen, nor have I tried to enable or disable it.
> 
>> Are your interface interrupts distributed across the CPUs?
> 
> 
> I'm using FC2, basically a default install.  It does seem to have an irq
> balance daemon running.  But, I'm not specifically binding IRQs or anything
> like that.  pktgen tx is running as a single thread, so the rx code could run
> mostly on the other CPU if locking allows...

again, never ass-u-me.

rick

Re: BCM5704 performance questions.

[Ben Greear]

Rick Jones wrote:
> 
>> There will be a bridge chip, and indeed I see better performance when 
>> I just use a 2-port Intel NIC as opposed to a 4 port, even if I am 
>> only actively using 2 of the 4 ports on the 4-port NIC.  For the tg3 
>> hardware I only have a
>> 4-port NIC.  I do assume that a 2-port tg3 NIC w/out a bridge chip 
>> would be
>> faster..but probably not too much.
> 
> 
> I have been taught by several wise old engineers that the proper 
> spelling of assume is ass-u-me :)
> 
> Bridge chips can in theory do all sorts of nasty things to performance.

Sure...but the end result is that I need 2 port NICs and I need 4 port NICs.
The 4-port ones need a bridge, so I'm stuck with a bridge.  The Intel 4-port
with a bridge works OK, the tg3 not so good.  If someone else has a 2-port
BCM NIC that handles full line speed, then that would be a good data point,
but I'm unlikely to purchase one just to satisfy my curiousity.  If someone
wants to send me one, I'll happily stick it in my system and report the results.

>> I'm using FC2, basically a default install.  It does seem to have an irq
>> balance daemon running.  But, I'm not specifically binding IRQs or 
>> anything
>> like that.  pktgen tx is running as a single thread, so the rx code 
>> could run
>> mostly on the other CPU if locking allows...
> 
> again, never ass-u-me.

I'm not assuming anything here...just reporting the setup.

Truth is, the e1000 works really well for my application in most
configurations.  It was interesting for me to learn that
some folks are getting very good tg3 performance for TCP
transfers when the e1000 was dropping frames (see thread
from the last week or so).  So, I was a little supprised that
I did not see such good tg3 numbers.

If the answer is that the tg3 just can't do it, no shame there...but
if my testing can help the tg3 driver improve, I will try to do my
part.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: BCM5704 performance questions.

[Jason Lunz]

mchan@broadcom.com said:
> On Thu, 2005-06-09 at 17:38 -0700, Ben Greear wrote:
>
>> 
>> *  Is the BCM5704 chipset/driver really that much slower?
>> 
>
> Unfortunately, the 5704 requires the "ONE_DMA" workaround which will
> limit throughput in a PCIX 100/133 bus. If you comment out the line that
> sets the DMA_RWCTRL_ONE_DMA flag in tg3.c, you should see improved
> performance. However, you may run into some DMA issues on certain
> systems.
>
>> *  Is there some information on tuning the tg3 somewhere?
>>     (I didn't see a Documentation/networking/tg3.txt file, for instance)
>> 
>> *  Is there a way to verify the bus speed that the NIC is running at?
>>     (ethtool -d ethX gives lots of meaningless (to me) hex)
>> 
>
> tg3 probing string for each device will tell you the bus type, width,
> and speed.

The patch below from http://article.gmane.org/gmane.linux.network/18734
does this too for e1000. Lennert Buytenhek posted it a while back.

Jason


diff -urpN linux-pf/drivers/net/e1000/e1000_main.c linux-bs/drivers/net/e1000/e1000_main.c
--- linux-pf/drivers/net/e1000/e1000_main.c	Fri Apr  8 15:06:34 2005
+++ linux-bs/drivers/net/e1000/e1000_main.c	Fri Apr  8 15:29:34 2005
@@ -617,6 +617,21 @@ e1000_probe(struct pci_dev *pdev,
 	if(eeprom_data & eeprom_apme_mask)
 		adapter->wol |= E1000_WUFC_MAG;
 
+	/* print bus type/speed/width info */
+	printk(KERN_INFO "%s: e1000 (PCI%s:%s:%s) ", netdev->name,
+		((adapter->hw.bus_type == e1000_bus_type_pcix) ? "X" : ""),
+		((adapter->hw.bus_speed == e1000_bus_speed_133) ? "133MHz" :
+		 (adapter->hw.bus_speed == e1000_bus_speed_120) ? "120MHz" :
+		 (adapter->hw.bus_speed == e1000_bus_speed_100) ? "100MHz" :
+		 (adapter->hw.bus_speed == e1000_bus_speed_66) ? "66MHz" :
+		  "33MHz"),
+		((adapter->hw.bus_width == e1000_bus_width_64) ? "64-bit" :
+		  "32-bit"));
+
+	for (i = 0; i < 6; i++)
+		printk("%2.2x%c", netdev->dev_addr[i],
+			i == 5 ? '\n' : ':');
+
 	/* reset the hardware with the new settings */
 	e1000_reset(adapter);
 

Re: BCM5704 performance questions.

[David S. Miller]

From: Ben Greear <greearb@candelatech.com>
Date: Thu, 09 Jun 2005 17:38:22 -0700

> I am trying to bridge as much traffic as possible across two interfaces,
> using a proprietary kernel module.

Ben, I'm going to just mention that I'm not going to
look into your bug report.  You consistently come here
without a test case or setup that other developers can
use to reproduce or investigate your problem.  You always
report things with your proprietary setup of this or that.

You have our code, we don't have your's.

Re: BCM5704 performance questions.

[Ben Greear]

David S. Miller wrote:
> From: Ben Greear <greearb@candelatech.com>
> Date: Thu, 09 Jun 2005 17:38:22 -0700
> 
> 
>>I am trying to bridge as much traffic as possible across two interfaces,
>>using a proprietary kernel module.
> 
> 
> Ben, I'm going to just mention that I'm not going to
> look into your bug report.  You consistently come here
> without a test case or setup that other developers can
> use to reproduce or investigate your problem.  You always
> report things with your proprietary setup of this or that.
> 
> You have our code, we don't have your's.

Fair enough.

I ran a test using pktgen to (try to) send 82kpps, 1514 byte packets between
two ports on the tg3 NIC.  It can do about 780Mbps in one direction,
and 880Mbps in the other direction.  Lots of harmless hard-start xmit errors reported
(tg3 may not stop it's tx queue correctly, or maybe pktgen is screwed up since
e1000 reports similar errors).
Intel e1000 can do about 960Mbps in both directions.

My pktgen is modified.  You can find my full patch against 2.6.11 here if you so wish:

http://www.candelatech.com/oss/candela_2.6.11.patch

If you need the exact arguments I used to configure pktgen, I can get those
for you as well.

I found the Mhz printout (thanks Michael!)

The tg3 NIC is in the 133Mhz slot.  That probably means the intel NIC is only
running at 100Mhz.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: BCM5704 performance questions.

[David S. Miller]

From: Ben Greear <greearb@candelatech.com>
Date: Thu, 09 Jun 2005 18:20:33 -0700

> I ran a test using pktgen to (try to) send 82kpps, 1514 byte packets between
> two ports on the tg3 NIC.  It can do about 780Mbps in one direction,
> and 880Mbps in the other direction.  Lots of harmless hard-start xmit errors reported
> (tg3 may not stop it's tx queue correctly, or maybe pktgen is screwed up since
> e1000 reports similar errors).

There is a known race on SMP with drivers using NETIF_F_LLTX that is
still not fixed.  It will cause the error message to be reported from
the driver's ->hard_start_xmit() routine when you hit this race.

The e1000 driver hits the same race, it just doesn't print any
message.

You're definitely on an SMP system if you are triggering that message.

Re: BCM5704 performance questions.

[Ben Greear]

David S. Miller wrote:
> From: Ben Greear <greearb@candelatech.com>
> Date: Thu, 09 Jun 2005 18:20:33 -0700
> 
> 
>>I ran a test using pktgen to (try to) send 82kpps, 1514 byte packets between
>>two ports on the tg3 NIC.  It can do about 780Mbps in one direction,
>>and 880Mbps in the other direction.  Lots of harmless hard-start xmit errors reported
>>(tg3 may not stop it's tx queue correctly, or maybe pktgen is screwed up since
>>e1000 reports similar errors).
> 
> 
> There is a known race on SMP with drivers using NETIF_F_LLTX that is
> still not fixed.  It will cause the error message to be reported from
> the driver's ->hard_start_xmit() routine when you hit this race.

Ok, it's not a big deal, since I can just retry the packet when
the tx queue wakes up again.  The message was coming from pktgen, btw.

> You're definitely on an SMP system if you are triggering that message.

Yes, dual xeon with HT as well.


Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com