Poor 8260 FCC Ethernet performance

Poor 8260 FCC Ethernet performance

[Allen Curtis]

I have successfully booted our board using an EST like configuration.
(yeah...) Then I started testing Ethernet performance and came across
something interesting.

1. Tested FTP transfers with a dumb 10T hub and got about 444.6 kBps
2. Swapped the 10T hub for a 100T switch and the performance went down to
41.7kBps!!

I do not understand what is happening. Went from an environment that was
slow and had many collisions to a much faster, no collision environment and
got majorly penalized on performance? It is almost like the error handling
helped to keep things moving. The 100T switch produced very bursty
performance where the 10T was much more consistent.

Source version:  linuxppc_2_4_devel - less than a week old
MII PHY option:  disabled in the kernel configuration

Has anyone experience this? Ideas on how to track down the problem?

BTW: Large FTP transfers work. (which was a problem with HHL 2.0) I assume
that this has to due with the thread safe page table fixes.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Poor 8260 FCC Ethernet performance

[Jean-Denis Boyer]

Allen,

The 8260 Ethernet controller needs to know about the duplex mode in which
the PHY operates. The driver sets it to half duplex by default, and needs
the feed back of the PHY to set it to full duplex.

As you may know, the hub always operates in half duplex, while the switch
usually operates in full duplex, if it is directly connected to your unit,
and your PHY accepts to negociate to full duplex.

Both the Ethernet controller and the PHY should be set to the same mode, or
your likely to run into communication problems (as you see ;-).

Regards,

--------------------------------------------
 Jean-Denis Boyer, B.Eng., System Architect
 Mediatrix Telecom Inc.
 4229 Garlock Street
 Sherbrooke (Québec)
 J1L 2C8  CANADA
 (819)829-8749 x241
--------------------------------------------

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Poor 8260 FCC Ethernet performance

[Allen Curtis]

> The 8260 Ethernet controller needs to know about the duplex mode in which
> the PHY operates. The driver sets it to half duplex by default, and needs
> the feed back of the PHY to set it to full duplex.
>
> As you may know, the hub always operates in half duplex, while the switch
> usually operates in full duplex, if it is directly connected to your unit,
> and your PHY accepts to negociate to full duplex.
>

Yes, the source code has been modified to force full-duplex.
(fcc_restart(dev,1) if I remember correctly :) In fact, before this change
was made Ethernet communications did not work at all. (At least it would not
get past bootp) I searched for something that looked like it was speed
related but found nothing. (Other than the MII reporting)

Other ideas?

BTW: We are using a BCM switch for our PHY. The interface to the processor
is always full-duplex, 100BT regardless of the external connections. Hence
the no MII PHY option.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Poor 8260 FCC Ethernet performance

[Conn Clark]

Allen Curtis wrote:
>
> > The 8260 Ethernet controller needs to know about the duplex mode in which
> > the PHY operates. The driver sets it to half duplex by default, and needs
> > the feed back of the PHY to set it to full duplex.
> >
> > As you may know, the hub always operates in half duplex, while the switch
> > usually operates in full duplex, if it is directly connected to your unit,
> > and your PHY accepts to negociate to full duplex.
> >
>
> Yes, the source code has been modified to force full-duplex.
> (fcc_restart(dev,1) if I remember correctly :) In fact, before this change
> was made Ethernet communications did not work at all. (At least it would not
> get past bootp) I searched for something that looked like it was speed
> related but found nothing. (Other than the MII reporting)
>
> Other ideas?

Could it be an analog problem? Have you looked at the signals comming in on
a scope?


>
> BTW: We are using a BCM switch for our PHY. The interface to the processor
> is always full-duplex, 100BT regardless of the external connections. Hence
> the no MII PHY option.

Let me get this straight, You have no PHY, no magnet, no connector? If this
is the case are you sure your voltages on the I/O compatible?

I have thought about building a Beowolf cluster in this manner out of 8260s
just to impress my geek friends and colleages. Of course I don't have the
time or money to do it, but its fun to think about.

Good Luck

	Conn

--

*****************************************************************
  If you live at home long enough, your parents will move out.
*****************************************************************

Conn Clark
Engineering Stooge				clark@esteem.com
Electronic Systems Technology Inc.		www.esteem.com

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Poor 8260 FCC Ethernet performance

[Allen Curtis]

> Could it be an analog problem? Have you looked at the signals
> comming in on
> a scope?
>

Looking at the signal with a scope is probably a good idea. Here is an
update from today's fun.

1. These is an error in the fcc_enet.c file which causes NO and CRC errors
when running in full-duplex mode. fcc_restart() should be updated to include
FCC_PSMR_LPB when configuring full-duplex mode. (cleared when not)

2. Some relative performance measurements (HHL 2.4.2 vs. 2.4.19pre9)
			10T Hub	|	100BT switch
		 -------------------------------------|
	2.4.2	 |	410KBps	|	750KBps	  |
		 -------------------------------------|
	2.4.19 |	440KBps	|	190KBps	  |
		 --------------------------------------

	RedHat 2.4.18-3 (x86)	3900KBps

3. TOP shows as much as 48% system utilization during a single FTP transfer.
With the fix mentioned in #1, there are 0 errors reported by ifconfig.

I may be missing something but I believe that these numbers are going the
wrong way. There is still the question of performance decrease when using a
100BT switch. Signal intregety can not be the only suspect considering the
performance did increase using an older version of the kernel. (although
nothing like the Workstation performance)


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Poor 8260 FCC Ethernet performance

[Adisorn Ermongkonchai]

Hi Allen and Linuxppc-embedded guru,

Referring to your 31 May 2002 post to LinuxPPC about
poor 8260 FCC Ether performance.  Did you figure
out why is the FCC so slow?  We have encountered this
problem as well. We use the newer MPC8270 with 400MHz
clock and 200MHz CPM speed with BCM5222 PHY.  It
ended up running at only ~5Mbps on 100Mbps Ethernet
line.  We just tried 1 FCC.

Does anyone on this post see the same problem? We use
linux 2.4.18 with PPC pqii driver (somehow the other
driver didn't seem to work for all 3 FCC's)  We
also tried on Motorola 8260 eval board and the result
is the same.  We also tried MEN Micro F6, it gave
better result from 11MBps to 30Mbps.  We also tried
just uboot to eliminate Linux out of the picture and
the result was about the same.

I was about to give up on this MPC8270 then I saw this
post so I thought I try.  I cannot believe that
Motorola would make a chip that support 3 FCC's
but can only run at this ridiculous speed.  I am
software engineer, this could be hardware.

Can anyone help me on this?  I'm about to have to
change the processor to PowerPC 4xx.

Thanks in advance.

Regards,
Adisorn Ermongkonchai


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Poor 8260 FCC Ethernet performance

[Dayton, Dean]

It's been awhile, but I seem to remember that some of the PPC drivers did
not set up full duplex properly. Make sure that fcc_fpsmr is setting
FCC_PSMR_FDE and FCC_PSMR_LPB properly. Also check to make sure that the phy
is actually set up for full duplex.


Having said all that. There is a limit to how fast the processor can forward
packets. This chip was designed to give a lot of flexibility in
connectivity. It will not forward 100Mbps of small packets, especially not
while running Linux. You could spend a lot of time and really optimize the
data path and drivers for a very significant improvement in performance. But
then you wouldn't have the flexibility of Linux.

I run a 266MHz 8265 with Linux. With a very simple kernel configuration it
can forward about 29000 pps (in on interface and out another). If I enable
iptables,nat and a few other network options, the performance drops by about
50%. If I start configuring complex iptables rule, performance will continue
to drop.

Modifying the driver to NAPI would probably help.

Dean Dayton

> -----Original Message-----
> From: Adisorn Ermongkonchai [mailto:aermongk@yahoo.com]
> Sent: Sunday, June 06, 2004 2:01 PM
> To: acurtis@onz.com
> Cc: linuxppc-embedded@lists.linuxppc.org
> Subject: Poor 8260 FCC Ethernet performance
>
>
>
> Hi Allen and Linuxppc-embedded guru,
>
> Referring to your 31 May 2002 post to LinuxPPC about
> poor 8260 FCC Ether performance.  Did you figure
> out why is the FCC so slow?  We have encountered this
> problem as well. We use the newer MPC8270 with 400MHz
> clock and 200MHz CPM speed with BCM5222 PHY.  It
> ended up running at only ~5Mbps on 100Mbps Ethernet
> line.  We just tried 1 FCC.
>
> Does anyone on this post see the same problem? We use
> linux 2.4.18 with PPC pqii driver (somehow the other
> driver didn't seem to work for all 3 FCC's)  We
> also tried on Motorola 8260 eval board and the result
> is the same.  We also tried MEN Micro F6, it gave
> better result from 11MBps to 30Mbps.  We also tried
> just uboot to eliminate Linux out of the picture and
> the result was about the same.
>
> I was about to give up on this MPC8270 then I saw this
> post so I thought I try.  I cannot believe that
> Motorola would make a chip that support 3 FCC's
> but can only run at this ridiculous speed.  I am
> software engineer, this could be hardware.
>
> Can anyone help me on this?  I'm about to have to
> change the processor to PowerPC 4xx.
>
> Thanks in advance.
>
> Regards,
> Adisorn Ermongkonchai
>
>

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Poor 8260 FCC Ethernet performance

[Pantelis Antoniou]

Dayton, Dean wrote:

>It's been awhile, but I seem to remember that some of the PPC drivers
>did not set up full duplex properly. Make sure that fcc_fpsmr is
>setting FCC_PSMR_FDE and FCC_PSMR_LPB properly. Also check to make sure
>that the phy is actually set up for full duplex.
>
>Having said all that. There is a limit to how fast the processor can
>forward packets. This chip was designed to give a lot of flexibility in
>connectivity. It will not forward 100Mbps of small packets, especially
>not while running Linux. You could spend a lot of time and really
>optimize the data path and drivers for a very significant improvement
>in performance. But then you wouldn't have the flexibility of Linux.
>
>I run a 266MHz 8265 with Linux. With a very simple kernel configuration
>it can forward about 29000 pps (in on interface and out another). If I
>enable iptables,nat and a few other network options, the performance
>drops by about 50%. If I start configuring complex iptables rule,
>performance will continue to drop.

Just to be a complete ass a 66MHz MPC885 with two FECs operating as
a virtual switch is able to keep two 100MBits ports fully saturated.
So I think that the hardware can certainly handle it.

Not under linux though.

>>-----Original Message-----
>>From: Adisorn Ermongkonchai [mailto:aermongk@yahoo.com]
>>Sent: Sunday, June 06, 2004 2:01 PM
>>Subject: Poor 8260 FCC Ethernet performance
>>
>>Referring to your 31 May 2002 post to LinuxPPC about poor 8260 FCC
>>Ether performance. Did you figure out why is the FCC so slow? We have
>>encountered this problem as well. We use the newer MPC8270 with 400MHz
>>clock and 200MHz CPM speed with BCM5222 PHY. It ended up running at
>>only ~5Mbps on 100Mbps Ethernet line. We just tried 1 FCC.
>>
>>Does anyone on this post see the same problem? We use linux 2.4.18
>>with PPC pqii driver (somehow the other driver didn't seem to work for
>>all 3 FCC's) We also tried on Motorola 8260 eval board and the result
>>is the same. We also tried MEN Micro F6, it gave better result from
>>11MBps to 30Mbps. We also tried just uboot to eliminate Linux out of
>>the picture and the result was about the same.
>>
>>I was about to give up on this MPC8270 then I saw this post so I
>>thought I try. I cannot believe that Motorola would make a chip that
>>support 3 FCC's but can only run at this ridiculous speed. I am
>>software engineer, this could be hardware.
>>
>>Can anyone help me on this? I'm about to have to change the processor
>>to PowerPC 4xx.

Regards

Pantelis

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Poor 8260 FCC Ethernet performance

[Dayton, Dean]

Your not being a complete ass;) I believe you. I also suspect that you have
a highly optimized data path and drivers, probably in cache, and probably
not using interupts. I also suspect that you are not saturating it with
minimum size (64 byte) packets.

BTW, the numbers I quoted earlier are roughly the same whether I use the FCC
ethernets or PCI tulip ethernets.

Dean Dayton

> Just to be a complete ass a 66MHz MPC885 with two FECs operating as a
> virtual switch is able to keep two 100MBits ports fully saturated. So
> I think that the hardware can certainly handle it.
>
> Not under linux though.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Poor 8260 FCC Ethernet performance

[Mark Chambers]

Hi Adisorn,

You should probably try monitoring your network with Ethereal or something
similar to see what the problem really is - in other words make sure that
the problem really is packets coming out of your board too slowly.

Mark Chambers

----- Original Message -----
From: "Adisorn Ermongkonchai" <aermongk@yahoo.com>
To: <acurtis@onz.com>
Cc: <linuxppc-embedded@lists.linuxppc.org>
Sent: Sunday, June 06, 2004 2:00 PM
Subject: Poor 8260 FCC Ethernet performance


>
> Hi Allen and Linuxppc-embedded guru,
>
> Referring to your 31 May 2002 post to LinuxPPC about
> poor 8260 FCC Ether performance.  Did you figure
> out why is the FCC so slow?  We have encountered this
> problem as well. We use the newer MPC8270 with 400MHz
> clock and 200MHz CPM speed with BCM5222 PHY.  It
> ended up running at only ~5Mbps on 100Mbps Ethernet
> line.  We just tried 1 FCC.
>
> Does anyone on this post see the same problem? We use
> linux 2.4.18 with PPC pqii driver (somehow the other
> driver didn't seem to work for all 3 FCC's)  We
> also tried on Motorola 8260 eval board and the result
> is the same.  We also tried MEN Micro F6, it gave
> better result from 11MBps to 30Mbps.  We also tried
> just uboot to eliminate Linux out of the picture and
> the result was about the same.
>
> I was about to give up on this MPC8270 then I saw this
> post so I thought I try.  I cannot believe that
> Motorola would make a chip that support 3 FCC's
> but can only run at this ridiculous speed.  I am
> software engineer, this could be hardware.
>
> Can anyone help me on this?  I'm about to have to
> change the processor to PowerPC 4xx.
>
> Thanks in advance.
>
> Regards,
> Adisorn Ermongkonchai
>
>
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Poor 8260 FCC Ethernet performance

[Adisorn Ermongkonchai]

--- Mark Chambers <markc@mail.com> wrote:
> Hi Adisorn,
>
> You should probably try monitoring your network with
> Ethereal or something
> similar to see what the problem really is - in other
> words make sure that
> the problem really is packets coming out of your
> board too slowly.
>
> Mark Chambers
>
It is the packet coming out from the board that is
slow.  But we also found that receiving side is
even slower.  What was the 8260 FCC performance that
you got?  According to Dean's response, he could
forward at 29000 pps and that was much, much
better than ours.

Adisorn.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Poor 8260 FCC Ethernet performance

[VanBaren, Gerald (AGRE)]

> -----Original Message-----
> From: owner-linuxppc-embedded@lists.linuxppc.org
> [mailto:owner-linuxppc-embedded@lists.linuxppc.org]On Behalf
> Of Adisorn
> Ermongkonchai
> Sent: Monday, June 07, 2004 1:21 PM
> To: Mark Chambers
> Cc: linuxppc-embedded@lists.linuxppc.org
> Subject: Re: Poor 8260 FCC Ethernet performance
>
>
>
> --- Mark Chambers <markc@mail.com> wrote:
> > Hi Adisorn,
> >
> > You should probably try monitoring your network with
> > Ethereal or something
> > similar to see what the problem really is - in other
> > words make sure that
> > the problem really is packets coming out of your
> > board too slowly.
> >
> > Mark Chambers
> >
> It is the packet coming out from the board that is
> slow.  But we also found that receiving side is
> even slower.  What was the 8260 FCC performance that
> you got?  According to Dean's response, he could
> forward at 29000 pps and that was much, much
> better than ours.
>
> Adisorn.
>

Hi Adisorn:

Did you check your settings in the FCC Ethernet Mode Register (FPSMR) bits 3 Local Protect Bit (LPB) and 5 Full Duplex Ethernet (FDE)?  In particular, LPB and FDE _MUST_ be set to '1' in full duplex.  This has been a problem in the past with FCC ethernet drivers, including the linux driver.

1) If you have a miss-match on full/half duplex between your switch/hub and/or your PHY and/or your FCC, you will have very poor ethernet performance, very similar to what you are complaining about.  Your Rx may be delayed/suppressed by Tx packets, or you may see lots of late collisions.

2) Another hint I haven't experienced personally (yet :-) is that, if the board has layout problems between the PHY and the 8260, you can get a lot of CRC errors due to noise messing up your transmissions.  I'm guessing that your problem is #1, however.

Question: What is your system setup?
  Hub or switch?
  PHY configuration is full/half? 10/100?
  FCC configuration is full/half, LPB = ? FDE = ?
  Do you see a lot of late collisions in the FCC registers?
  Do you see a lot of CRC failures?

gvb


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Poor 8260 FCC Ethernet performance

[Wojciech Kromer]

<cut>
yeah, and another common mistake is switching tx/rx clocks ;)


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Poor 8260 FCC Ethernet performance

[Rune Torgersen]

Does the FCC performance issue have something to do with the fact that
the current FCC drivers in linux puts the buffer descriptors in main
memory and not in dual port?

>From arch/ppc/8260_io/fcc_enet.c (line 1563)

	/* Allocate space for the buffer descriptors in the DP ram.
	 * These are relative offsets in the DP ram address space.
	 * Initialize base addresses for the buffer descriptors.
	 */
#if 0
	/* I really want to do this, but for some reason it doesn't
	 * work with the data cache enabled, so I allocate from the
	 * main memory instead.
	 */
	i = m8260_cpm_dpalloc(sizeof(cbd_t) * RX_RING_SIZE, 8);
	ep->fen_genfcc.fcc_rbase = (uint)&immap->im_dprambase[i];
	cep->rx_bd_base = (cbd_t *)&immap->im_dprambase[i];

	i = m8260_cpm_dpalloc(sizeof(cbd_t) * TX_RING_SIZE, 8);
	ep->fen_genfcc.fcc_tbase = (uint)&immap->im_dprambase[i];
	cep->tx_bd_base = (cbd_t *)&immap->im_dprambase[i];
#else
	cep->rx_bd_base = (cbd_t *)m8260_cpm_hostalloc(sizeof(cbd_t) *
RX_RING_SIZE, 8);
	ep->fen_genfcc.fcc_rbase = __pa(cep->rx_bd_base);
	cep->tx_bd_base = (cbd_t *)m8260_cpm_hostalloc(sizeof(cbd_t) *
TX_RING_SIZE, 8);
	ep->fen_genfcc.fcc_tbase = __pa(cep->tx_bd_base);
#endif

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Poor 8260 FCC Ethernet performance

[Gary Thomas]

On Wed, 2004-06-09 at 16:19, Rune Torgersen wrote:
> Does the FCC performance issue have something to do with the fact that
> the current FCC drivers in linux puts the buffer descriptors in main
> memory and not in dual port?

I hope not since the latest word from Motorola is that they need to be
in main memory - at least on the latest processors, the CPM DPRAM can't
handle cache bursts (or so I've been told - I have a least one customer
that has experienced this)

--
Gary Thomas <gary@mlbassoc.com>
MLB Associates


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Poor 8260 FCC Ethernet performance

[Rune Torgersen]

BTW, found the problem with that piece of code about 2 seconds after
sending it....
The fcc_rbase &tbase parameters are not given the physical address, but
the virtual one...

Change
 	ep->fen_genfcc.fcc_rbase = (uint)&immap->im_dprambase[i];
To
	ep->fen_genfcc.fcc_rbase = __pa((uint)&immap->im_dprambase[i]);

And
	ep->fen_genfcc.fcc_tbase = (uint)&immap->im_dprambase[i];
To
	ep->fen_genfcc.fcc_tbase = __pa((uint)&immap->im_dprambase[i]);


And everything works great...
Probably even increased performance a bit...

Proper patch:
--- ../fcc_enet.c       2004-06-09 22:33:14.000000000 -0500
+++ arch/ppc/8260_io/fcc_enet.c 2004-06-09 22:26:47.000000000 -0500
@@ -1564,17 +1564,17 @@ init_fcc_param(fcc_info_t *fip, struct n
         * These are relative offsets in the DP ram address space.
         * Initialize base addresses for the buffer descriptors.
         */
-#if 0
+#if 1
        /* I really want to do this, but for some reason it doesn't
         * work with the data cache enabled, so I allocate from the
         * main memory instead.
         */
        i = m8260_cpm_dpalloc(sizeof(cbd_t) * RX_RING_SIZE, 8);
-       ep->fen_genfcc.fcc_rbase = (uint)&immap->im_dprambase[i];
+       ep->fen_genfcc.fcc_rbase = __pa((uint)&immap->im_dprambase[i]);
        cep->rx_bd_base = (cbd_t *)&immap->im_dprambase[i];

        i = m8260_cpm_dpalloc(sizeof(cbd_t) * TX_RING_SIZE, 8);
-       ep->fen_genfcc.fcc_tbase = (uint)&immap->im_dprambase[i];
+       ep->fen_genfcc.fcc_tbase = __pa((uint)&immap->im_dprambase[i]);
        cep->tx_bd_base = (cbd_t *)&immap->im_dprambase[i];
 #else
        cep->rx_bd_base = (cbd_t *)m8260_cpm_hostalloc(sizeof(cbd_t) *
RX_RING_SIZE, 8);

---

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Poor 8260 FCC Ethernet performance

[Dan Malek]

On Jun 9, 2004, at 6:33 PM, Rune Torgersen wrote:

>
> BTW, found the problem with that piece of code about 2 seconds after
> sending it....
> The fcc_rbase &tbase parameters are not given the physical address, but
> the virtual one...

What?  No, no no.

You must not use pa() on immap addresses.  All you did was convert
a proper address to some bogus one, you are writing over some other
part of memory that is eventually going to cause a failure.  The immap
addresses are identity mapped virtual to physical.

There isn't any performance problem  with this driver if your hardware
is working properly.  I cant' believe this change would have done
anything except eventually causing the system to fail.

What kernel are you using?

Thanks.

	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Poor 8260 FCC Ethernet performance

[Rune Torgersen]

> -----Original Message-----
> From: Dan Malek [mailto:dan@embeddededge.com]
> Sent: Thursday, June 10, 2004 12:57
> What?  No, no no.

> There isn't any performance problem  with this driver if your
> hardware is working properly.  I cant' believe this change
> would have done anything except eventually causing the system to fail.

It did fail after a while. Had to move the BD's back into host memory to
get it to work correctly again.
Strage thing is that of the two CPU's on my board, it worked on one
(sort of) , and failed immediately on the other...

>
> What kernel are you using?

2.6.5


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/