Gianfar is slower than fcc_enet on MPC8541 ???

Gianfar is slower than fcc_enet on MPC8541 ???

[Laurent Lagrange]

Hello,

I work on a cutom MPC8541 board with Linux 2.6.9.
The kernel activates the L1 cache (instructions and data)
and the L2 cache (entirely used as cache and not as sram).

I configure 
1 FCC (FCC1),
2 TSECs with or without NAPI (no effect) but without stashing in L2 sram.
All PHYs are automatically configured in 100MB full duplex.

	eth0: Gianfar Ethernet Controller Version 1.1, 00:10:cd:48:48:e0
	eth0: Running with NAPI disabled
	eth0: 64/64 RX/TX BD ring size
	eth1: Gianfar Ethernet Controller Version 1.1, 00:10:cd:48:48:e1
	eth1: Running with NAPI disabled
	eth1: 64/64 RX/TX BD ring size
	eth2: FCC ENET Version custom, 00:10:cd:48:48:e2

Then I launch 3 simple TCP servers, one on each ports.

>From remote machines I runs 3 TCP clients.
The client sends messages of 1000 bytes, 
The server receives and echoes the message
The client receives the echoed message, check the content
and sends a new message again.

The result is that the 2 TSECs are 2 times slower than the FCC.

If I run a "top" application on the board, I use less than 10% of the CPU
Each port consumes about 1/3 of the CPU.

Any idea on how to configure the gianfar driver ?

Thanks
Laurent
 

Re: Gianfar is slower than fcc_enet on MPC8541 ???

[Pantelis Antoniou]

[-- Attachment #1: Type: text/plain, Size: 1755 bytes --]

Hi Laurent,

I found that pretty hard to believe.

What are you measuring exactly?

Speed of replies? If so it's explainable since the TSECs use
NAPI.

Regards

Pantelis

On 2/14/06, Laurent Lagrange <lagrange@fr.oleane.com> wrote:
>
>
> Hello,
>
> I work on a cutom MPC8541 board with Linux 2.6.9.
> The kernel activates the L1 cache (instructions and data)
> and the L2 cache (entirely used as cache and not as sram).
>
> I configure
> 1 FCC (FCC1),
> 2 TSECs with or without NAPI (no effect) but without stashing in L2 sram.
> All PHYs are automatically configured in 100MB full duplex.
>
>         eth0: Gianfar Ethernet Controller Version 1.1, 00:10:cd:48:48:e0
>         eth0: Running with NAPI disabled
>         eth0: 64/64 RX/TX BD ring size
>         eth1: Gianfar Ethernet Controller Version 1.1, 00:10:cd:48:48:e1
>         eth1: Running with NAPI disabled
>         eth1: 64/64 RX/TX BD ring size
>         eth2: FCC ENET Version custom, 00:10:cd:48:48:e2
>
> Then I launch 3 simple TCP servers, one on each ports.
>
> From remote machines I runs 3 TCP clients.
> The client sends messages of 1000 bytes,
> The server receives and echoes the message
> The client receives the echoed message, check the content
> and sends a new message again.
>
> The result is that the 2 TSECs are 2 times slower than the FCC.
>
> If I run a "top" application on the board, I use less than 10% of the CPU
> Each port consumes about 1/3 of the CPU.
>
> Any idea on how to configure the gianfar driver ?
>
> Thanks
> Laurent
>
>
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded
>

[-- Attachment #2: Type: text/html, Size: 2485 bytes --]

RE: Gianfar is slower than fcc_enet on MPC8541 ???

[Laurent Lagrange]

[-- Attachment #1: Type: text/plain, Size: 2386 bytes --]

Hi Pantelis,

Thanks for the express reply.

I know that what I say seems incredible. But I don't understand what NAPI
does.
My measure is very simple. I display a message on the client when 1000
exchanges are done.
I already check the ifconfig stats on the board after some seconds.
The measures seems the same with or without NAPI.

More details ?
Thanks again
Laurent


  -----Message d'origine-----
  De : Pantelis Antoniou [mailto:pantelis.antoniou@gmail.com]
  Envoyé : mar. 14 février 2006 16:29
  À : Laurent Lagrange
  Cc : linuxppc-embedded@ozlabs.org
  Objet : Re: Gianfar is slower than fcc_enet on MPC8541 ???


  Hi Laurent,

  I found that pretty hard to believe.

  What are you measuring exactly?

  Speed of replies? If so it's explainable since the TSECs use
  NAPI.

  Regards

  Pantelis


  On 2/14/06, Laurent Lagrange <lagrange@fr.oleane.com> wrote:

    Hello,

    I work on a cutom MPC8541 board with Linux 2.6.9.
    The kernel activates the L1 cache (instructions and data)
    and the L2 cache (entirely used as cache and not as sram).

    I configure
    1 FCC (FCC1),
    2 TSECs with or without NAPI (no effect) but without stashing in L2
sram.
    All PHYs are automatically configured in 100MB full duplex.

            eth0: Gianfar Ethernet Controller Version 1.1, 00:10:cd:48:48:e0
            eth0: Running with NAPI disabled
            eth0: 64/64 RX/TX BD ring size
            eth1: Gianfar Ethernet Controller Version 1.1, 00:10:cd:48:48:e1
            eth1: Running with NAPI disabled
            eth1: 64/64 RX/TX BD ring size
            eth2: FCC ENET Version custom, 00:10:cd:48:48:e2

    Then I launch 3 simple TCP servers, one on each ports.

    From remote machines I runs 3 TCP clients.
    The client sends messages of 1000 bytes,
    The server receives and echoes the message
    The client receives the echoed message, check the content
    and sends a new message again.

    The result is that the 2 TSECs are 2 times slower than the FCC.

    If I run a "top" application on the board, I use less than 10% of the
CPU
    Each port consumes about 1/3 of the CPU.

    Any idea on how to configure the gianfar driver ?

    Thanks
    Laurent


    _______________________________________________
    Linuxppc-embedded mailing list
    Linuxppc-embedded@ozlabs.org
    https://ozlabs.org/mailman/listinfo/linuxppc-embedded



[-- Attachment #2: Type: text/html, Size: 5369 bytes --]

Re: Gianfar is slower than fcc_enet on MPC8541 ???

[Pantelis Antoniou]

[-- Attachment #1: Type: text/plain, Size: 2721 bytes --]

Can you provide some more information?

Actual timings, and a tcpdump fragment?

On 2/14/06, Laurent Lagrange <lagrange@fr.oleane.com> wrote:
>
> Hi Pantelis,
>
> Thanks for the express reply.
>
> I know that what I say seems incredible. But I don't understand what NAPI
> does.
> My measure is very simple. I display a message on the client when 1000
> exchanges are done.
> I already check the ifconfig stats on the board after some seconds.
> The measures seems the same with or without NAPI.
>
> More details ?
> Thanks again
> Laurent
>
>
>
> -----Message d'origine-----
> *De :* Pantelis Antoniou [mailto:pantelis.antoniou@gmail.com]
> *Envoyé :* mar. 14 février 2006 16:29
> *À :* Laurent Lagrange
> *Cc :* linuxppc-embedded@ozlabs.org
> *Objet :* Re: Gianfar is slower than fcc_enet on MPC8541 ???
>
> Hi Laurent,
>
> I found that pretty hard to believe.
>
> What are you measuring exactly?
>
> Speed of replies? If so it's explainable since the TSECs use
> NAPI.
>
> Regards
>
> Pantelis
>
> On 2/14/06, Laurent Lagrange <lagrange@fr.oleane.com> wrote:
> >
> >
> > Hello,
> >
> > I work on a cutom MPC8541 board with Linux 2.6.9.
> > The kernel activates the L1 cache (instructions and data)
> > and the L2 cache (entirely used as cache and not as sram).
> >
> > I configure
> > 1 FCC (FCC1),
> > 2 TSECs with or without NAPI (no effect) but without stashing in L2
> > sram.
> > All PHYs are automatically configured in 100MB full duplex.
> >
> >         eth0: Gianfar Ethernet Controller Version 1.1, 00:10:cd:48:48:e0
> >         eth0: Running with NAPI disabled
> >         eth0: 64/64 RX/TX BD ring size
> >         eth1: Gianfar Ethernet Controller Version 1.1, 00:10:cd:48:48:e1
> >         eth1: Running with NAPI disabled
> >         eth1: 64/64 RX/TX BD ring size
> >         eth2: FCC ENET Version custom, 00:10:cd:48:48:e2
> >
> > Then I launch 3 simple TCP servers, one on each ports.
> >
> > From remote machines I runs 3 TCP clients.
> > The client sends messages of 1000 bytes,
> > The server receives and echoes the message
> > The client receives the echoed message, check the content
> > and sends a new message again.
> >
> > The result is that the 2 TSECs are 2 times slower than the FCC.
> >
> > If I run a "top" application on the board, I use less than 10% of the
> > CPU
> > Each port consumes about 1/3 of the CPU.
> >
> > Any idea on how to configure the gianfar driver ?
> >
> > Thanks
> > Laurent
> >
> >
> > _______________________________________________
> > Linuxppc-embedded mailing list
> > Linuxppc-embedded@ozlabs.org
> > https://ozlabs.org/mailman/listinfo/linuxppc-embedded
> >
>
>

[-- Attachment #2: Type: text/html, Size: 5752 bytes --]

Re: Gianfar is slower than fcc_enet on MPC8541 ???

[Vitaly Bordug]

Laurent, 
btw, have you gived netperf test a try? 
It is a commonly-used tool to measure network performance, and has dozens of settings to test...

On Tue, 14 Feb 2006 11:17:46 -0500
Pantelis Antoniou <pantelis.antoniou@gmail.com> wrote:

> Can you provide some more information?
> 
> Actual timings, and a tcpdump fragment?
> 
> On 2/14/06, Laurent Lagrange <lagrange@fr.oleane.com> wrote:
> >
> > Hi Pantelis,
> >
> > Thanks for the express reply.
> >
> > I know that what I say seems incredible. But I don't understand what NAPI
> > does.
> > My measure is very simple. I display a message on the client when 1000
> > exchanges are done.
> > I already check the ifconfig stats on the board after some seconds.
> > The measures seems the same with or without NAPI.
> >
> > More details ?
> > Thanks again
> > Laurent
> >
> >
> >
> > -----Message d'origine-----
> > *De :* Pantelis Antoniou [mailto:pantelis.antoniou@gmail.com]
> > *Envoyé :* mar. 14 février 2006 16:29
> > *À :* Laurent Lagrange
> > *Cc :* linuxppc-embedded@ozlabs.org
> > *Objet :* Re: Gianfar is slower than fcc_enet on MPC8541 ???
> >
> > Hi Laurent,
> >
> > I found that pretty hard to believe.
> >
> > What are you measuring exactly?
> >
> > Speed of replies? If so it's explainable since the TSECs use
> > NAPI.
> >
> > Regards
> >
> > Pantelis
> >
> > On 2/14/06, Laurent Lagrange <lagrange@fr.oleane.com> wrote:
> > >
> > >
> > > Hello,
> > >
> > > I work on a cutom MPC8541 board with Linux 2.6.9.
> > > The kernel activates the L1 cache (instructions and data)
> > > and the L2 cache (entirely used as cache and not as sram).
> > >
> > > I configure
> > > 1 FCC (FCC1),
> > > 2 TSECs with or without NAPI (no effect) but without stashing in L2
> > > sram.
> > > All PHYs are automatically configured in 100MB full duplex.
> > >
> > >         eth0: Gianfar Ethernet Controller Version 1.1, 00:10:cd:48:48:e0
> > >         eth0: Running with NAPI disabled
> > >         eth0: 64/64 RX/TX BD ring size
> > >         eth1: Gianfar Ethernet Controller Version 1.1, 00:10:cd:48:48:e1
> > >         eth1: Running with NAPI disabled
> > >         eth1: 64/64 RX/TX BD ring size
> > >         eth2: FCC ENET Version custom, 00:10:cd:48:48:e2
> > >
> > > Then I launch 3 simple TCP servers, one on each ports.
> > >
> > > From remote machines I runs 3 TCP clients.
> > > The client sends messages of 1000 bytes,
> > > The server receives and echoes the message
> > > The client receives the echoed message, check the content
> > > and sends a new message again.
> > >
> > > The result is that the 2 TSECs are 2 times slower than the FCC.
> > >
> > > If I run a "top" application on the board, I use less than 10% of the
> > > CPU
> > > Each port consumes about 1/3 of the CPU.
> > >
> > > Any idea on how to configure the gianfar driver ?
> > >
> > > Thanks
> > > Laurent
> > >
> > >
> > > _______________________________________________
> > > Linuxppc-embedded mailing list
> > > Linuxppc-embedded@ozlabs.org
> > > https://ozlabs.org/mailman/listinfo/linuxppc-embedded
> > >
> >
> >


-- 
Sincerely, 
Vitaly

Re: Gianfar is slower than fcc_enet on MPC8541 ???

[Andy Fleming]

My guess is that you are measuring latency.  By default, interrupt  
coalescing is on, and used to be set to very poor default values.  In  
drivers/net/gianfar.h, change the defaults to look like this:

#define DEFAULT_TX_COALESCE 1
#define DEFAULT_TXCOUNT 16
#define DEFAULT_TXTIME  4

#define DEFAULT_RX_COALESCE 1
#define DEFAULT_RXCOUNT 16
#define DEFAULT_RXTIME  4

The problem was that the timeout was quite long, so a small number of  
packets would have to wait a whole millisecond (or more!) to get  
processed.  While it wouldn't affect bandwidth tests, which send many  
packets, it would affect a simple test like ping.

If you don't feel like recompiling the kernel, you can use ethtool to  
change the timeout values.


On Feb 14, 2006, at 09:26, Laurent Lagrange wrote:

>
> Hello,
>
> I work on a cutom MPC8541 board with Linux 2.6.9.
> The kernel activates the L1 cache (instructions and data)
> and the L2 cache (entirely used as cache and not as sram).
>
> I configure
> 1 FCC (FCC1),
> 2 TSECs with or without NAPI (no effect) but without stashing in L2  
> sram.
> All PHYs are automatically configured in 100MB full duplex.
>
> 	eth0: Gianfar Ethernet Controller Version 1.1, 00:10:cd:48:48:e0
> 	eth0: Running with NAPI disabled
> 	eth0: 64/64 RX/TX BD ring size
> 	eth1: Gianfar Ethernet Controller Version 1.1, 00:10:cd:48:48:e1
> 	eth1: Running with NAPI disabled
> 	eth1: 64/64 RX/TX BD ring size
> 	eth2: FCC ENET Version custom, 00:10:cd:48:48:e2
>
> Then I launch 3 simple TCP servers, one on each ports.
>
>> From remote machines I runs 3 TCP clients.
> The client sends messages of 1000 bytes,
> The server receives and echoes the message
> The client receives the echoed message, check the content
> and sends a new message again.
>
> The result is that the 2 TSECs are 2 times slower than the FCC.
>
> If I run a "top" application on the board, I use less than 10% of  
> the CPU
> Each port consumes about 1/3 of the CPU.
>
> Any idea on how to configure the gianfar driver ?
>
> Thanks
> Laurent
>
>
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded

RE: Gianfar is slower than fcc_enet on MPC8541 ???

[Laurent Lagrange]

Hi everybody,

Sorry to reply so late. I was out of office.

I tried the below Andy's idea. It works fine.
It is my TCP clients which now slow the traffic.

But I don't know why the default timeouts are so high.
If the traffic is high, the timeout does not fire.
If the traffic is low, the timeout seems too long (???).

Thanks all for your ideas and works.
Best regards
Laurent


> De la part de Andy Fleming
> Envoye : mar. 14 fevrier 2006 18:37
> A : Laurent Lagrange
> Cc : linuxppc-embedded@ozlabs.org
> Objet : Re: Gianfar is slower than fcc_enet on MPC8541 ???
> 
> My guess is that you are measuring latency.  By default, interrupt  
> coalescing is on, and used to be set to very poor default 
> values.  In drivers/net/gianfar.h, change the defaults to look like this:
> 
> #define DEFAULT_TX_COALESCE 1
> #define DEFAULT_TXCOUNT 16
> #define DEFAULT_TXTIME  4
> 
> #define DEFAULT_RX_COALESCE 1
> #define DEFAULT_RXCOUNT 16
> #define DEFAULT_RXTIME  4
> 
> The problem was that the timeout was quite long, so a small number of  
> packets would have to wait a whole millisecond (or more!) to get  
> processed.  While it wouldn't affect bandwidth tests, which send many  
> packets, it would affect a simple test like ping.
> 
> If you don't feel like recompiling the kernel, you can use ethtool to  
> change the timeout values.
 

Re: Gianfar is slower than fcc_enet on MPC8541 ???

[Andy Fleming]

On Feb 23, 2006, at 11:28, Laurent Lagrange wrote:

> Hi everybody,
>
> I tried the below Andy's idea. It works fine.
> It is my TCP clients which now slow the traffic.
>
> But I don't know why the default timeouts are so high.
> If the traffic is high, the timeout does not fire.
> If the traffic is low, the timeout seems too long (???).

The answer there is simple: stupidity!  :)  I just didn't carefully  
test the values for performance when I chose them.  I probably also  
did the math wrong, because I was more concerned about seeing if it  
worked at all.  It's also possible it got set that way to see a  
measurable difference to prove it was working, and then got left as  
the default.  Rest assured, there was not a deliberate reason.  We  
submitted a patch once this performance issue was discovered.

RE: Gianfar is slower than fcc_enet on MPC8541 ???

[Laurent Lagrange]

Hello Andy,

> The answer there is simple: stupidity!
The word is too hard, just say forgetting  :-)

Thanks again
Laurent

> -----Message d'origine-----
> De : Andy Fleming [mailto:afleming@freescale.com]
> Envoye : jeu. 23 fevrier 2006 21:08
> A : Laurent Lagrange
> Cc : linuxppc-embedded@ozlabs.org; vbordug@ru.mvista.com;
> pantelis.antoniou@gmail.com
> Objet : Re: Gianfar is slower than fcc_enet on MPC8541 ???
> 
> 
> 
> On Feb 23, 2006, at 11:28, Laurent Lagrange wrote:
> 
> > Hi everybody,
> >
> > I tried the below Andy's idea. It works fine.
> > It is my TCP clients which now slow the traffic.
> >
> > But I don't know why the default timeouts are so high.
> > If the traffic is high, the timeout does not fire.
> > If the traffic is low, the timeout seems too long (???).
> 
> The answer there is simple: stupidity!  :)  I just didn't carefully  
> test the values for performance when I chose them.  I probably also  
> did the math wrong, because I was more concerned about seeing if it  
> worked at all.  It's also possible it got set that way to see a  
> measurable difference to prove it was working, and then got left as  
> the default.  Rest assured, there was not a deliberate reason.  We  
> submitted a patch once this performance issue was discovered.
> 
> 
> 
> 
>