PPC440GX ethernet oddities

PPC440GX ethernet oddities

[Jeff Mock]

I'm having a slightly strange behavior with PPC440GX ethernet, I'm
looking for a little advice where I can poke around to see what's going on.

I have a custom 440GX board, I use the two RGMII gigabit interfaces to
two Vistesse PHYs.  This works nicely.

The board has a large FPGA signal processor that is DMA'ing data into
main memory, the PPC sends data from main memory out the ethernet
interfaces.  This all works well.  For testing purposes I'm DMA'ing a
pseudo random sequence at 80MB/s, sending this over ethernet on a TCP
socket to a server machine and checking the sequence at the receiving
end.  So far so good.  Runs for days on several prototype machines.

As part of the DMA diagnostic program I keep track of the maximum
occupied capacity of the main memory ring buffer holding data from my
FPGA device driver.  This lets me keep track of how close I get to a
buffer overflow seeing as I'm running the gigabit ethernet port close to
the edge at 80MB/s.  The ring buffer will typically reach a maximum
level of 512kB.  This is how far the network connection gets behind the
realtime DMA from the FPGAs.

Here's the weird part.  On one of the four prototype boxes, if I plug
the second ethernet port into gigabit switch and get a link light (2nd
interface is not enabled under linux), the DMA behavior will change and
I can see the ring buffer get as large as 25MB (up from 512kB!)

Only one of my four boxes shows this strange behavior, and only when the
second ethernet port is connected to an ethernet switch. Everything
still works properly, my 80MB/s pseudo random sequence is still
generated by the FPGAs and checked by a server on the other end of the
network connection.  I let the ring buffer get as large as 64MB before
failing, but the large ring buffer says that the network connection
sometimes gets as much as 25MB behind the FPGA DMA, or 25/80 = 0.3125
seconds, which seems kind of crazy.

I look at "ifconfig" (busybox ifconfig) and I see no errors on the
ethernet interface.  I'm guessing there might be some design problem or
maybe just a problem with this one particular board that is causing
errors that occasionally slows down the TCP connection, perhaps
crosstalk between the two RGMII interfaces or maybe some interaction
between the magnetics on the two ports, but I can't figure out where to
look to measure errors on the physical ethernet interface.

Can someone give me a hint about where to look for this problem? This is
a 2.6.15 kernel.

Thanks for reading, I went on a bit long...
jeff

Re: PPC440GX ethernet oddities

[Eugene Surovegin]

On Sun, Nov 26, 2006 at 08:05:49PM -0800, Jeff Mock wrote:
> I'm having a slightly strange behavior with PPC440GX ethernet, I'm
> looking for a little advice where I can poke around to see what's going on.
> 
> I have a custom 440GX board, I use the two RGMII gigabit interfaces to
> two Vistesse PHYs.  This works nicely.

What is the CPU clock speed?

> The board has a large FPGA signal processor that is DMA'ing data into
> main memory, the PPC sends data from main memory out the ethernet
> interfaces.  This all works well.  For testing purposes I'm DMA'ing a
> pseudo random sequence at 80MB/s, sending this over ethernet on a TCP
> socket to a server machine and checking the sequence at the receiving
> end. 

Could you elaborate a little here. Is it a user-space program or some 
kernel mode code which does the copying?

[snip]

> Only one of my four boxes shows this strange behavior, and only when the
> second ethernet port is connected to an ethernet switch. Everything
> still works properly, my 80MB/s pseudo random sequence is still
> generated by the FPGAs and checked by a server on the other end of the
> network connection.  I let the ring buffer get as large as 64MB before
> failing, but the large ring buffer says that the network connection
> sometimes gets as much as 25MB behind the FPGA DMA, or 25/80 = 0.3125
> seconds, which seems kind of crazy.

Well, 300ms doesn't look particularly crazy to me given a data stream 
and the fact you are using non-realtime OSes on both ends.

> 
> I look at "ifconfig" (busybox ifconfig) and I see no errors on the
> ethernet interface.

Try ethtool -S, EMAC driver supports it.

> I'm guessing there might be some design problem or
> maybe just a problem with this one particular board that is causing
> errors that occasionally slows down the TCP connection, perhaps
> crosstalk between the two RGMII interfaces or maybe some interaction
> between the magnetics on the two ports, but I can't figure out where to
> look to measure errors on the physical ethernet interface.
> 
> Can someone give me a hint about where to look for this problem? This is
> a 2.6.15 kernel.

There are some stuff you can do:

1) Try another GigE switch, better if from the different vendor

2) Try another peer system you are streaming your data to (better with 
different GigE NIC)

3) Check if hw flow-control is enabled and try disabling it on both 
ends

4) Play with socket buffer sizes on both ends

5) And the final, but the most useful one - get a realtime capture of 
the TCP stream when your ring buffer starts growing. The best way 
is to get GigE switch which supports port mirroring or use special 
hw network analyzer.

-- 
Eugene

Re: PPC440GX ethernet oddities

[Jeff Mock]

Eugene Surovegin wrote:
> On Sun, Nov 26, 2006 at 08:05:49PM -0800, Jeff Mock wrote:
>> I'm having a slightly strange behavior with PPC440GX ethernet, I'm
>> looking for a little advice where I can poke around to see what's going on.
>>
>> I have a custom 440GX board, I use the two RGMII gigabit interfaces to
>> two Vistesse PHYs.  This works nicely.
> 
> What is the CPU clock speed?
> 

They are 800MHz rev.F 440GX's.

>> The board has a large FPGA signal processor that is DMA'ing data into
>> main memory, the PPC sends data from main memory out the ethernet
>> interfaces.  This all works well.  For testing purposes I'm DMA'ing a
>> pseudo random sequence at 80MB/s, sending this over ethernet on a TCP
>> socket to a server machine and checking the sequence at the receiving
>> end. 
> 
> Could you elaborate a little here. Is it a user-space program or some 
> kernel mode code which does the copying?
> 

The device driver for the FPGA allocates 64MB of memory at boot time and
uses this as a ring buffer.  The user process mmap()s the entire buffer.
When the user process does a read it blocks until data is available and
the device driver returns pointers into the ring buffer for the
currently available data.  The user process passes pointers to the ring
buffer to the write() used to send data out the network connection.

This isn't terribly memory efficient, but the embedded 440GX main
purpose in life is to move data from the FPGAs across the network. At
80MB/s the 440GX CPU seems to be about 40% utilized.


> 
> Well, 300ms doesn't look particularly crazy to me given a data stream 
> and the fact you are using non-realtime OSes on both ends.
> 

I'm starting to think that you might be right about that.  The odd thing
that caused me to seek community help is that the ring buffer only fills
to about 512kB under normal circumstances, but fills to 25MB if I
connect the second Gb ethernet port of the 440GX.


> Try ethtool -S, EMAC driver supports it.
> 

This is a great idea, I didn't realize that ethtool supports the IBM
EMAC driver.  I suspected that I would find some electrical problem that
was causing additional packet errors when the second ethernet port is
connected.  A ran about 2TB last night, ethtool -S on the PPC says:

# ./ethtool -S eth0
NIC statistics:
     rx_packets: 683430611
     rx_bytes: 45405076316
     tx_packets: 1331225622
     tx_bytes: 2005276384145
     rx_packets_csum: 683428670
     tx_packets_csum: 1331224150
     tx_undo: 0
     rx_dropped_stack: 80
     rx_dropped_oom: 0
     rx_dropped_error: 0
     [lots more 0's...]

The number of checksum errors seems okay and doesn't really change when
I plug in the second network connection, but the ring buffer usage still
increases dramatically.

Thanks for the other suggestions.  I have a friend at a 'real' company
where I might be able to borrow a network analyzer.

jeff

Re: PPC440GX ethernet oddities

[Eugene Surovegin]

On Mon, Nov 27, 2006 at 09:34:59AM -0800, Jeff Mock wrote:
> 
> Eugene Surovegin wrote:
> > On Sun, Nov 26, 2006 at 08:05:49PM -0800, Jeff Mock wrote:
> >> I'm having a slightly strange behavior with PPC440GX ethernet, I'm
> >> looking for a little advice where I can poke around to see what's going on.
> >>
> >> I have a custom 440GX board, I use the two RGMII gigabit interfaces to
> >> two Vistesse PHYs.  This works nicely.
> > 
> > What is the CPU clock speed?
> > 
> 
> They are 800MHz rev.F 440GX's.

OK, this explains why you can get 80MB/s over Linux sockets :)

> 
> >> The board has a large FPGA signal processor that is DMA'ing data into
> >> main memory, the PPC sends data from main memory out the ethernet
> >> interfaces.  This all works well.  For testing purposes I'm DMA'ing a
> >> pseudo random sequence at 80MB/s, sending this over ethernet on a TCP
> >> socket to a server machine and checking the sequence at the receiving
> >> end. 
> > 
> > Could you elaborate a little here. Is it a user-space program or some 
> > kernel mode code which does the copying?
> > 
> 
> The device driver for the FPGA allocates 64MB of memory at boot time and
> uses this as a ring buffer.  The user process mmap()s the entire buffer.
> When the user process does a read it blocks until data is available and
> the device driver returns pointers into the ring buffer for the
> currently available data.  The user process passes pointers to the ring
> buffer to the write() used to send data out the network connection.
> 
> This isn't terribly memory efficient, but the embedded 440GX main
> purpose in life is to move data from the FPGAs across the network. At
> 80MB/s the 440GX CPU seems to be about 40% utilized.

Yeah, you probably can save one memory copy by using kernel mode code, 
but 40% seems OK if this is the only thing 440GX has to do.

> > 
> > Well, 300ms doesn't look particularly crazy to me given a data stream 
> > and the fact you are using non-realtime OSes on both ends.
> > 
> 
> I'm starting to think that you might be right about that.  The odd thing
> that caused me to seek community help is that the ring buffer only fills
> to about 512kB under normal circumstances, but fills to 25MB if I
> connect the second Gb ethernet port of the 440GX.

Although it's interesting to figure out what caused this change in 
behaviour, you have to keep in mind that TCP isn't a real-time 
protocol and it can create some jitter. You have to be prepared to 
buffer upto several seconds of your data when using TCP on local LAN 
and more if using intranet/Internet.

> 
> 
> > Try ethtool -S, EMAC driver supports it.
> > 
> 
> This is a great idea, I didn't realize that ethtool supports the IBM
> EMAC driver.  I suspected that I would find some electrical problem that
> was causing additional packet errors when the second ethernet port is
> connected.  A ran about 2TB last night, ethtool -S on the PPC says:
> 
> # ./ethtool -S eth0
> NIC statistics:
>      rx_packets: 683430611
>      rx_bytes: 45405076316
>      tx_packets: 1331225622
>      tx_bytes: 2005276384145
>      rx_packets_csum: 683428670
>      tx_packets_csum: 1331224150
>      tx_undo: 0
>      rx_dropped_stack: 80
>      rx_dropped_oom: 0
>      rx_dropped_error: 0
>      [lots more 0's...]
> 
> The number of checksum errors seems okay and doesn't really change when
> I plug in the second network connection, but the ring buffer usage still
> increases dramatically.

Stats look OK.  BTW, tx/rx_packets_csum is not a number of checksum 
errors, that's a number of packets where driver used hw checksum 
acceleration :).

-- 
Eugene

Re: PPC440GX ethernet oddities

[Jeff Mock]

Eugene Surovegin wrote:
> On Mon, Nov 27, 2006 at 09:34:59AM -0800, Jeff Mock wrote:
>> # ./ethtool -S eth0
>> NIC statistics:
>>      rx_packets: 683430611
>>      rx_bytes: 45405076316
>>      tx_packets: 1331225622
>>      tx_bytes: 2005276384145
>>      rx_packets_csum: 683428670
>>      tx_packets_csum: 1331224150
>>      tx_undo: 0
>>      rx_dropped_stack: 80
>>      rx_dropped_oom: 0
>>      rx_dropped_error: 0
>>      [lots more 0's...]
>>
>> The number of checksum errors seems okay and doesn't really change when
>> I plug in the second network connection, but the ring buffer usage still
>> increases dramatically.
> 
> Stats look OK.  BTW, tx/rx_packets_csum is not a number of checksum 
> errors, that's a number of packets where driver used hw checksum 
> acceleration :).
> 

Ah, silly me.  Can you tell me where I can find the number of received
packets with bad ethernet checksums?  I'm still thinking that I might
have a little hardware problem that is increasing the bit error rate
when I use both ports, ultimately increasing my buffer utilization.

thanks,
jeff

Re: PPC440GX ethernet oddities

[Eugene Surovegin]

On Mon, Nov 27, 2006 at 02:19:33PM -0800, Jeff Mock wrote:
> 
> 
> Eugene Surovegin wrote:
> > On Mon, Nov 27, 2006 at 09:34:59AM -0800, Jeff Mock wrote:
> >> # ./ethtool -S eth0
> >> NIC statistics:
> >>      rx_packets: 683430611
> >>      rx_bytes: 45405076316
> >>      tx_packets: 1331225622
> >>      tx_bytes: 2005276384145
> >>      rx_packets_csum: 683428670
> >>      tx_packets_csum: 1331224150
> >>      tx_undo: 0
> >>      rx_dropped_stack: 80
> >>      rx_dropped_oom: 0
> >>      rx_dropped_error: 0
> >>      [lots more 0's...]
> >>
> >> The number of checksum errors seems okay and doesn't really change when
> >> I plug in the second network connection, but the ring buffer usage still
> >> increases dramatically.
> > 
> > Stats look OK.  BTW, tx/rx_packets_csum is not a number of checksum 
> > errors, that's a number of packets where driver used hw checksum 
> > acceleration :).
> > 
> 
> Ah, silly me.  Can you tell me where I can find the number of received
> packets with bad ethernet checksums?  I'm still thinking that I might
> have a little hardware problem that is increasing the bit error rate
> when I use both ports, ultimately increasing my buffer utilization.

Packets with bad ethernet checksums are dropped by the MAC before 
being handed off to MAL. You can disable this behavior by setting 
RFP bit in EMACx_RMR register. In this case I think IP stack will drop 
these packets because IP and/or TCP checksum will fail. Use netstat -s 
to check network stack statistics.

Here is untested patch which enables RFP bit. It's against recent-ish 
kernel.org git tree:

diff --git a/drivers/net/ibm_emac/ibm_emac_core.c b/drivers/net/ibm_emac/ibm_emac_core.c
index ffeafb2..56ad988 100644
--- a/drivers/net/ibm_emac/ibm_emac_core.c
+++ b/drivers/net/ibm_emac/ibm_emac_core.c
@@ -372,7 +372,7 @@ static void emac_hash_mc(struct ocp_enet
 static inline u32 emac_iff2rmr(struct net_device *ndev)
 {
        u32 r = EMAC_RMR_SP | EMAC_RMR_SFCS | EMAC_RMR_IAE | EMAC_RMR_BAE |
-           EMAC_RMR_BASE;
+           EMAC_RMR_BASE | EMAC_RMR_RFP;

        if (ndev->flags & IFF_PROMISC)
                r |= EMAC_RMR_PME;



-- 
Eugene

Re: PPC440GX ethernet oddities

[Wolfgang Denk]

In message <456A641D.7070904@mock.com> you wrote:
>
> Here's the weird part.  On one of the four prototype boxes, if I plug
> the second ethernet port into gigabit switch and get a link light (2nd
> interface is not enabled under linux), the DMA behavior will change and
> I can see the ring buffer get as large as 25MB (up from 512kB!)
> 
> Only one of my four boxes shows this strange behavior, and only when the
> second ethernet port is connected to an ethernet switch. Everything

Are you using exactly the same switch,  and  the  same  port  on  the
switch, and the same network cable with all of your boxen?

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
The universe does not have laws - it has habits, and  habits  can  be
broken.

Re: PPC440GX ethernet oddities

[Jeff Mock]

Wolfgang Denk wrote:
> In message <456A641D.7070904@mock.com> you wrote:
>> Here's the weird part.  On one of the four prototype boxes, if I plug
>> the second ethernet port into gigabit switch and get a link light (2nd
>> interface is not enabled under linux), the DMA behavior will change and
>> I can see the ring buffer get as large as 25MB (up from 512kB!)
>>
>> Only one of my four boxes shows this strange behavior, and only when the
>> second ethernet port is connected to an ethernet switch. Everything
> 
> Are you using exactly the same switch,  and  the  same  port  on  the
> switch, and the same network cable with all of your boxen?
> 

Yeah, I'm using the same switch, cables, and server machines for these
tests.  I've done quite a lot of fiddling with servers, cables, and
switches looking for some differential behavior, but the only thing I
find is that with second ethernet port unconnected, my ring buffer will
consistently only fill to 512kB, but with the second ethernet port
connect to get a link light (disabled under linux), the ring buffer will
get as full as 25MB over time.

jeff