FEC ERROR 41200000 on CLLF-860T

FEC ERROR 41200000 on CLLF-860T

[Graham Stoney]

Hi gang,

I'm using the linux-2.2.13 kernel from the linuxppc.cs.nmt.edu on a couple of
Embedded Planet CLLF-860T boards, and I'm wondering if anyone can explain the
following messages from the FEC for me please:
    FEC ERROR 41200000
    FEC ENET: rcv is not +last

Is this something I should be worried about?

Thanks!
Graham

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: FEC ERROR 41200000 on CLLF-860T

[Dan Malek]

Graham Stoney wrote:


>     FEC ERROR 41200000
>     FEC ENET: rcv is not +last
> 
> Is this something I should be worried about?


Oh yes.....I don't have the FEC driver updated for that board
yet.  Although my instructions tell you how to download through
that port and get the CLLF utility program to initialize it
because I have been too lazy, something still isn't quite
right.

If it appears to be working, then just keep smiling and go
on about your business.  Don't try any performance tests.......

That particular error looks legitimate.  Probably some kind
of a hard network error.  Since the buffer wasn't marked as +last,
it looks like someone is jumping on the bus with late collisions
or something.  How often does it occur?


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: FEC ERROR 41200000 on CLLF-860T

[Graham Stoney]

Dan Malek writes:
> Oh yes.....I don't have the FEC driver updated for that board
> yet.  Although my instructions tell you how to download through
> that port and get the CLLF utility program to initialize it
> because I have been too lazy, something still isn't quite
> right.

I've got the kernel to boot thanks to your instructions, and it does keep
running after the error appears. I notice that the fec driver in the 2.3.18
kernel is a little different, but I haven't managed to get that kernel to run
at all yet, so I'm hanging with 2.2.13 at least for now.

> If it appears to be working, then just keep smiling and go
> on about your business.  Don't try any performance tests.......

Hmmm... performance tests are something I'm thinking about trying pretty soon!

> That particular error looks legitimate.  Probably some kind
> of a hard network error.  Since the buffer wasn't marked as +last,
> it looks like someone is jumping on the bus with late collisions
> or something.  How often does it occur?

Every couple of minutes or so; it seems to happen most during NFS traffic. I
haven't really hunted it hard yet though.

Thanks,
Graham

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: FEC ERROR 41200000 on CLLF-860T

[Dan Malek]

Graham Stoney wrote:

> .... I notice that the fec driver in the 2.3.18
> kernel is a little different, but I haven't managed to get that kernel to run
> at all yet, so I'm hanging with 2.2.13 at least for now.

Yes, I have learned the importance of being able to start and
shutdown network drivers (the SCC doesn't do this).  The FEC also
has to be configured for half/full duplex, and this we must get
the link change interrupts to detect this.  If you never disconnect
the Ethernet, this wouldn't be necessary, but that isn't real
life.  The drivers are changing to accomodate all of these things.


> Hmmm... performance tests are something I'm thinking about trying pretty soon!

Keep in mind that both the SCC and FEC Ethernet can send and
receive back-to-back packets.  They also have some tuning parameters
for things like backoff timers to be more friendly on the network,
and for SDMA system performance.  The SCC Ethernet on a 50 MHz 860
can easily maintain the theoretical network limits between 8
and 9 Mbit/sec with the TCP/IP stack.

I could never understand why people use this as a "benchmark"
as it is more important what you do with the data when you
receive it or before you send it.  This is a function of your
task and the speed of the processor.  There are no bounded
latency guarantees or predictable behavior with Ethernet, so
why try to use it like that?

The SCC and FEC will send packets as fast as you can create them,
and will discard packets if you can't keep up.  They provide
efficient burst mode DMA and off load the PPC core.  What more
do you need to know?


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: FEC ERROR 41200000 on CLLF-860T

[Graham Stoney]

Dan Malek writes:
> The drivers are changing to accomodate all of these things.

Sounds great; I'd be interested in testing an update whenever it's doable.

> The SCC Ethernet on a 50 MHz 860 can easily maintain the theoretical network
> limits between 8 and 9 Mbit/sec with the TCP/IP stack.

We're using the FEC exclusively, and I'm hoping for a similar story at around
8 Mbytes/sec with the TCP/IP stack.

> I could never understand why people use this as a "benchmark"
> as it is more important what you do with the data when you
> receive it or before you send it.  This is a function of your
> task and the speed of the processor.  There are no bounded
> latency guarantees or predictable behavior with Ethernet, so
> why try to use it like that?

You're absolutely right, of course: what matters here is system throughput in
which the Ethernet is just one link. Our system is heavily pipelined all the
way from the remote application down to the lowest hardware level. So long as
none of the links in the pipeline stalls long enough to become the bottleneck,
all the data should fly through nicely. Buffering in the various pipeline
stages (including the TCP/IP buffers) affects how far a stall will propagate
back up the pipeline. As you say, the important part here is what we do with
the data when we receive it and before we send it; if we get that right, the
ethernet link throughput will fall out in the wash.

Regards,
Graham

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: FEC ERROR 41200000 on CLLF-860T

[Marcus Sundberg]

greyham@research.canon.com.au (Graham Stoney) writes:

> Dan Malek writes:
> > The drivers are changing to accomodate all of these things.
> 
> Sounds great; I'd be interested in testing an update whenever it's doable.
> 
> > The SCC Ethernet on a 50 MHz 860 can easily maintain the theoretical network
> > limits between 8 and 9 Mbit/sec with the TCP/IP stack.
> 
> We're using the FEC exclusively, and I'm hoping for a similar story at around
> 8 Mbytes/sec with the TCP/IP stack.

Well, that you can probably forget. :-(
Using a program that sends 1500 bytes udp packets as fast as it
can we get 22-23 Mbit throughput (50MHz 860T).
On vxworks the same program gives about 27 Mbit throughput, and
Windriver claims this is a limitation in the hardware...

(Our PII server connected to the same hub gives about 91 Mbit)

//Marcus
-- 
Signature under construction, please come back later.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: FEC ERROR 41200000 on CLLF-860T

[Dan Malek]

Marcus Sundberg wrote:


> On vxworks the same program gives about 27 Mbit throughput, and
> Windriver claims this is a limitation in the hardware...

Of course they would.  There are lots of stories floating
around about the 8xx hardware limitations.  There are manual
pages and application notes that describe how the 8xx hardware
works (in particular the CPM), and how you compute the capabilities.
In my experience, it has lived up to the publications.  You can
do things that are simply inefficient, or think about the
system architecture or application and program it to do the
things it can perform.

Even if it is "only" 20 something Mbits, how does that affect
your application?  That is more than double what Graham was
expecting for his.

Remember Ethernet is a shared resource without any
deterministic or bandwidth guarantees.  If it doesn't meet
your requirements, perhaps you should choose from the
variety of other interfaces the 8xx provides.

It can run an infinite loop as fast as anything else........



	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: FEC ERROR 41200000 on CLLF-860T

[Marcus Sundberg]

Dan Malek <dan@netx4.com> writes:

> Even if it is "only" 20 something Mbits, how does that affect
> your application?  That is more than double what Graham was
> expecting for his.

No, Graham was expecting/hoping for 8 M _bytes_, not bits.
So in fact it's actual performance is about 1/3 of that, which
I pointed out to him.

//Marcus
-- 
Signature under construction, please come back later.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: FEC ERROR 41200000 on CLLF-860T

[Graham Stoney]

Marcus Sundberg writes:
> No, Graham was expecting/hoping for 8 M _bytes_, not bits.
> So in fact it's actual performance is about 1/3 of that, which
> I pointed out to him.

Marcus is right; we'd like to use all the bandwidth available on the FEC for
optimal system performance. Can anyone elaborate or provide pointers to more
information on the FEC throughput issue? Is there really a hardware problem in
the CPU, or is it a board specific issue?

Thanks,
Graham

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: FEC ERROR 41200000 on CLLF-860T

[Dan Malek]

Graham Stoney wrote:


> Marcus is right;


My apologies.


> ....... Can anyone elaborate or provide pointers to more
> information on the FEC throughput issue? Is there really a hardware problem in
> the CPU, or is it a board specific issue?

The FEC must transfer data at the Ethernet rate because it doesn't
have enough FIFO for an entire packet (it has some clever
optimizations for collision and retrasmit).  The SDMA is always
to system memory, so if the buffers are there, the bits will fly.

It all comes down to what your application is doing, and if
the PPC core has enough computes to provide buffers at the
rate you require.  I am sure I can write a demo application
that will run at the maximum rate, and maybe it will represent
what you are trying to accomplish......I am working on it.


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: FEC ERROR 41200000 on CLLF-860T

[Marcus Sundberg]

greyham@research.canon.com.au (Graham Stoney) writes:

> Marcus Sundberg writes:
> > No, Graham was expecting/hoping for 8 M _bytes_, not bits.
> > So in fact it's actual performance is about 1/3 of that, which
> > I pointed out to him.
> 
> Marcus is right; we'd like to use all the bandwidth available on the FEC for
> optimal system performance. Can anyone elaborate or provide pointers to more
> information on the FEC throughput issue? Is there really a hardware problem in
> the CPU, or is it a board specific issue?

I did some more quick testing.
The bottleneck seems to be the context-switching of Linux. When
increasing the amount of data sent per sendto() call to 11k I got
41Mbit, and with 22k I got 46Mbit. If bandwidth is very important,
maybe you should consider using an 80MHz 860P instead of the 860T.

//Marcus
-- 
Signature under construction, please come back later.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/