8260 and skb implementation

8260 and skb implementation

[Allen Curtis]

We have an 8260 based product and have many production boards with pre-B.3
rev processors. As mentioned on this list, we had to modify all CP related
communications to operate from local RAM to avoid lock-up problems.
Unfortunately this also means that all communication bufferss, especially
Ethernet, must be copied between the two memory spaces. Has anyone
considered moving the SKB memory into local RAM as well? On our hardware, we
have 8MB of local RAM. Is this a reasonable amount of memory for SKB or is
typical usage much higher?

TIA for the sanity check.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: 8260 and skb implementation

[Dan Malek]

Allen Curtis wrote:

> We have an 8260 based product and have many production boards with pre-B.3
> rev processors. As mentioned on this list, we had to modify all CP related
> communications to operate from local RAM to avoid lock-up problems.

I don't remember why you had to do this........

> ....Has anyone
> considered moving the SKB memory into local RAM as well? ....

Yes, except I have never had a platform with a working local bus interface
where this would work.

> ..... we
> have 8MB of local RAM. Is this a reasonable amount of memory for SKB..

Should be plenty.  If you have this much data backed up in the protocol
stack you have big trouble elsewhere.  The only failure mode is the
driver will start dropping receive packets if it can't allocate a buffer.


	-- Dan


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

RE: 8260 and skb implementation

[Allen Curtis]

> > We have an 8260 based product and have many production boards
> with pre-B.3
> > rev processors. As mentioned on this list, we had to modify all
> CP related
> > communications to operate from local RAM to avoid lock-up problems.
>
> I don't remember why you had to do this........

If you have another 60X bus master, such as a PowerSpan PCI controller, you
need to move all CP buffers and descriptors out of main memory or the CP
will freeze.

> > ....Has anyone
> > considered moving the SKB memory into local RAM as well? ....
>
> Yes, except I have never had a platform with a working local bus interface
> where this would work.

I do not know how to help you with hardware. I thought that the EST or
Tundra eval cards had local RAM. (However, we had cache problems with the
Tundra card.) Do you believe moving SKB to local RAM has merit? Any oppinion
about the impact on the zero-copy implementation?


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: 8260 and skb implementation

[Dan Malek]

Allen Curtis wrote:

> If you have another 60X bus master, .....

Ah, right, now I remember.

> I do not know how to help you with hardware. I thought that the EST or
> Tundra eval cards had local RAM.

I had one of the original EST boards long ago.  When trying to use
the local RAM it would hang the CPM.  There are remnants of code in
the FCC driver that if enabled would keep the CPM buffer descriptors
and things in the local RAM.

> ... Do you believe moving SKB to local RAM has merit?

I wouldn't make any decisions without some benchmarks, but I suspect
CPM intensive access is more likely to benefit from using the local RAM.
Things like buffer descriptors and ATM control information will likely
win, I don't think SK bufs will because of the high processor core access.

> ... Any oppinion
> about the impact on the zero-copy implementation?

We should make the change to the receive side so this happens.  The
transmit doesn't copy.  I just copied the code from the 8xx long ago
and never made the change.  We can't properly do zero copy receive
on processors that don't have coherent caches due to alignment and
other information potentially shared in a cache line.

Thanks.


	-- Dan


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

RE: 8260 and skb implementation

[Allen Curtis]

>
> > ... Do you believe moving SKB to local RAM has merit?
>
> ...I don't think SK bufs will because of the high processor core access.

I am not sure what you are saying here?

> > ... Any oppinion
> > about the impact on the zero-copy implementation?
>
> We should make the change to the receive side so this happens.  The
> transmit doesn't copy.  I just copied the code from the 8xx long ago
> and never made the change.  We can't properly do zero copy receive
> on processors that don't have coherent caches due to alignment and
> other information potentially shared in a cache line.

You would think that alignment was a common implementation concern.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: 8260 and skb implementation

[Dan Malek]

Allen Curtis wrote:


>>...I don't think SK bufs will because of the high processor core access.
>
>
> I am not sure what you are saying here?

The advantage of the local RAM is for the CPM to better satisfy its
memory resource requirements by not having to compete with the PowerPC
core (or other bus masters) for 60x bus access.  Since the SK buffers are
predominantly used by the PowerPC core during its IP stack and application
processing, I doubt there is any performance benefit.  However, it isn't
hard to cause SK buffers to be allocated from this memory for testing
purposes, and I would run some benchmarks that represent your application
rather than design by speculation :-)


> You would think that alignment was a common implementation concern.

It's a concern, but we are lucky they are aligned properly.  The transmit
buffers are 16-bit aligned due to the way the headers are built, which
causes trouble for some controllers.


	-- Dan


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: 8260 and skb implementation

[Paul Mackerras]

Dan Malek writes:

> We should make the change to the receive side so this happens.  The
> transmit doesn't copy.  I just copied the code from the 8xx long ago
> and never made the change.  We can't properly do zero copy receive
> on processors that don't have coherent caches due to alignment and
> other information potentially shared in a cache line.

Doing zero-copy receive requires intelligent adaptor firmware/hardware
that can look in the headers and put the packet in different places
depending on what it finds.  Very few adaptors can do this and I
assume the CPM can't.

Cache coherence shouldn't be a problem on transmission generally since
the adaptor is just reading the packet.  On reception we need to make
sure the data area of the skb is cacheline-aligned.  Since the driver
allocates the skbs this should be easy.

Paul.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: 8260 and skb implementation

[Dan Malek]

Paul Mackerras wrote:

> Doing zero-copy receive requires intelligent adaptor firmware/hardware
> that can look in the headers and put the packet in different places
> depending on what it finds.

????

> ....  Very few adaptors can do this and I
> assume the CPM can't.

Piece of cake with the CPM.....I just didn't know the Linux protocol
stack wanted these separated and where to put them if it did.  I can
also assemble packets from fragments just as easily.  I can even
align the headers and data in separate cache lines if you want.

When we discuss the 'zero-copy' I assumed (I guess incorrectly) that
we DMA directly into the receive SK buffer.  In the past on 8xx, we have
allocated uncached memory for the CPM receive buffers and then copied
into the receive SK buffer.  We have to do this because of other
data shared by the cache line and the SK buffer.

> Cache coherence shouldn't be a problem on transmission ....

It never has been and we have always used the sk buffer directly for
DMA output.  It's always been the ability to properly allocate the
sk buffer for the received.

In any case, none of this is an issue for the 82xx, which originally
started this thread.

Thanks.


	-- Dan


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/