question on howto use invalidate_dcache_range

question on howto use invalidate_dcache_range

[Stephan Linke]

Hi,

in November there was the discussion about Joakims patch to 8xx_io/enet.c.
(http://lists.linuxppc.org/linuxppc-embedded/200211/msg00122.html)

Since the result of the discussion didn't seam that clear to me I still have a question related to this.

After all is it realy neccessary to use invalidate_dcache range before giving the transmit buffer to the DMA?
  That's one of Joakims last modifications but what exactly is the reason? It's obvious for the flush command but I couldn't figure
out the reason for invalidate.
Beside the cache line alignment problem (causing nearby data to be flushed or invalidated) are there any other problems to be taken
care of?

Thanks, Stephan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: question on howto use invalidate_dcache_range

[Joakim Tjernlund]

> Hi,
>
> in November there was the discussion about Joakims patch to 8xx_io/enet.c.
> (http://lists.linuxppc.org/linuxppc-embedded/200211/msg00122.html)
>
> Since the result of the discussion didn't seam that clear to me I still have a question related to this.
>
> After all is it realy neccessary to use invalidate_dcache range before giving the transmit buffer to the DMA?
>   That's one of Joakims last modifications but what exactly is the reason? It's obvious for the flush command but I
> couldn't figure
> out the reason for invalidate.

Yes, it is neccessary. When you allocate a new skb buffer it you may get a recently used
buffer. The TCP/UPD/IP layer(s) may have modified some of the contents in that buffer before
freeing it. Therefore can the dcache contain dirty(modified) cache lines which belong to the
buffer. When the CPU needs more dcache, it will flush those dirty cache lines to memory which
now belong to the CPM. By invalidating the buffer you tell the CPU to free those cachelines
whithout writeing them to memory.

> Beside the cache line alignment problem (causing nearby data to be flushed or invalidated) are there any other problems
> to be taken
> care of?

There are no cache line alignment problems. There was some concern(expressed by Dan) that
some vital data at the end of the skb buffer could be invalidated due to bad aligment.
See http://lists.linuxppc.org/linuxppc-embedded/200302/msg00135.html
These has been fixed in the skb layer long ago.

The patch has been around for many months now and I have not received any problem
reports.

 Jocke

> Thanks, Stephan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: question on howto use invalidate_dcache_range

[Stephan Linke]

Hi Joke,

thanks for your explaination. Verry plausible to me now.
I guess we have to take care for the alignment anyway since we don't use sk buffer for some reason :).

Thanks, Stephan

> -----Original Message-----
> From: Joakim Tjernlund [mailto:joakim.tjernlund@lumentis.se]
> Sent: Donnerstag, 13. März 2003 08:41
> To: Stephan Linke; Linuxppc-Embedded
> Subject: RE: question on howto use invalidate_dcache_range
>
>
> > Hi,
> >
> > in November there was the discussion about Joakims patch to 8xx_io/enet.c.
> > (http://lists.linuxppc.org/linuxppc-embedded/200211/msg00122.html)
> >
> > Since the result of the discussion didn't seam that clear to me I still have a question related to this.
> >
> > After all is it realy neccessary to use invalidate_dcache range before giving the transmit buffer to the DMA?
> >   That's one of Joakims last modifications but what exactly is the reason? It's obvious for the flush command but I
> > couldn't figure
> > out the reason for invalidate.
>
> Yes, it is neccessary. When you allocate a new skb buffer it you may get a recently used
> buffer. The TCP/UPD/IP layer(s) may have modified some of the contents in that buffer before
> freeing it. Therefore can the dcache contain dirty(modified) cache lines which belong to the
> buffer. When the CPU needs more dcache, it will flush those dirty cache lines to memory which
> now belong to the CPM. By invalidating the buffer you tell the CPU to free those cachelines
> whithout writeing them to memory.
>
> > Beside the cache line alignment problem (causing nearby data to be flushed or invalidated) are there any other problems
> > to be taken
> > care of?
>
> There are no cache line alignment problems. There was some concern(expressed by Dan) that
> some vital data at the end of the skb buffer could be invalidated due to bad aligment.
> See http://lists.linuxppc.org/linuxppc-embedded/200302/msg00135.html
> These has been fixed in the skb layer long ago.
>
> The patch has been around for many months now and I have not received any problem
> reports.
>
>  Jocke
>
> > Thanks, Stephan
>
>
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: question on howto use invalidate_dcache_range

[Joakim Tjernlund]

> Hi Joke,
>
> thanks for your explaination. Verry plausible to me now.
:-)

> I guess we have to take care for the alignment anyway since we don't
> use sk buffer for some reason :).

What do you mean? Do have a custom driver which uses some other
buffer mgmt?

 Jocke


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: question on howto use invalidate_dcache_range

[Stephan Linke]

Hi Joke,
>
> What do you mean? Do have a custom driver which uses some other
> buffer mgmt?
>
That's it.

Thanks, Stephan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: question on howto use invalidate_dcache_range

[Dan Malek]

Stephan Linke wrote:

> After all is it realy neccessary to use invalidate_dcache range before giving the transmit buffer to the DMA?

The logically correct cache operation is to invalidate an incoming buffer
after the DMA has completed, and to (obviously) push an outgoing buffer
prior to starting the DMA.

	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: question on howto use invalidate_dcache_range

[Joakim Tjernlund]

>
> Stephan Linke wrote:
>
> > After all is it realy neccessary to use invalidate_dcache range before giving the transmit buffer to the DMA?
>
> The logically correct cache operation is to invalidate an incoming buffer
> after the DMA has completed

Yes, but you still need to invalidate the buffer before also or invalidate it when the buffer is freed which
may be possible if you are using your own buffer mgmt.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: question on howto use invalidate_dcache_range

[Dan Malek]

Joakim Tjernlund wrote:

> Yes, but you still need to invalidate the buffer before also or invalidate it when the buffer is freed which
> may be possible if you are using your own buffer mgmt.

Why?


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: question on howto use invalidate_dcache_range

[Roland Dreier]

    Joakim> Yes, but you still need to invalidate the buffer before
    Joakim> also or invalidate it when the buffer is freed which may
    Joakim> be possible if you are using your own buffer mgmt.

    Dan> Why?

Is the following not possible:

  CPU has a dirty cache line inside the buffer
  Driver tells device to start DMA
  Device finishes DMA
  CPU writes back dirty cache line
  Driver gets notification that DMA completed
  Driver invalidates cache for buffer

The invalidate above is too late: the data is already corrupted by the
write back of the dirty cache.

 - Roland

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: question on howto use invalidate_dcache_range

[Dan Malek]

Roland Dreier wrote:

> Is the following not possible:

That's a problem in any system design.  Consider what would happen
if you had an architecture that was hardware cache coherent.
Software that writes to buffers at any time that is also a
DMA target will result in unpredictable behavior.  The system
design for what you described requires a higher level of
shared memory software synchronization as it is exactly the
unpredictability that has to be prevented in any SMP design.

What you described is a real problem with cache unaligned
buffers that has to be avoided in systems that do not have
hardware cache coherency.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: question on howto use invalidate_dcache_range

[Roland Dreier]

    Dan> That's a problem in any system design.  Consider what would
    Dan> happen if you had an architecture that was hardware cache
    Dan> coherent.  Software that writes to buffers at any time that
    Dan> is also a DMA target will result in unpredictable behavior.
    Dan> The system design for what you described requires a higher
    Dan> level of shared memory software synchronization as it is
    Dan> exactly the unpredictability that has to be prevented in any
    Dan> SMP design.

I think you misunderstood.  The dirty cache line could be produced by
a write to memory executed long before the DMA is initiated.  In a
hardware cache coherent architecture, the bus controller would just
snoop the CPU's cache and invalidate the dirty cache line.

In any case the current Linux kernel seems to do things the way I
believe is necessary: pci_map_single(...PCI_DMA_FROMDEVICE)
invalidates the cache (and is done _before_ the DMA is initiated),
while pci_unmap_single() is a NOOP.

 - Roland


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: question on howto use invalidate_dcache_range

[Joakim Tjernlund]

> Joakim Tjernlund wrote:
>
> > Yes, but you still need to invalidate the buffer before also or invalidate it when the buffer is freed which
> > may be possible if you are using your own buffer mgmt.
>
> Why?

See http://lists.linuxppc.org/linuxppc-embedded/200303/msg00184.html

 Jocke

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/