Re: 8260 + cache/dma issues

Re: 8260 + cache/dma issues

[ashish anand]

though i haven't dealt with 8260 specific cache consistency protocol  and all , but feel using
pci_alloc_consistent may be cause.
also snooping should be clearly understood whether it is presenting a consistent view
between cpu and devices or diffeent cpu in a smp setup.

David Ashley wrote:
>
> I'm running into repeated problems related to cache problems on
> ppc 8260 based hardware. The kernel is based on Montevista's 2.4.2.
> The same problem keeps coming up again and again. Here it is:
>
> If some external device, such as the pci usb device, writes into system
> ram, the 8260 doesn't get the changes--instead when the 8260 reads the
> ram it uses the data in the cache, defeating the whole purpose. We're
> using the usb-ohci.c driver. In order to get it to work I've had to hack
> the driver to instead of using pci_alloc_consistent, allocate memory
> out of a section of local bus ram, which is non-cacheable. The burden
> of doing this for every driver is just too great.
>
> The original driver itself assumes that the dma works properly, meaning it
> does nothing to invalidate the cache or cause a refresh/flush/update or
> whatever. The driver initiates a transfer, the usb writes into ram, an
> interrupt occurs, the driver examines the buffer--and crashes and generally
> doesn't work because the data it sees isn't in ram, it is what was in the
> cache.
>

> My belief is there is something misconfigured on the hardware level. Meaning
> the cache should work properly. In studying the 8260 docs I read hints at
> snooping, but no details. My understanding is that snooping is when the
> 8260 watches the bus to see what activity there is, and if there is a write
> to something the 8260 holds in a cache, the cache is updated accordingly.
> Perhaps this snooping is turned off?
>
> Also there is talk of C + R bits in the page tables (C = changed, R =
> referenced). Possibly is the external dma mechanism supposed to update the
> C + R bits in the page tables itself, causing the 8260 to realize something
> has been done to the page and so it needs to be updated?
>
> The driver doesn't ask for any special non-cacheable memory, just dma
> capable. Following this through in the code that means in the case of
> the ppc, all of dram because there is no limit on the dma range. On the
> i386 tree dma appears limited to the first 16 megabytes. But nothing appears
> to be demanding non-cacheable memory.
>
> I guess I have several questions:
> 1) How is the system supposed to work so that the cpu and other pci devices
>    have the same concept of the contents of memory?
> 2) What is missing in our system that is causing this not to work?
> 3) What is snooping, and can it be turned on or off by the 8260, or is it
>    fixed in the hardware?
> 4) Am I misunderstanding how the linux drivers where, such that when they
>    allocate memory for dma transfers, it is automatically marked as
>    non-cacheable?
> 5) While I'm at it, I'm getting lock ups related to the external pci device
>    writing into dram. I've heard mention of cache deadlocks, but I don't know
>    details. Any pointers to documentation will be helpful.
>
> Thanks very much for any ideas/advice.
>
> -Dave
> dash@xdr.com
>

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

8260 + cache/dma issues

[David Ashley]

I'm running into repeated problems related to cache problems on
ppc 8260 based hardware. The kernel is based on Montevista's 2.4.2.
The same problem keeps coming up again and again. Here it is:

If some external device, such as the pci usb device, writes into system
ram, the 8260 doesn't get the changes--instead when the 8260 reads the
ram it uses the data in the cache, defeating the whole purpose. We're
using the usb-ohci.c driver. In order to get it to work I've had to hack
the driver to instead of using pci_alloc_consistent, allocate memory
out of a section of local bus ram, which is non-cacheable. The burden
of doing this for every driver is just too great.

The original driver itself assumes that the dma works properly, meaning it
does nothing to invalidate the cache or cause a refresh/flush/update or
whatever. The driver initiates a transfer, the usb writes into ram, an
interrupt occurs, the driver examines the buffer--and crashes and generally
doesn't work because the data it sees isn't in ram, it is what was in the
cache.

My belief is there is something misconfigured on the hardware level. Meaning
the cache should work properly. In studying the 8260 docs I read hints at
snooping, but no details. My understanding is that snooping is when the
8260 watches the bus to see what activity there is, and if there is a write
to something the 8260 holds in a cache, the cache is updated accordingly.
Perhaps this snooping is turned off?

Also there is talk of C + R bits in the page tables (C = changed, R =
referenced). Possibly is the external dma mechanism supposed to update the
C + R bits in the page tables itself, causing the 8260 to realize something
has been done to the page and so it needs to be updated?

The driver doesn't ask for any special non-cacheable memory, just dma
capable. Following this through in the code that means in the case of
the ppc, all of dram because there is no limit on the dma range. On the
i386 tree dma appears limited to the first 16 megabytes. But nothing appears
to be demanding non-cacheable memory.

I guess I have several questions:
1) How is the system supposed to work so that the cpu and other pci devices
   have the same concept of the contents of memory?
2) What is missing in our system that is causing this not to work?
3) What is snooping, and can it be turned on or off by the 8260, or is it
   fixed in the hardware?
4) Am I misunderstanding how the linux drivers where, such that when they
   allocate memory for dma transfers, it is automatically marked as
   non-cacheable?
5) While I'm at it, I'm getting lock ups related to the external pci device
   writing into dram. I've heard mention of cache deadlocks, but I don't know
   details. Any pointers to documentation will be helpful.

Thanks very much for any ideas/advice.

-Dave
dash@xdr.com

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: 8260 + cache/dma issues

[Wolfgang Denk]

In message <200112260427.fBQ4Rvi00224@dave.home> you wrote:
>
> I'm running into repeated problems related to cache problems on
> ppc 8260 based hardware. The kernel is based on Montevista's 2.4.2.
> The same problem keeps coming up again and again. Here it is:

Which mask revision are you using?

Are you running in 60x bus mode, or in 8260 bus mode?

Is ther L2 cache on your board?

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
"Language shapes the way we think, and determines what we  can  think
about."                                                 - B. L. Whorf

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: 8260 + cache/dma issues

[David Ashley]

>> I'm running into repeated problems related to cache problems on
>> ppc 8260 based hardware. The kernel is based on Montevista's 2.4.2.
>> The same problem keeps coming up again and again. Here it is:
>
>Which mask revision are you using?
>
>Are you running in 60x bus mode, or in 8260 bus mode?
>
>Is ther L2 cache on your board?
>
>Wolfgang Denk

The mask is A.1.

It is running in 60x bus mode.

There is no L2 cache.

-Dave

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: 8260 + cache/dma issues

[David Ashley]

WD forgot to cc the newsgroup but I thought this worth including here.

Thanks for the information! I don't know if that applies to our situation
though, it seems impossible to believe that motorola could release a chip
with such a critical design flaw. The cache support is supposed to be a
selling point for the chip.

We're using FCC's with ethernet with the data cache enabled, and that
appears to be working ok, at least until other stuff starts happening
(such as a pci device acting as a bus master writing into local bus ram,
while doing some heavy process that uses __copy_tofrom_user, and there
is incoming ethernet traffic, in which case lockups occur).

-Dave

>>
>> >Which mask revision are you using?
>> >
>> >Are you running in 60x bus mode, or in 8260 bus mode?
>> >
>> >Is ther L2 cache on your board?
>> >
>> >Wolfgang Denk
>>
>> The mask is A.1.
>>
>> It is running in 60x bus mode.
>
>I'm afraid there is bad news for you. From what I've seen so far, all
>systems with rev. A.1 (0K26N) silicon had serious problems when doing
>DMA transfers with data cache enabled (usually this shows when  using
>one of the FCC's with some traffic on the ethernet).
>
>Either you turn off the data cache, or you use  a  more  recent  mask
>revision (at least rev. C.2 - 6K23A).
>
>The kernel source on our FTP server has a config option to  turn  off
>the  data  cache  (CONFIG_DCACHE_DISABLE), but be warned: performance
>breaks down to a f*cking 10%; you can find our kernel at
>ftp://ftp.denx.de/pub/LinuxPPC/usr/src/linux-2.4.4-2001-11-24.tar.bz2
>
>Hope this helps,
>
>Wolfgang Denk
>
>--
>Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
>Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
>Quote from a recent meeting:   "We are going to continue having these
>meetings everyday until I find out why no work is getting done."

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/