Cache coherency question

Cache coherency question

[Martin, Tim]

I'm using an MVME6100 with Linux 2.6.14 and experiencing what I think is
a cache coherency problem.  An external PCIX master performs a DMA
transfer of a well known data pattern into SDRAM, but when the device
driver goes to look at the data it is mostly okay, but occasionally has
garbage.

I'm allocating the memory using alloc_skb, then giving a PCI translated
pointer to the PCIX master.  Since I think it is a cache coherency
problem, I tried to use the dma_cache_inv macro to invalidate the cache
before looking at the data, but realized that for this board, that
function does nothing since CONFIG_NOT_COHERENT_CACHE is undefined.  So
questions:

1) Should the processor bus cache snooping actually work on an MVME6100?
E.g. is it correct that CONFIG_NOT_COHERENT_CACHE is undefined?
1a) Does this type of bus snooping work when a bus master physically
external to the PowerPC chip is doing the transfer?

2) If this really is a cache coherency problem, are there other cache
management functions available in ppc linux besides the dma_cache_inv,
dma_cache_wback, dma_cache_wback_inv macros I should be looking at?

Tim

RE: Cache coherency question

[Liu Dave-r63238]

Tim> I'm using an MVME6100 with Linux 2.6.14 and experiencing what=20
Tim> I think is a cache coherency problem.  An external PCIX=20
Tim> master performs a DMA transfer of a well known data pattern=20
Tim> into SDRAM, but when the device driver goes to look at the=20
Tim> data it is mostly okay, but occasionally has garbage.

If the hardware is reliable, I think you are right. The cache coherency
problem.

Tim> I'm allocating the memory using alloc_skb, then giving a PCI=20
Tim> translated pointer to the PCIX master.  Since I think it is a=20
Tim> cache coherency problem, I tried to use the dma_cache_inv=20
Tim> macro to invalidate the cache before looking at the data, but=20
Tim> realized that for this board, that function does nothing=20
Tim> since CONFIG_NOT_COHERENT_CACHE is undefined.  So
Tim> questions:

If the CONFIG_NOT_COHERENT_CACHE is not set.
You are assuming and using the hardware cache coherency mechanism.

Tim> 1) Should the processor bus cache snooping actually work on=20
Tim> an MVME6100?
Tim> E.g. is it correct that CONFIG_NOT_COHERENT_CACHE is undefined?

Yes, you can undefine the CONFIG_NOT_COHERENT_CACHE, but you must
make sure the host bridge did the snooping setup for PCI inbound
transaction.
If you setup the snooping window, I think the host bridge will assert
/GLB signal
to processor. The processor will snoop the 60x/MPX bus to keep cache
coherence.

Also, you can define the CONFIG_NOT_COHERENT_CACHE, then you are
assuming
The system has not hardware coherency. You need use the software to keep
the cache
coherency.

Tim> 1a) Does this type of bus snooping work when a bus master=20
Tim> physically external to the PowerPC chip is doing the transfer?

As above, you must tell the bridge what address need snoop. If you did,
The proceesor will do snooping.

Tim> 2) If this really is a cache coherency problem, are there=20
Tim> other cache management functions available in ppc linux=20
Tim> besides the dma_cache_inv, dma_cache_wback,=20
Tim> dma_cache_wback_inv macros I should be looking at?

I think it is a cache coherency problem,
if you define the CONFIG_NOT_COHERENT_CACHE
You can get these functions, dma_map_single


-Dave

RE: Cache coherency question

[Martin, Tim]

>=20
> Tim> I'm using an MVME6100 with Linux 2.6.14 and experiencing what I=20
> Tim> think is a cache coherency problem.  An external PCIX master=20
> Tim> performs a DMA transfer of a well known data pattern into SDRAM,=20
> Tim> but when the device driver goes to look at the data it is mostly=20
> Tim> okay, but occasionally has garbage.
>=20
> If the hardware is reliable, I think you are right. The cache=20
> coherency problem.

The hardware is reliable.

>=20
> Tim> 1) Should the processor bus cache snooping actually work on an=20
> Tim> MVME6100?
> Tim> E.g. is it correct that CONFIG_NOT_COHERENT_CACHE is undefined?
>=20
> Yes, you can undefine the CONFIG_NOT_COHERENT_CACHE, but you=20
> must make sure the host bridge did the snooping setup for PCI=20
> inbound transaction.
> If you setup the snooping window, I think the host bridge=20
> will assert /GLB signal to processor. The processor will=20
> snoop the 60x/MPX bus to keep cache coherence.
>=20

Ah.  This must be the problem.  I have a few PCI devices, and on one of
them it looked like snooping was working.  I just assumed the other
device was setup correctly.

> Also, you can define the CONFIG_NOT_COHERENT_CACHE, then you=20
> are assuming The system has not hardware coherency. You need=20
> use the software to keep the cache coherency.
>=20

I tried this, and got compiler errors.

>=20
> Tim> 2) If this really is a cache coherency problem, are there other=20
> Tim> cache management functions available in ppc linux besides the=20
> Tim> dma_cache_inv, dma_cache_wback, dma_cache_wback_inv=20
> macros I should=20
> Tim> be looking at?
>=20
> I think it is a cache coherency problem, if you define the=20
> CONFIG_NOT_COHERENT_CACHE You can get these functions, dma_map_single
>=20

I added some inline assembly dcbi/dcbf (invalidate/flush) instructions
to the particular code in question, and the problems went away.  So
definitely a cache problem.  As I said above, defining
CONFIG_NOT_COHERENT_CACHE causes compiler errors, so I'm going to look
into this more.  I suppose whatever file implements the
include/linux/dma-mapping.h stuff isn't BSP specific, so its probably
just not being compiled in?  Will look into it.

Thanks for you help,
Tim

RE: Cache coherency question

[Liu Dave-r63238]

> Ah.  This must be the problem.  I have a few PCI devices, and=20
> on one of them it looked like snooping was working.  I just=20
> assumed the other device was setup correctly.

Why one is looked like snooping was working, and the other device
not working?

> > Also, you can define the CONFIG_NOT_COHERENT_CACHE, then you are=20
> > assuming The system has not hardware coherency. You need use the=20
> > software to keep the cache coherency.
> >=20
>=20
> I tried this, and got compiler errors.

What is the file got compiler errors? I notice that the
/ev64360_defconfig
Did define CONFIG_NOT_COHERENT_CACHE.


> I added some inline assembly dcbi/dcbf (invalidate/flush)=20
> instructions to the particular code in question, and the=20
> problems went away.  So definitely a cache problem.  As I=20
> said above, defining CONFIG_NOT_COHERENT_CACHE causes=20
> compiler errors, so I'm going to look into this more.  I=20
> suppose whatever file implements the=20
> include/linux/dma-mapping.h stuff isn't BSP specific, so its=20
> probably just not being compiled in?  Will look into it.

Somebody said from maillist. The bridge MV64360 seems
having some issue about cache coherent. I don't know if it is really?

-Dave

RE: Cache coherency question

[Li Yang-r58472]

> -----Original Message-----
> From: linuxppc-embedded-bounces+leoli=3Dfreescale.com@ozlabs.org
> [mailto:linuxppc-embedded-bounces+leoli=3Dfreescale.com@ozlabs.org] On
Behalf Of
> Liu Dave-r63238
> Sent: Monday, August 28, 2006 11:18 AM
> To: Martin, Tim; ppc
> Subject: RE: Cache coherency question
>=20
> > Ah.  This must be the problem.  I have a few PCI devices, and
> > on one of them it looked like snooping was working.  I just
> > assumed the other device was setup correctly.
>=20
> Why one is looked like snooping was working, and the other device
> not working?
>=20
> > > Also, you can define the CONFIG_NOT_COHERENT_CACHE, then you are
> > > assuming The system has not hardware coherency. You need use the
> > > software to keep the cache coherency.
> > >
> >
> > I tried this, and got compiler errors.

Which arch are you using?  powerpc or ppc?  CONFIG_NOT_COHERENT_CACHE
doesn't seem to be ported to powerpc arch completely.

>=20
> What is the file got compiler errors? I notice that the
> /ev64360_defconfig
> Did define CONFIG_NOT_COHERENT_CACHE.
>=20
>=20
> > I added some inline assembly dcbi/dcbf (invalidate/flush)
> > instructions to the particular code in question, and the
> > problems went away.  So definitely a cache problem.  As I
> > said above, defining CONFIG_NOT_COHERENT_CACHE causes
> > compiler errors, so I'm going to look into this more.  I
> > suppose whatever file implements the
> > include/linux/dma-mapping.h stuff isn't BSP specific, so its
> > probably just not being compiled in?  Will look into it.
>=20
> Somebody said from maillist. The bridge MV64360 seems
> having some issue about cache coherent. I don't know if it is really?

Looking from source code, there is such problem for MV64360.  It's
likely that MV54360 has the same problem.

RE: Cache coherency question

[Liu Dave-r63238]

<snip>
> Looking from source code, there is such problem for MV64360. =20
> It's likely that MV54360 has the same problem.

What is the MV54360?=20

-Dave

RE: Cache coherency question

[Li Yang-r58472]

> -----Original Message-----
> From: Liu Dave-r63238
> Sent: Monday, August 28, 2006 3:00 PM
> To: Li Yang-r58472; 'Martin, Tim'; 'ppc'
> Subject: RE: Cache coherency question
>=20
> <snip>
> > Looking from source code, there is such problem for MV64360.
> > It's likely that MV54360 has the same problem.
>=20
> What is the MV54360?

>From document at
http://www.motorola.com/mot/doc/5/5503_MotDoc.pdf#search=3D%22MV54360%22
The board uses MV54360.

I don't know if it is a typo or new variant though.
>=20
> -Dave

Re: Cache coherency question

[Gerhard Jaeger]

On Monday 28 August 2006 09:08, Li Yang-r58472 wrote:
> 
> > -----Original Message-----
> > From: Liu Dave-r63238
> > Sent: Monday, August 28, 2006 3:00 PM
> > To: Li Yang-r58472; 'Martin, Tim'; 'ppc'
> > Subject: RE: Cache coherency question
> > 
> > <snip>
> > > Looking from source code, there is such problem for MV64360.
> > > It's likely that MV54360 has the same problem.
> > 
> > What is the MV54360?
> 
> From document at
> http://www.motorola.com/mot/doc/5/5503_MotDoc.pdf#search=%22MV54360%22
> The board uses MV54360.
> 
> I don't know if it is a typo or new variant though.

It's definitely a typo. In the block diagramm you'll see Discovery II
as system bridge and this one is the Marvel MV64360 (at least on the MVME6100).

-Gerhard

-- 
Gerhard Jaeger <gjaeger@sysgo.com>            
SYSGO AG                      Embedded and Real-Time Software
www.sysgo.com | www.elinos.com | www.pikeos.com | www.osek.de