Help with an odd problem sharing memory on the ppc460ex

[Ayman El-Khashab <ayman@elkhashab.com>]

  I've got custom boards that have been running for a while on rev A 
460ex parts but when the rev B parts became available some problems 
surfaced.  We are trying to work around the issues in software.  To make 
this simple, I've got 2 460exs connected together via PCI and PCIe so i 
can switch the transport.  For now, I am using PCI.  I've setup the PIMs 
and POMs to map one CPUs DRAM across the bus to the other CPU.  So they 
have a sort of shared memory scheme.  This works fine in many cases.

I think I have a caching problem though even though I think the cache is 
disabled via u-boot.  What happens is that CPU0 will write into CPU1's 
memory.  CPU1 will still see a stale value though ... I can't figure out 
how this happens.  If CPU0 comes back and reads from CPU1's memory 
though, it gets the correct value.   Now if I reduce the frequency of 
the reads/writes to something low (maybe 1 or 2 per second), then very 
often CPU1 will see the change.  I continue to think it is a cache 
issue, but I am not sure.  Also not sure how to work around that.  Is 
that something I need to do via the TLBs?  Is the cache really disabled 
or does Linux mess with the TLB/cache settings?  When I boot Linux I 
reserve the upper 16Mb of memory (from 112-128Mb space) and then mmap 
that entire region into CPU1's space.  Not sure if when it is reserved 
if it is treated differently by Linux or not.

Any help is greatly appreciated.

Thanks
Ayman