Re: IP30: SMP, Almost there?

[Ralf Baechle <ralf@linux-mips.org>]

On Mon, May 18, 2015 at 08:01:07AM -0400, Joshua Kinard wrote:

> What is the relationship between the cache-coherency algorithm and SMP?  IP30
> hardware is supposed to be cache-coherent.  A value of '5' sets the processors
> to "cacheable coherent exclusive on write" (per the R10K manual).  But I am not
> sure why things are still flakey.
> 
> --J

For a cache coherent platform with the R10000 you must use CCA 5 for all
RAM access or all hell will break loose.

For a 32 bit kernel this means the CCA bits of c0_config need to be set
to CCA 5.  64 bit kernels such as those on IP30 are running XKPHYS, not
CSEG0 but still need to use CCA 5.  That means the address bits that
select the CCA need to be set to 5.  Which means kernel addresses will
start with 0xa8.

The same holds true for TLB mappings, they also need to use mode 5.

Also, all accesses to a particular page of physical memory need to use the
same CCA.  Mixing modes is undefined and will in all likelyhood set above
mention hell loose.

All SMP systems need to be coherent between their CPUs.  Traditionally
only SMP MIPS systems are coherent will systems that do not support
multiple processors are non-coherent.  Those may use CCA 3 but again
mixing is not permitted.

Finally there's CCA 2 which is uncached.  That is only sensible for
I/O purposes, data structures such rings as ethernet drivers, gfx bitmaps.
Yet again multiple access modes is not permitted.

The kernel's cca command line option is a bit of a hack meant for hardware
testing and debug.  For a 64 bit kernel these lines in <asm/mach-generic/-
spaces.h> select the suitable base address in XKPHYS:

#ifndef CAC_BASE
#ifdef CONFIG_DMA_NONCOHERENT
#define CAC_BASE                _AC(0x9800000000000000, UL)
#else
#define CAC_BASE                _AC(0xa800000000000000, UL)
#endif
#endif

So you simply need to not select DMA_NONCOHERENT for IP30 and the right
value of 0xa800000000000000 will be used for the kernel base address.

Btw, don't tinker with the CCA bits in c0_config; the firmware will have
configured that correctly for your platform.  The kernel reads that
value and uses it for the CCA field for any TLB mappings.

  Ralf