Re: Syncing CPU caches from userland on MIPS

[Ralf Baechle <ralf@linux-mips.org>]

On Wed, Nov 25, 2009 at 03:01:05PM +0100, Florian Lohoff wrote:

> > | static inline void flush_icache_range(unsigned long start, unsigned long stop)
> > | {
> > |     cacheflush ((void *)start, stop-start, ICACHE);
> > | }
> 
> Would this only evict stuff from the ICACHE? When trying to execute
> a just written buffer and with a writeback DCACHE you would need to 
> explicitly writeback the DCACHE to memory and invalidate the ICACHE.

No; ICACHE really means the kernel will do whatever it takes to make the
I-cache coherent with the D-cache.  For most processors this requires
writing back the D-cache to S-cache or if no S-cache present, memory
then invalidating the I-cache.

> > It seems this is not enough, as sometimes, some executed code does not
> > correspond to the assembly dump of this memory region. This seems to be 
> > especially the case of memory regions that are written twice, due to
> > relocations:
> > 1) a branch instruction is written with an offset of 0
> > 2) the offset is patched
> > 3) cacheflush is called
> > 
> > Sometimes the executed code correspond to the code written in 1), which
> > means the branch is skipped.
> 
> Which proves my theory - as long as you have cache pressure you will happily
> writeback the contents to memory before trying to execute (you invalidate
> the ICACHE above) - In case you DCACHE does not suffer from pressure
> the contents will not been written back and you'll execute stale code.

You could - on a uni-processor system - do something like this to invalidate
the cache:

int array[32768 / sizeof(int)];	/* size of D-cache on SB1250  */

blow_away_d(void)
{
	memset(array, 0, sizeof(array) / sizeof(int) - 2);
	array[sizeof(int) - 2] = 0x03E00008;	/* jr $ra	*/
	array[sizeof(int) - 1] = 0x00000000;	/* nop		*/
}

blow_away_i(void)
{
	void (*fn)(void) = (void *) array;

	blow_away_d();
	fn();
}

blow_away_setup(void)
{
	blow_away_d();
	cacheflush(array, sizeof(array), BCACHE);
}

The memset and array assignment operations result in victim writebacks
that is basically an D-cache writeback.  Later on executing the code in
the array will result in the entire I-cache getting refilled so anything
of your qemu code array that might happen to live in the I-cache will be
discarded out of the cache.

  Ralf