Re: Syncing CPU caches from userland on MIPS

[David Daney <ddaney@caviumnetworks.com>]

Aurelien Jarno wrote:
> Hi all,
> 
> This question is not really kernel related, but still MIPS related, I
> hope you don't mind.
> 
> Arnaud Patard and myself are trying to get qemu working on MIPS [1],
> which includes translating TCG code (internal representation) into MIPS
> instructions, that are then executed. Most of the code works, but we 
> have some strange behaviors that seems related to CPU caches.
> 
> The code is written to a buffer, which is then executed. Before the
> execution, the caches are synced using the cacheflush syscall:
> 
> | #include <sys/cachectl.h>
> |  
> | 
> | static inline void flush_icache_range(unsigned long start, unsigned long stop)
> | {
> |     cacheflush ((void *)start, stop-start, ICACHE);
> | }
> 
> It seems this is not enough, as sometimes, some executed code does not
> correspond to the assembly dump of this memory region. This seems to be 
> especially the case of memory regions that are written twice, due to
> relocations:
> 1) a branch instruction is written with an offset of 0
> 2) the offset is patched

Try inserting an 'asm volatile ("sync" ::: "memory");' here.  If that 
fixes things, then we can assume that your cacheflush system call is 
buggy, and would need to add a sync.

> 3) cacheflush is called
> 
> Sometimes the executed code correspond to the code written in 1), which
> means the branch is skipped.
> 
> Does someone knows and/or has example code to correctly sync the CPU 
> caches from userland on MIPS?
> 

http://gcc.gnu.org/viewcvs/trunk/libffi/src/mips/ffi.c

Look at ffi_prep_closure_loc() around line 721.


We also do a similar thing in the kernel in flush_cache_sigtramp(), but 
that is not really userland.


David Daney