Re: [PATCH 04/36] Add Cavium OCTEON processor support files to arch/mips/mm.

[David Daney <ddaney@caviumnetworks.com>]

Ralf Baechle wrote:
> On Mon, Oct 27, 2008 at 05:02:36PM -0700, David Daney wrote:
[...]
>> +static void octeon_flush_icache_all_cores(struct vm_area_struct *vma)
>> +{
>> +	extern struct plat_smp_ops *mp_ops;	/* private */
>> +#ifdef CONFIG_SMP
>> +	int i;
>> +	int cpu;
>> +#endif
>> +
>> +	preempt_disable();
>> +#ifdef CONFIG_SMP
>> +	cpu = smp_processor_id();
>> +#endif
>> +	mb();
>> +
>> +	/* If we have a vma structure, we only need to worry about cores it
>> +	   has been used on */
>> +	if (vma) {
>> +#ifdef CONFIG_SMP
>> +		for (i = 0; i < NR_CPUS; i++)
>> +			if (cpu_isset(i, vma->vm_mm->cpu_vm_mask) && i != cpu)
>> +				mp_ops->send_ipi_single(i, SMP_ICACHE_FLUSH);
>> +#endif
>> +		asm volatile ("synci 0($0)\n");
>> +	} else {
>> +		/* No extra info available. Flush the icache on all cores that
>> +		   are online */
>> +#ifdef CONFIG_SMP
>> +		for (i = 0; i < NR_CPUS; i++)
>> +			if (cpu_online(i) && i != cpu)
>> +				mp_ops->send_ipi_single(i, SMP_ICACHE_FLUSH);
>> +#endif
>> +		asm volatile ("synci 0($0)\n");
>> +	}
> 
> You can avoid the entire #ifdef CONFIG_SMP mess with for_each_online_cpu().
> 
> For some workloads IPIs can be performance limiting.  You can avoid them
> entirely if the mm that is being flush out of the CPU is not active on the
> remote processor by simply setting the remote's context for that mm to
> zero.  This means the remote CPU will allocate a new context.  That's a
> matter of a few instructions as long as the remote's ASID counter doesn't
> overflow that is 255 out of 256 times.
> 
> The price to pay is to accept that both the vtag'ed I-cache and the TLB
> of the remote need to be reloaded which seems relativly harmless compared
> to the alternatives.
> 

Unfortunately, I don't think that will work.  In order to do the 
optimization you must check some things in the mm.  To do this in a race 
free manner, you need to hold the mmap_sem.  flush_icache_all is 
sometimes called without acquiring the mmap_sem, and I have not figured 
out how to acquire it in such a manner that the system doesn't crash.

David Daney