[PATCH 2/2] arm64: Use last level TLBI for user pte changes

[will.deacon@arm.com (Will Deacon)]

On Fri, Jul 24, 2015 at 09:59:56AM +0100, Catalin Marinas wrote:
> The flush_tlb_page() function is used on user address ranges when PTEs
> (or PMDs/PUDs for huge pages) were changed (attributes or clearing). For
> such cases, it is more efficient to invalidate only the last level of
> the TLB with the "tlbi vale1is" instruction.
> 
> In the TLB shoot-down case, the TLB caching of the intermediate page
> table levels (pmd, pud, pgd) is handled by __flush_tlb_pgtable() via the
> __(pte|pmd|pud)_free_tlb() functions and it is not deferred to
> tlb_finish_mmu() (as of commit 285994a62c80 - "arm64: Invalidate the TLB
> corresponding to intermediate page table levels"). The tlb_flush()
> function only needs to invalidate the TLB for the last level of page
> tables; a new arm64-specific __flush_tlb_page_range() function performs
> only the last level TLBI.

[...]

> diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
> index 1643908eb5f3..48794ab79cc0 100644
> --- a/arch/arm64/include/asm/tlbflush.h
> +++ b/arch/arm64/include/asm/tlbflush.h
> @@ -87,7 +87,7 @@ static inline void flush_tlb_page(struct vm_area_struct *vma,
>  		((unsigned long)ASID(vma->vm_mm) << 48);
>  
>  	dsb(ishst);
> -	asm("tlbi	vae1is, %0" : : "r" (addr));
> +	asm("tlbi	vale1is, %0" : : "r" (addr));
>  	dsb(ish);
>  }
>  
> @@ -97,6 +97,26 @@ static inline void flush_tlb_page(struct vm_area_struct *vma,
>   */
>  #define MAX_TLB_RANGE	(1024UL << PAGE_SHIFT)
>  
> +static inline void __flush_tlb_page_range(struct vm_area_struct *vma,
> +					  unsigned long start, unsigned long end)
> +{
> +	unsigned long asid = (unsigned long)ASID(vma->vm_mm) << 48;
> +	unsigned long addr;
> +
> +	if ((end - start) > MAX_TLB_RANGE) {
> +		flush_tlb_mm(vma->vm_mm);
> +		return;
> +	}
> +
> +	start = asid | (start >> 12);
> +	end = asid | (end >> 12);
> +
> +	dsb(ishst);
> +	for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12))
> +		asm("tlbi vale1is, %0" : : "r"(addr));
> +	dsb(ish);
> +}

This is identical to __flush_tlb_range apart from the asm op.

What happens if you add a "const bool leaf" parameter and stick a
conditional inside the loop?

Will