[RFC PATCH] arm64: Elide dsb in kernel TLB invalidations

[RFC PATCH] arm64: Elide dsb in kernel TLB invalidations

[Dev Jain]

dsb(ishst) is used to ensure that prior pagetable updates are completed.
But, set_pmd/set_pud etc already issue a dsb-isb sequence for the exact
same purpose. Therefore, we can elide the dsb in kernel tlb invalidation.

There were no issues observed while running mm selftests, including
test_vmalloc.sh selftest to stress the vmalloc subsystem.

Signed-off-by: Dev Jain <dev.jain@arm.com>
---
 arch/arm64/include/asm/tlbflush.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
index eba1a98657f1..9b4adf1ee45e 100644
--- a/arch/arm64/include/asm/tlbflush.h
+++ b/arch/arm64/include/asm/tlbflush.h
@@ -508,7 +508,7 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end
 		return;
 	}
 
-	dsb(ishst);
+	/* dsb(ishst) not needed as callers (set_pxd) have that */
 	__flush_tlb_range_op(vaale1is, start, pages, stride, 0,
 			     TLBI_TTL_UNKNOWN, false, lpa2_is_enabled());
 	dsb(ish);
@@ -523,7 +523,7 @@ static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
 {
 	unsigned long addr = __TLBI_VADDR(kaddr, 0);
 
-	dsb(ishst);
+	/* dsb(ishst) not needed as callers (set_pxd) have that */
 	__tlbi(vaae1is, addr);
 	dsb(ish);
 	isb();
-- 
2.30.2

Re: [RFC PATCH] arm64: Elide dsb in kernel TLB invalidations

[Dev Jain]

Forgot to add Ryan.

On 22/05/25 5:14 pm, Dev Jain wrote:
> dsb(ishst) is used to ensure that prior pagetable updates are completed.
> But, set_pmd/set_pud etc already issue a dsb-isb sequence for the exact
> same purpose. Therefore, we can elide the dsb in kernel tlb invalidation.
>
> There were no issues observed while running mm selftests, including
> test_vmalloc.sh selftest to stress the vmalloc subsystem.
>
> Signed-off-by: Dev Jain <dev.jain@arm.com>
> ---
>   arch/arm64/include/asm/tlbflush.h | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
> index eba1a98657f1..9b4adf1ee45e 100644
> --- a/arch/arm64/include/asm/tlbflush.h
> +++ b/arch/arm64/include/asm/tlbflush.h
> @@ -508,7 +508,7 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end
>   		return;
>   	}
>   
> -	dsb(ishst);
> +	/* dsb(ishst) not needed as callers (set_pxd) have that */
>   	__flush_tlb_range_op(vaale1is, start, pages, stride, 0,
>   			     TLBI_TTL_UNKNOWN, false, lpa2_is_enabled());
>   	dsb(ish);
> @@ -523,7 +523,7 @@ static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
>   {
>   	unsigned long addr = __TLBI_VADDR(kaddr, 0);
>   
> -	dsb(ishst);
> +	/* dsb(ishst) not needed as callers (set_pxd) have that */
>   	__tlbi(vaae1is, addr);
>   	dsb(ish);
>   	isb();

Re: [RFC PATCH] arm64: Elide dsb in kernel TLB invalidations

[Catalin Marinas]

On Thu, May 22, 2025 at 05:14:14PM +0530, Dev Jain wrote:
> dsb(ishst) is used to ensure that prior pagetable updates are completed.
> But, set_pmd/set_pud etc already issue a dsb-isb sequence for the exact
> same purpose. Therefore, we can elide the dsb in kernel tlb invalidation.
> 
> There were no issues observed while running mm selftests, including
> test_vmalloc.sh selftest to stress the vmalloc subsystem.
> 
> Signed-off-by: Dev Jain <dev.jain@arm.com>
> ---
>  arch/arm64/include/asm/tlbflush.h | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
> index eba1a98657f1..9b4adf1ee45e 100644
> --- a/arch/arm64/include/asm/tlbflush.h
> +++ b/arch/arm64/include/asm/tlbflush.h
> @@ -508,7 +508,7 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end
>  		return;
>  	}
>  
> -	dsb(ishst);
> +	/* dsb(ishst) not needed as callers (set_pxd) have that */
>  	__flush_tlb_range_op(vaale1is, start, pages, stride, 0,
>  			     TLBI_TTL_UNKNOWN, false, lpa2_is_enabled());
>  	dsb(ish);
> @@ -523,7 +523,7 @@ static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
>  {
>  	unsigned long addr = __TLBI_VADDR(kaddr, 0);
>  
> -	dsb(ishst);
> +	/* dsb(ishst) not needed as callers (set_pxd) have that */
>  	__tlbi(vaae1is, addr);
>  	dsb(ish);
>  	isb();

What about __set_pte()? We only issue (or rather queue) a barrier if we
set a valid kernel pte. I recall we added them for the case where a TLBI
won't happen, see commit 7f0b1bf04511 ("arm64: Fix barriers used for
page table modifications"). When we clear a PTE, we rely on the TLB
maintenance to issue barriers.

Maybe something that can be optimised following Ryan's reworking but I
don't think it is safe to remove them now, given that
__set_pte_complete() skips the barriers for invalid ptes. Possibly the
__flush_tlb_kernel_pgtable() is alright but not the
flush_tlb_kernel_range() one.

-- 
Catalin

Re: [RFC PATCH] arm64: Elide dsb in kernel TLB invalidations

[Dev Jain]

On 22/05/25 8:11 pm, Catalin Marinas wrote:
> On Thu, May 22, 2025 at 05:14:14PM +0530, Dev Jain wrote:
>> dsb(ishst) is used to ensure that prior pagetable updates are completed.
>> But, set_pmd/set_pud etc already issue a dsb-isb sequence for the exact
>> same purpose. Therefore, we can elide the dsb in kernel tlb invalidation.
>>
>> There were no issues observed while running mm selftests, including
>> test_vmalloc.sh selftest to stress the vmalloc subsystem.
>>
>> Signed-off-by: Dev Jain <dev.jain@arm.com>
>> ---
>>   arch/arm64/include/asm/tlbflush.h | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h
>> index eba1a98657f1..9b4adf1ee45e 100644
>> --- a/arch/arm64/include/asm/tlbflush.h
>> +++ b/arch/arm64/include/asm/tlbflush.h
>> @@ -508,7 +508,7 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end
>>   		return;
>>   	}
>>   
>> -	dsb(ishst);
>> +	/* dsb(ishst) not needed as callers (set_pxd) have that */
>>   	__flush_tlb_range_op(vaale1is, start, pages, stride, 0,
>>   			     TLBI_TTL_UNKNOWN, false, lpa2_is_enabled());
>>   	dsb(ish);
>> @@ -523,7 +523,7 @@ static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
>>   {
>>   	unsigned long addr = __TLBI_VADDR(kaddr, 0);
>>   
>> -	dsb(ishst);
>> +	/* dsb(ishst) not needed as callers (set_pxd) have that */
>>   	__tlbi(vaae1is, addr);
>>   	dsb(ish);
>>   	isb();
> What about __set_pte()? We only issue (or rather queue) a barrier if we
> set a valid kernel pte. I recall we added them for the case where a TLBI
> won't happen, see commit 7f0b1bf04511 ("arm64: Fix barriers used for
> page table modifications"). When we clear a PTE, we rely on the TLB
> maintenance to issue barriers.


I see, so I think one example also can be __set_fixmap -> __pte_clear -> 
__set_pte.

My original motivation was that it would be good to identify all the callers

instead of unconditionally issuing it for every tlb flush.


>
> Maybe something that can be optimised following Ryan's reworking but I
> don't think it is safe to remove them now, given that
> __set_pte_complete() skips the barriers for invalid ptes. Possibly the
> __flush_tlb_kernel_pgtable() is alright but not the
> flush_tlb_kernel_range() one.


Sure, I think Ryan's optimizations haven't landed yet? I'll take a look 
after that.