[PATCH 0/2] arm64 head.S fixes

[PATCH 0/2] arm64 head.S fixes

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

Ensure the MMU is off before changing HCR_EL2.E2H when entering at EL2.
Drop MMU disable workarounds where they are not needed.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>

Ard Biesheuvel (2):
  arm64/head: Drop unnecessary pre-disable-MMU workaround
  arm64/head: Disable MMU at EL2 before clearing HCR_EL2.E2H

 arch/arm64/kernel/head.S | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

-- 
2.44.0.683.g7961c838ac-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 1/2] arm64/head: Drop unnecessary pre-disable-MMU workaround

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

The Falkor erratum that results in the need for an ISB before clearing
the M bit in SCTLR_ELx only applies to execution at exception level x,
and so the workaround is not needed when disabling the EL1 MMU while
running at EL2.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/arm64/kernel/head.S | 2 --
 1 file changed, 2 deletions(-)

diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index 06234c3a15f3..b8bbd72cb194 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -323,13 +323,11 @@ SYM_INNER_LABEL(init_el2, SYM_L_LOCAL)
 	cbz	x0, 2f
 
 	/* Set a sane SCTLR_EL1, the VHE way */
-	pre_disable_mmu_workaround
 	msr_s	SYS_SCTLR_EL12, x1
 	mov	x2, #BOOT_CPU_FLAG_E2H
 	b	3f
 
 2:
-	pre_disable_mmu_workaround
 	msr	sctlr_el1, x1
 	mov	x2, xzr
 3:
-- 
2.44.0.683.g7961c838ac-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH 2/2] arm64/head: Disable MMU at EL2 before clearing HCR_EL2.E2H

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

Even though the boot protocol stipulates otherwise, an exception has
been made for the EFI stub, and entering the core kernel with the MMU
enabled is permitted. This allows a substantial amount of cache
maintenance to be elided, wich is significant when fast boot times are
critical (e.g., for booting micro-VMs)

Once the initial ID map has been populated, the MMU is disabled as part
of the logic sequence that puts all system registers into a known state.
Any code that needs to execute within the window where the MMU is off is
cleaned to the PoC explicitly, which includes all of HYP text when
entering at EL2.

However, the current sequence of initializing the EL2 system registers
is not safe: HCR_EL2 is set to its nVHE initial state before SCTLR_EL2
is reprogrammed, and this means that a VHE-to-nVHE switch may occur
while the MMU is enabled. This switch causes some system registers as
well as page table descriptors to be interpreted in a different way,
potentially resulting in spurious exceptions relating to MMU
translation.

So disable the MMU explicitly first when entering in EL2 with the MMU
and caches enabled.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/arm64/kernel/head.S | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index b8bbd72cb194..cb68adcabe07 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -289,6 +289,11 @@ SYM_INNER_LABEL(init_el2, SYM_L_LOCAL)
 	adr_l	x1, __hyp_text_end
 	adr_l	x2, dcache_clean_poc
 	blr	x2
+
+	mov_q	x0, INIT_SCTLR_EL2_MMU_OFF
+	pre_disable_mmu_workaround
+	msr	sctlr_el2, x0
+	isb
 0:
 	mov_q	x0, HCR_HOST_NVHE_FLAGS
 
-- 
2.44.0.683.g7961c838ac-goog


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 1/2] arm64/head: Drop unnecessary pre-disable-MMU workaround

[Marc Zyngier]

On Mon, 15 Apr 2024 08:54:14 +0100,
Ard Biesheuvel <ardb+git@google.com> wrote:
> 
> From: Ard Biesheuvel <ardb@kernel.org>
> 
> The Falkor erratum that results in the need for an ISB before clearing
> the M bit in SCTLR_ELx only applies to execution at exception level x,
> and so the workaround is not needed when disabling the EL1 MMU while
> running at EL2.
> 
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> ---
>  arch/arm64/kernel/head.S | 2 --
>  1 file changed, 2 deletions(-)
> 
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index 06234c3a15f3..b8bbd72cb194 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -323,13 +323,11 @@ SYM_INNER_LABEL(init_el2, SYM_L_LOCAL)
>  	cbz	x0, 2f
>  
>  	/* Set a sane SCTLR_EL1, the VHE way */
> -	pre_disable_mmu_workaround
>  	msr_s	SYS_SCTLR_EL12, x1
>  	mov	x2, #BOOT_CPU_FLAG_E2H
>  	b	3f
>  
>  2:
> -	pre_disable_mmu_workaround
>  	msr	sctlr_el1, x1
>  	mov	x2, xzr
>  3:

Acked-by: Marc Zyngier <maz@kernel.org>

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/2] arm64/head: Disable MMU at EL2 before clearing HCR_EL2.E2H

[Marc Zyngier]

On Mon, 15 Apr 2024 08:54:15 +0100,
Ard Biesheuvel <ardb+git@google.com> wrote:
> 
> From: Ard Biesheuvel <ardb@kernel.org>
> 
> Even though the boot protocol stipulates otherwise, an exception has
> been made for the EFI stub, and entering the core kernel with the MMU
> enabled is permitted. This allows a substantial amount of cache
> maintenance to be elided, wich is significant when fast boot times are
> critical (e.g., for booting micro-VMs)
> 
> Once the initial ID map has been populated, the MMU is disabled as part
> of the logic sequence that puts all system registers into a known state.
> Any code that needs to execute within the window where the MMU is off is
> cleaned to the PoC explicitly, which includes all of HYP text when
> entering at EL2.
> 
> However, the current sequence of initializing the EL2 system registers
> is not safe: HCR_EL2 is set to its nVHE initial state before SCTLR_EL2
> is reprogrammed, and this means that a VHE-to-nVHE switch may occur
> while the MMU is enabled. This switch causes some system registers as
> well as page table descriptors to be interpreted in a different way,
> potentially resulting in spurious exceptions relating to MMU
> translation.
> 
> So disable the MMU explicitly first when entering in EL2 with the MMU
> and caches enabled.
> 
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> ---
>  arch/arm64/kernel/head.S | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index b8bbd72cb194..cb68adcabe07 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -289,6 +289,11 @@ SYM_INNER_LABEL(init_el2, SYM_L_LOCAL)
>  	adr_l	x1, __hyp_text_end
>  	adr_l	x2, dcache_clean_poc
>  	blr	x2
> +
> +	mov_q	x0, INIT_SCTLR_EL2_MMU_OFF
> +	pre_disable_mmu_workaround
> +	msr	sctlr_el2, x0
> +	isb
>  0:
>  	mov_q	x0, HCR_HOST_NVHE_FLAGS
>  

Acked-by: Marc Zyngier <maz@kernel.org>

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 1/2] arm64/head: Drop unnecessary pre-disable-MMU workaround

[Mark Rutland]

On Mon, Apr 15, 2024 at 09:54:14AM +0200, Ard Biesheuvel wrote:
> From: Ard Biesheuvel <ardb@kernel.org>
> 
> The Falkor erratum that results in the need for an ISB before clearing
> the M bit in SCTLR_ELx only applies to execution at exception level x,
> and so the workaround is not needed when disabling the EL1 MMU while
> running at EL2.
> 
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>

Acked-by: Mark Rutland <mark.rutland@arm.com>

Mark.

> ---
>  arch/arm64/kernel/head.S | 2 --
>  1 file changed, 2 deletions(-)
> 
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index 06234c3a15f3..b8bbd72cb194 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -323,13 +323,11 @@ SYM_INNER_LABEL(init_el2, SYM_L_LOCAL)
>  	cbz	x0, 2f
>  
>  	/* Set a sane SCTLR_EL1, the VHE way */
> -	pre_disable_mmu_workaround
>  	msr_s	SYS_SCTLR_EL12, x1
>  	mov	x2, #BOOT_CPU_FLAG_E2H
>  	b	3f
>  
>  2:
> -	pre_disable_mmu_workaround
>  	msr	sctlr_el1, x1
>  	mov	x2, xzr
>  3:
> -- 
> 2.44.0.683.g7961c838ac-goog
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/2] arm64/head: Disable MMU at EL2 before clearing HCR_EL2.E2H

[Mark Rutland]

On Mon, Apr 15, 2024 at 09:54:15AM +0200, Ard Biesheuvel wrote:
> From: Ard Biesheuvel <ardb@kernel.org>
> 
> Even though the boot protocol stipulates otherwise, an exception has
> been made for the EFI stub, and entering the core kernel with the MMU
> enabled is permitted. This allows a substantial amount of cache
> maintenance to be elided, wich is significant when fast boot times are
> critical (e.g., for booting micro-VMs)
> 
> Once the initial ID map has been populated, the MMU is disabled as part
> of the logic sequence that puts all system registers into a known state.
> Any code that needs to execute within the window where the MMU is off is
> cleaned to the PoC explicitly, which includes all of HYP text when
> entering at EL2.
> 
> However, the current sequence of initializing the EL2 system registers
> is not safe: HCR_EL2 is set to its nVHE initial state before SCTLR_EL2
> is reprogrammed, and this means that a VHE-to-nVHE switch may occur
> while the MMU is enabled. This switch causes some system registers as
> well as page table descriptors to be interpreted in a different way,
> potentially resulting in spurious exceptions relating to MMU
> translation.
> 
> So disable the MMU explicitly first when entering in EL2 with the MMU
> and caches enabled.
> 
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>

Acked-by: Mark Rutland <mark.rutland@arm.com>

Mark.

> ---
>  arch/arm64/kernel/head.S | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
> index b8bbd72cb194..cb68adcabe07 100644
> --- a/arch/arm64/kernel/head.S
> +++ b/arch/arm64/kernel/head.S
> @@ -289,6 +289,11 @@ SYM_INNER_LABEL(init_el2, SYM_L_LOCAL)
>  	adr_l	x1, __hyp_text_end
>  	adr_l	x2, dcache_clean_poc
>  	blr	x2
> +
> +	mov_q	x0, INIT_SCTLR_EL2_MMU_OFF
> +	pre_disable_mmu_workaround
> +	msr	sctlr_el2, x0
> +	isb
>  0:
>  	mov_q	x0, HCR_HOST_NVHE_FLAGS
>  
> -- 
> 2.44.0.683.g7961c838ac-goog
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 0/2] arm64 head.S fixes

[Catalin Marinas]

On Mon, 15 Apr 2024 09:54:13 +0200, Ard Biesheuvel wrote:
> Ensure the MMU is off before changing HCR_EL2.E2H when entering at EL2.
> Drop MMU disable workarounds where they are not needed.
> 
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Mark Rutland <mark.rutland@arm.com>
> 
> [...]

Applied to arm64 (for-next/fixes), thanks!

[1/2] arm64/head: Drop unnecessary pre-disable-MMU workaround
      https://git.kernel.org/arm64/c/2b504e162037
[2/2] arm64/head: Disable MMU at EL2 before clearing HCR_EL2.E2H
      https://git.kernel.org/arm64/c/34e526cb7d46

I added a Fixes tag and cc stable on the second patch, it goes all the
way to 6.3 (this should actually be 6.6 LTS).

-- 
Catalin


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel