[PATCH v2 0/2] x86: Fix kexec 5-level to 4-level paging transition

[PATCH v2 0/2] x86: Fix kexec 5-level to 4-level paging transition

[Usama Arif]

This series addresses critical bugs in the kexec path when transitioning
from a kernel using 5-level page tables to one using 4-level page tables.

The root cause is improper handling of PGD entry value during the page level
transition. Specifically PGD entry value is masked with PAGE_MASK instead of
PTE_PFN_MASK, failing to account for high-order software bits like
_PAGE_BIT_NOPTISHADOW (bit 58).

When bit 58 (_PAGE_BIT_NOPTISHADOW) is set in the source kernel, the target
4-level kernel doesn't recognize it and fails to mask it properly, leading
to kexec failure.

Patch 1: Fixes the x86 boot compressed code path by replacing direct CR3
dereferencing with read_cr3_pa() and using PTE_PFN_MASK instead
of PAGE_MASK.

Patch 2: Applies the same fix to the EFI stub code path. (Done in a
separate patch as Fixes tag is different).


Co-developed-by: Kiryl Shutsemau <kas@kernel.org>
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Signed-off-by: Usama Arif <usamaarif642@gmail.com>
Reported-by: Michael van der Westhuizen <rmikey@meta.com>
Reported-by: Tobias Fleig <tfleig@meta.com>

The patches are based on aaa9c3550b60d6259d6ea8b1175ade8d1242444e (next-20251022)

v1 -> v2:
- Remove patch 3 from v1 to fix kexec for source kernel in 5-level to 4-level
  transition where the 4 level kernel doesnt have patch 1 and 2 (Dave Hansen)
- Add include for asm/pgtable.h to fix build for x86_64-allnoconfig (kernel test bot)
- Use native_read_cr3_pa and for both paths (Ard Biesheuvel)
 
Usama Arif (2):
  x86/boot: Fix page table access in 5-level to 4-level paging
    transition
  efi/libstub: Fix page table access in 5-level to 4-level paging
    transition

 arch/x86/boot/compressed/pgtable_64.c   | 11 +++++++----
 drivers/firmware/efi/libstub/x86-5lvl.c |  4 ++--
 2 files changed, 9 insertions(+), 6 deletions(-)

-- 
2.47.3

[PATCH v2 1/2] x86/boot: Fix page table access in 5-level to 4-level paging transition

[Usama Arif]

When transitioning from 5-level to 4-level paging, the existing code
incorrectly accesses page table entries by directly dereferencing CR3
and applying PAGE_MASK. This approach has several issues:

- __native_read_cr3() returns the raw CR3 register value, which on
  x86_64 includes not just the physical address but also flags. Bits
  above the physical address width of the system i.e. above
  __PHYSICAL_MASK_SHIFT) are also not masked.
- The PGD entry is masked by PAGE_SIZE which doesn't take into account
  the higher bits such as _PAGE_BIT_NOPTISHADOW.

Replace this with proper accessor functions:
- native_read_cr3_pa(): Uses CR3_ADDR_MASK properly clearing SME encryption
  bit and extracting only the physical address portion.
- mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for
  flags above physical address (_PAGE_BIT_NOPTISHADOW in particular).

Fixes: e9d0e6330eb8 ("x86/boot/compressed/64: Prepare new top-level page table for trampoline")
Co-developed-by: Kiryl Shutsemau <kas@kernel.org>
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Signed-off-by: Usama Arif <usamaarif642@gmail.com>
Reported-by: Michael van der Westhuizen <rmikey@meta.com>
Reported-by: Tobias Fleig <tfleig@meta.com>
---
 arch/x86/boot/compressed/pgtable_64.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c
index bdd26050dff77..f812b81a538c2 100644
--- a/arch/x86/boot/compressed/pgtable_64.c
+++ b/arch/x86/boot/compressed/pgtable_64.c
@@ -3,6 +3,7 @@
 #include <asm/bootparam.h>
 #include <asm/bootparam_utils.h>
 #include <asm/e820/types.h>
+#include <asm/pgtable.h>
 #include <asm/processor.h>
 #include "../string.h"
 #include "efi.h"
@@ -168,9 +169,10 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable)
 		 * For 4- to 5-level paging transition, set up current CR3 as
 		 * the first and the only entry in a new top-level page table.
 		 */
-		*trampoline_32bit = __native_read_cr3() | _PAGE_TABLE_NOENC;
+		*trampoline_32bit = native_read_cr3_pa() | _PAGE_TABLE_NOENC;
 	} else {
-		unsigned long src;
+		u64 *new_cr3;
+		pgd_t *pgdp;
 
 		/*
 		 * For 5- to 4-level paging transition, copy page table pointed
@@ -180,8 +182,9 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable)
 		 * We cannot just point to the page table from trampoline as it
 		 * may be above 4G.
 		 */
-		src = *(unsigned long *)__native_read_cr3() & PAGE_MASK;
-		memcpy(trampoline_32bit, (void *)src, PAGE_SIZE);
+		pgdp = (pgd_t *)native_read_cr3_pa();
+		new_cr3 = (u64 *)(pgd_val(pgdp[0]) & PTE_PFN_MASK);
+		memcpy(trampoline_32bit, new_cr3, PAGE_SIZE);
 	}
 
 	toggle_la57(trampoline_32bit);
-- 
2.47.3

Re: [PATCH v2 1/2] x86/boot: Fix page table access in 5-level to 4-level paging transition

[Dave Hansen]

On 10/28/25 03:55, Usama Arif wrote:
> - native_read_cr3_pa(): Uses CR3_ADDR_MASK properly clearing SME encryption
>   bit and extracting only the physical address portion.

I guess we can apply these as-is. They do fix a bug.

But I find these descriptions a bit unsatisfying. CR3_ADDR_MASK happens
to work here on 64-bit. Interestingly enough, it wouldn't have been as
good of a fix on PAE paging because it ignores those upper bits instead
of reserving them.

But CR3_ADDR_MASK doesn't "extract... only the physical address
portion". It also extracts reserved bits.

It also doesn't mention the LAM bits. It's not just SME.

This would be better:

 - native_read_cr3_pa(): Uses CR3_ADDR_MASK to additionally mask
   metadata out of CR3 (like SME or LAM bits). All remaining bits are
   real address bits or reserved and must be 0.

> - mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for
>   flags above physical address (_PAGE_BIT_NOPTISHADOW in particular).

This also isn't _quite_ right. The "flags above physical" address are
dynamic. They move because the max physical address (MAXPHYADDR) is
enumerated and changes from CPU to CPU.

It's OK in this case because moving MAXPHYADDR down just changes bits
from address bits to reserved (must be 0).

In a perfect world, we would construct a kexec CR3 with the dynamic
MAXPHYADDR (plus masking out the lower 12 bits). That would be obviously
correct for *all* 32-bit and 64-bit cases and wouldn't even rely on
knowing where the boundary is between ignored and reserved. The approach
in these patches is a fine improvement We don't need to be perfect.

Ideally this second bullet would be:

 - mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for
   flags above bit 51 (_PAGE_BIT_NOPTISHADOW in particular). Bits below
   51, but above the max physical address are reserved and must be 0.

But it's fine-ish as-is.

Acked-by: Dave Hansen <dave.hansen@linux.intel.com>

[PATCH v2 2/2] efi/libstub: Fix page table access in 5-level to 4-level paging transition

[Usama Arif]

When transitioning from 5-level to 4-level paging, the existing code
incorrectly accesses page table entries by directly dereferencing CR3
and applying PAGE_MASK. This approach has several issues:

- __native_read_cr3() returns the raw CR3 register value, which on
  x86_64 includes not just the physical address but also flags Bits
  above the physical address width of the system (i.e. above
  __PHYSICAL_MASK_SHIFT) are also not masked.
- The pgd value is masked by PAGE_SIZE which doesn't take into account
  the higher bits such as _PAGE_BIT_NOPTISHADOW.

Replace this with proper accessor functions:
- native_read_cr3_pa(): Uses CR3_ADDR_MASK properly clearing SME encryption
  bit and extracting only the physical address portion.
- mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for
  flags above physical address (_PAGE_BIT_NOPTISHADOW in particular).

Fixes: cb1c9e02b0c1 ("x86/efistub: Perform 4/5 level paging switch from the stub")
Co-developed-by: Kiryl Shutsemau <kas@kernel.org>
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Signed-off-by: Usama Arif <usamaarif642@gmail.com>
Reported-by: Michael van der Westhuizen <rmikey@meta.com>
Reported-by: Tobias Fleig <tfleig@meta.com>
---
 drivers/firmware/efi/libstub/x86-5lvl.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/firmware/efi/libstub/x86-5lvl.c b/drivers/firmware/efi/libstub/x86-5lvl.c
index f1c5fb45d5f7c..36b4a611f6adf 100644
--- a/drivers/firmware/efi/libstub/x86-5lvl.c
+++ b/drivers/firmware/efi/libstub/x86-5lvl.c
@@ -66,7 +66,7 @@ void efi_5level_switch(void)
 	bool have_la57 = native_read_cr4() & X86_CR4_LA57;
 	bool need_toggle = want_la57 ^ have_la57;
 	u64 *pgt = (void *)la57_toggle + PAGE_SIZE;
-	u64 *cr3 = (u64 *)__native_read_cr3();
+	pgd_t *cr3 = (pgd_t *)native_read_cr3_pa();
 	u64 *new_cr3;
 
 	if (!la57_toggle || !need_toggle)
@@ -82,7 +82,7 @@ void efi_5level_switch(void)
 		new_cr3[0] = (u64)cr3 | _PAGE_TABLE_NOENC;
 	} else {
 		/* take the new root table pointer from the current entry #0 */
-		new_cr3 = (u64 *)(cr3[0] & PAGE_MASK);
+		new_cr3 = (u64 *)(pgd_val(cr3[0]) & PTE_PFN_MASK);
 
 		/* copy the new root table if it is not 32-bit addressable */
 		if ((u64)new_cr3 > U32_MAX)
-- 
2.47.3

Re: [PATCH v2 2/2] efi/libstub: Fix page table access in 5-level to 4-level paging transition

[Borislav Petkov]

On Tue, Oct 28, 2025 at 10:55:57AM +0000, Usama Arif wrote:
> When transitioning from 5-level to 4-level paging, the existing code
> incorrectly accesses page table entries by directly dereferencing CR3
> and applying PAGE_MASK. This approach has several issues:
> 
> - __native_read_cr3() returns the raw CR3 register value, which on
>   x86_64 includes not just the physical address but also flags Bits
>   above the physical address width of the system (i.e. above
>   __PHYSICAL_MASK_SHIFT) are also not masked.
> - The pgd value is masked by PAGE_SIZE which doesn't take into account
>   the higher bits such as _PAGE_BIT_NOPTISHADOW.
> 
> Replace this with proper accessor functions:
> - native_read_cr3_pa(): Uses CR3_ADDR_MASK properly clearing SME encryption
>   bit and extracting only the physical address portion.
> - mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for
>   flags above physical address (_PAGE_BIT_NOPTISHADOW in particular).
> 
> Fixes: cb1c9e02b0c1 ("x86/efistub: Perform 4/5 level paging switch from the stub")
> Co-developed-by: Kiryl Shutsemau <kas@kernel.org>
> Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
> Signed-off-by: Usama Arif <usamaarif642@gmail.com>
> Reported-by: Michael van der Westhuizen <rmikey@meta.com>
> Reported-by: Tobias Fleig <tfleig@meta.com>
> ---
>  drivers/firmware/efi/libstub/x86-5lvl.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

allmodconfig build:

ld: error: unplaced orphan section `__bug_table' from `drivers/firmware/efi/libstub/x86-5lvl.stub.o'
ld: error: unplaced orphan section `.altinstructions' from `drivers/firmware/efi/libstub/x86-5lvl.stub.o'
ld: error: unplaced orphan section `.altinstr_replacement' from `drivers/firmware/efi/libstub/x86-5lvl.stub.o'
ld: Unexpected GOT/PLT entries detected!
ld: Unexpected run-time procedure linkages detected!
ld: Unexpected run-time relocations (.rela) detected!
ld: drivers/firmware/efi/libstub/x86-5lvl.stub.o: in function `efi_5level_switch':
x86-5lvl.c:(.text+0x13e): undefined reference to `pv_ops'
ld: x86-5lvl.c:(.text+0x14d): undefined reference to `pv_ops'
ld: drivers/firmware/efi/libstub/x86-5lvl.stub.o:(.altinstr_replacement+0x1): undefined reference to `BUG_func'
ld: arch/x86/boot/compressed/vmlinux: hidden symbol `pv_ops' isn't defined
ld: final link failed: bad value
make[3]: *** [arch/x86/boot/compressed/Makefile:116: arch/x86/boot/compressed/vmlinux] Error 1
make[2]: *** [arch/x86/boot/Makefile:96: arch/x86/boot/compressed/vmlinux] Error 2
make[1]: *** [arch/x86/Makefile:308: bzImage] Error 2
make: *** [Makefile:248: __sub-make] Error 2

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH v2 2/2] efi/libstub: Fix page table access in 5-level to 4-level paging transition

[Ard Biesheuvel]

On Fri, 31 Oct 2025 at 15:40, Borislav Petkov <bp@alien8.de> wrote:
>
> On Tue, Oct 28, 2025 at 10:55:57AM +0000, Usama Arif wrote:
> > When transitioning from 5-level to 4-level paging, the existing code
> > incorrectly accesses page table entries by directly dereferencing CR3
> > and applying PAGE_MASK. This approach has several issues:
> >
> > - __native_read_cr3() returns the raw CR3 register value, which on
> >   x86_64 includes not just the physical address but also flags Bits
> >   above the physical address width of the system (i.e. above
> >   __PHYSICAL_MASK_SHIFT) are also not masked.
> > - The pgd value is masked by PAGE_SIZE which doesn't take into account
> >   the higher bits such as _PAGE_BIT_NOPTISHADOW.
> >
> > Replace this with proper accessor functions:
> > - native_read_cr3_pa(): Uses CR3_ADDR_MASK properly clearing SME encryption
> >   bit and extracting only the physical address portion.
> > - mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for
> >   flags above physical address (_PAGE_BIT_NOPTISHADOW in particular).
> >
> > Fixes: cb1c9e02b0c1 ("x86/efistub: Perform 4/5 level paging switch from the stub")
> > Co-developed-by: Kiryl Shutsemau <kas@kernel.org>
> > Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
> > Signed-off-by: Usama Arif <usamaarif642@gmail.com>
> > Reported-by: Michael van der Westhuizen <rmikey@meta.com>
> > Reported-by: Tobias Fleig <tfleig@meta.com>
> > ---
> >  drivers/firmware/efi/libstub/x86-5lvl.c | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
>
> allmodconfig build:
>
> ld: error: unplaced orphan section `__bug_table' from `drivers/firmware/efi/libstub/x86-5lvl.stub.o'
> ld: error: unplaced orphan section `.altinstructions' from `drivers/firmware/efi/libstub/x86-5lvl.stub.o'
> ld: error: unplaced orphan section `.altinstr_replacement' from `drivers/firmware/efi/libstub/x86-5lvl.stub.o'
> ld: Unexpected GOT/PLT entries detected!
> ld: Unexpected run-time procedure linkages detected!
> ld: Unexpected run-time relocations (.rela) detected!
> ld: drivers/firmware/efi/libstub/x86-5lvl.stub.o: in function `efi_5level_switch':
> x86-5lvl.c:(.text+0x13e): undefined reference to `pv_ops'
> ld: x86-5lvl.c:(.text+0x14d): undefined reference to `pv_ops'
> ld: drivers/firmware/efi/libstub/x86-5lvl.stub.o:(.altinstr_replacement+0x1): undefined reference to `BUG_func'
> ld: arch/x86/boot/compressed/vmlinux: hidden symbol `pv_ops' isn't defined
> ld: final link failed: bad value
> make[3]: *** [arch/x86/boot/compressed/Makefile:116: arch/x86/boot/compressed/vmlinux] Error 1
> make[2]: *** [arch/x86/boot/Makefile:96: arch/x86/boot/compressed/vmlinux] Error 2
> make[1]: *** [arch/x86/Makefile:308: bzImage] Error 2
> make: *** [Makefile:248: __sub-make] Error 2
>

This code should be using native_pgd_val() not pgd_val().

Re: [PATCH v2 2/2] efi/libstub: Fix page table access in 5-level to 4-level paging transition

[Borislav Petkov]

On Fri, Oct 31, 2025 at 03:43:25PM +0100, Ard Biesheuvel wrote:
> This code should be using native_pgd_val() not pgd_val().

Seems to fix it, thanks.

I'll let Usama do more testing along with the usual build smoke tests - all
permutations of the below:

ARCHES=('x86_64' 'i386')
SMOKE_CONFIGS=("allnoconfig" "defconfig" "allmodconfig" "allyesconfig")

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH v2 0/2] x86: Fix kexec 5-level to 4-level paging transition

[Ard Biesheuvel]

On Tue, 28 Oct 2025 at 11:57, Usama Arif <usamaarif642@gmail.com> wrote:
>
> This series addresses critical bugs in the kexec path when transitioning
> from a kernel using 5-level page tables to one using 4-level page tables.
>
> The root cause is improper handling of PGD entry value during the page level
> transition. Specifically PGD entry value is masked with PAGE_MASK instead of
> PTE_PFN_MASK, failing to account for high-order software bits like
> _PAGE_BIT_NOPTISHADOW (bit 58).
>
> When bit 58 (_PAGE_BIT_NOPTISHADOW) is set in the source kernel, the target
> 4-level kernel doesn't recognize it and fails to mask it properly, leading
> to kexec failure.
>
> Patch 1: Fixes the x86 boot compressed code path by replacing direct CR3
> dereferencing with read_cr3_pa() and using PTE_PFN_MASK instead
> of PAGE_MASK.
>
> Patch 2: Applies the same fix to the EFI stub code path. (Done in a
> separate patch as Fixes tag is different).
>
>
> Co-developed-by: Kiryl Shutsemau <kas@kernel.org>
> Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
> Signed-off-by: Usama Arif <usamaarif642@gmail.com>
> Reported-by: Michael van der Westhuizen <rmikey@meta.com>
> Reported-by: Tobias Fleig <tfleig@meta.com>
>
> The patches are based on aaa9c3550b60d6259d6ea8b1175ade8d1242444e (next-20251022)
>
> v1 -> v2:
> - Remove patch 3 from v1 to fix kexec for source kernel in 5-level to 4-level
>   transition where the 4 level kernel doesnt have patch 1 and 2 (Dave Hansen)
> - Add include for asm/pgtable.h to fix build for x86_64-allnoconfig (kernel test bot)
> - Use native_read_cr3_pa and for both paths (Ard Biesheuvel)
>
> Usama Arif (2):
>   x86/boot: Fix page table access in 5-level to 4-level paging
>     transition
>   efi/libstub: Fix page table access in 5-level to 4-level paging
>     transition
>

Reviewed-by: Ard Biesheuvel <ardb@kernel.org>

Re: [PATCH v2 0/2] x86: Fix kexec 5-level to 4-level paging transition

[Borislav Petkov]

On Tue, Oct 28, 2025 at 10:55:55AM +0000, Usama Arif wrote:
> This series addresses critical bugs in the kexec path when transitioning
> from a kernel using 5-level page tables to one using 4-level page tables.

Out of curiosity: what is the real-life use case for this?

Judging by the Reported-by's I guess Meta is doing some kexec-ing into default
kernels which are 4-level so that they can work on any machine ...

Close?

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [PATCH v2 0/2] x86: Fix kexec 5-level to 4-level paging transition

[Kiryl Shutsemau]

On Wed, Oct 29, 2025 at 09:48:14PM +0100, Borislav Petkov wrote:
> On Tue, Oct 28, 2025 at 10:55:55AM +0000, Usama Arif wrote:
> > This series addresses critical bugs in the kexec path when transitioning
> > from a kernel using 5-level page tables to one using 4-level page tables.
> 
> Out of curiosity: what is the real-life use case for this?
> 
> Judging by the Reported-by's I guess Meta is doing some kexec-ing into default
> kernels which are 4-level so that they can work on any machine ...
> 
> Close?

Older kernels in our fleet run with 5-level paging disabled. The newer
one enables it. Machines need to switch between kernel version from time
to time for different reasons. Switching from the newer kernel to an
older one triggered the issue.

-- 
  Kiryl Shutsemau / Kirill A. Shutemov

Re: [PATCH v2 0/2] x86: Fix kexec 5-level to 4-level paging transition

[Borislav Petkov]

On Thu, Oct 30, 2025 at 10:23:11AM +0000, Kiryl Shutsemau wrote:
> Older kernels in our fleet run with 5-level paging disabled. The newer
> one enables it. Machines need to switch between kernel version from time
> to time for different reasons. Switching from the newer kernel to an
> older one triggered the issue.

Thx, makes sense.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette