[PATCH 0/6] x86/efi/mixed: Decouple from legacy decompressor

[PATCH 0/6] x86/efi/mixed: Decouple from legacy decompressor

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

Since commit

  a1b87d54f4e4 ("x86/efistub: Avoid legacy decompressor when doing EFI boot")

booting via the EFI stub no longer relies on the legacy decompressor,
and instead, the kernel proper is decompressed by code executing in the
context of the EFI boot services, and subsequently invoked directly.

The only remaining dependency is the EFI mixed mode startup code, which
makes a detour via the legacy decompressor's 32-bit entrypoint, in order
to obtain a 1:1 mapping of memory, which is a prerequisite for 64-bit
execution on x86.

This detour requires some fiddly setup on the part of the mixed mode
startup code, which has to stash the firmware stack pointer and boot
arguments in memory, and create a fake struct boot_params to trick the
code in startup_32 to behave as intended.

This dependency also impedes reuse of the EFI stub code in other
contexts, such as generic EFI zboot, which will reuse the EFI stub but
not the legacy decompressor.

So remove this dependency, by replacing this detour with a minimal
reimplementation of the 1:1 mapping code. With some further cleanup
applied on top, the line count barely changes, but the resulting code
can operate independently from the legacy decompressor, and is therefore
moved out of arch/x86/boot/compressed and into the EFI libstub/
directory.

Ard Biesheuvel (6):
  x86/efi/mixed: Check CPU compatibility without relying on verify_cpu()
  x86/efi/mixed: Remove dependency on legacy startup_32 code
  x86/efi/mixed: Don't bother preserving 64-bit mode segment selectors
  x86/efi/mixed: Simplify and document thunking logic
  x86/efi/mixed: Reduce padding by moving some code around
  x86/efi/mixed: Move mixed mode startup code into libstub

 arch/x86/boot/compressed/Makefile                                                |   1 -
 arch/x86/boot/compressed/head_64.S                                               |   7 -
 drivers/firmware/efi/libstub/Makefile                                            |   3 +
 arch/x86/boot/compressed/efi_mixed.S => drivers/firmware/efi/libstub/x86-mixed.S | 320 ++++++++++----------
 4 files changed, 169 insertions(+), 162 deletions(-)
 rename arch/x86/boot/compressed/efi_mixed.S => drivers/firmware/efi/libstub/x86-mixed.S (54%)

-- 
2.47.1.613.gc27f4b7a9f-goog

[PATCH 1/6] x86/efi/mixed: Check CPU compatibility without relying on verify_cpu()

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

In order for the EFI mixed mode startup code to be reusable in a context
where the legacy decompressor is not used, replace the call to
verify_cpu() [which performs an elaborate set of checks] with a simple
check against the 'long mode' bit in the appropriate CPUID leaf.

This is reasonable, given that EFI support is implied when booting in
this manner, and so there is no need to consider very old CPUs when
performing this check.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/x86/boot/compressed/efi_mixed.S | 22 ++++++++------------
 1 file changed, 9 insertions(+), 13 deletions(-)

diff --git a/arch/x86/boot/compressed/efi_mixed.S b/arch/x86/boot/compressed/efi_mixed.S
index 876fc6d46a13..067e10eb7897 100644
--- a/arch/x86/boot/compressed/efi_mixed.S
+++ b/arch/x86/boot/compressed/efi_mixed.S
@@ -293,24 +293,20 @@ SYM_FUNC_END(efi32_entry)
  *			       efi_system_table_32_t *sys_table)
  */
 SYM_FUNC_START(efi32_pe_entry)
-	pushl	%ebp
-	movl	%esp, %ebp
 	pushl	%ebx				// save callee-save registers
-	pushl	%edi
-
-	call	verify_cpu			// check for long mode support
-	testl	%eax, %eax
-	movl	$0x80000003, %eax		// EFI_UNSUPPORTED
-	jnz	2f
 
-	movl	8(%ebp), %ecx			// image_handle
-	movl	12(%ebp), %edx			// sys_table
+	/* Check whether the CPU supports long mode */
+	movl	$0x80000001, %eax		// assume extended info support
+	cpuid
+	btl	$29, %edx			// check long mode bit
+	jnc	1f
+	leal	8(%esp), %esp			// preserve stack alignment
+	movl	(%esp), %ecx			// image_handle
+	movl	4(%esp), %edx			// sys_table
 	jmp	efi32_entry			// pass %ecx, %edx
 						// no other registers remain live
-
-2:	popl	%edi				// restore callee-save registers
+1:	movl	$0x80000003, %eax		// EFI_UNSUPPORTED
 	popl	%ebx
-	leave
 	RET
 SYM_FUNC_END(efi32_pe_entry)
 
-- 
2.47.1.613.gc27f4b7a9f-goog

[PATCH 2/6] x86/efi/mixed: Remove dependency on legacy startup_32 code

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

The EFI mixed mode startup code calls into startup_32 in the legacy
decompressor with a mocked up boot_params struct, only to get it to set
up the 1:1 mapping of the lower 4 GiB of memory and switch to a GDT that
supports 64-bit mode.

In order to be able to reuse the EFI mixed mode startup code in EFI
zboot images, which do not incorporate the legacy decompressor code,
decouple it, by building the 1:1 map and installing the GDT directly.

This also removes the need to preserve and restore the stack pointer and
the arguments passed to the entrypoint, and instead, the firmware stack
can be retained, and the arguments can be loaded from it at the point
where they are needed, i.e., before calling into the EFI stub C code.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/x86/boot/compressed/efi_mixed.S | 143 ++++++++++++--------
 arch/x86/boot/compressed/head_64.S   |   7 -
 2 files changed, 86 insertions(+), 64 deletions(-)

diff --git a/arch/x86/boot/compressed/efi_mixed.S b/arch/x86/boot/compressed/efi_mixed.S
index 067e10eb7897..c010aba7a0d0 100644
--- a/arch/x86/boot/compressed/efi_mixed.S
+++ b/arch/x86/boot/compressed/efi_mixed.S
@@ -15,15 +15,18 @@
  */
 
 #include <linux/linkage.h>
-#include <asm/asm-offsets.h>
 #include <asm/msr.h>
 #include <asm/page_types.h>
+#include <asm/pgtable_types.h>
 #include <asm/processor-flags.h>
 #include <asm/segment.h>
-#include <asm/setup.h>
 
 	.code64
 	.text
+	.balign	8
+SYM_DATA_LOCAL(gdt, .quad 0x0, 0x0, 0xaf9a000000ffff)	/* __KERNEL_CS */
+	.set	gdt_size, . - gdt
+
 /*
  * When booting in 64-bit mode on 32-bit EFI firmware, startup_64_mixed_mode()
  * is the first thing that runs after switching to long mode. Depending on
@@ -35,30 +38,34 @@
  * pointer is used to disambiguate.
  *
  *                                                             +--------------+
- *  +------------------+     +------------+            +------>| efi_pe_entry |
- *  | efi32_pe_entry   |---->|            |            |       +-----------+--+
- *  +------------------+     |            |     +------+----------------+  |
- *                           | startup_32 |---->| startup_64_mixed_mode |  |
- *  +------------------+     |            |     +------+----------------+  |
- *  | efi32_stub_entry |---->|            |            |                   |
- *  +------------------+     +------------+            |                   |
+ *  +------------------+    +-------------+            +------>| efi_pe_entry |
+ *  | efi32_pe_entry   |--->|             |            |       +-----------+--+
+ *  +------------------+    |             |     +------+----------------+  |
+ *                          | efi32_entry |---->| startup_64_mixed_mode |  |
+ *  +------------------+    |             |     +------+----------------+  |
+ *  | efi32_stub_entry |--->|             |            |                   |
+ *  +------------------+    +-------------+            |                   |
  *                                                     V                   |
- *                           +------------+     +----------------+         |
- *                           | startup_64 |<----| efi_stub_entry |<--------+
- *                           +------------+     +----------------+
+ *                          +-------------+     +----------------+         |
+ *                          |   vmlinux   |<----| efi_stub_entry |<--------+
+ *                          +-------------+     +----------------+
  */
-SYM_FUNC_START(startup_64_mixed_mode)
-	lea	efi32_boot_args(%rip), %rdx
-	mov	0(%rdx), %edi
-	mov	4(%rdx), %esi
+SYM_FUNC_START_LOCAL_NOALIGN(startup_64_mixed_mode)
+	xorl	%eax, %eax
+	movl	%eax, %ds
+	movl	%eax, %es
+	movl	%eax, %ss
+	movl	%eax, %fs
+	movl	%eax, %gs
 
-	/* Switch to the firmware's stack */
-	movl	efi32_boot_sp(%rip), %esp
-	andl	$~7, %esp
+	movl	0(%rsp), %ecx		// MS calling convention
+	movl	4(%rsp), %edx
 
 #ifdef CONFIG_EFI_HANDOVER_PROTOCOL
-	mov	8(%rdx), %edx		// saved bootparams pointer
-	test	%edx, %edx
+	test	%edi, %edi		// struct boot_params provided?
+	movl	%edx, %esi		// SysV calling convention
+	cmovnzl	%edi, %edx
+	movl	%ecx, %edi
 	jnz	efi_stub_entry
 #endif
 	/*
@@ -69,8 +76,6 @@ SYM_FUNC_START(startup_64_mixed_mode)
 	 * the correct stack alignment for entry.
 	 */
 	sub	$40, %rsp
-	mov	%rdi, %rcx		// MS calling convention
-	mov	%rsi, %rdx
 	jmp	efi_pe_entry
 SYM_FUNC_END(startup_64_mixed_mode)
 
@@ -151,7 +156,6 @@ SYM_FUNC_END(__efi64_thunk)
 SYM_FUNC_START(efi32_stub_entry)
 	call	1f
 1:	popl	%ecx
-	leal	(efi32_boot_args - 1b)(%ecx), %ebx
 
 	/* Clear BSS */
 	xorl	%eax, %eax
@@ -163,10 +167,7 @@ SYM_FUNC_START(efi32_stub_entry)
 	rep	stosl
 
 	add	$0x4, %esp		/* Discard return address */
-	popl	%ecx
-	popl	%edx
-	popl	%esi
-	movl	%esi, 8(%ebx)
+	movl	8(%esp), %edi		/* struct boot_params pointer */
 	jmp	efi32_entry
 SYM_FUNC_END(efi32_stub_entry)
 #endif
@@ -241,8 +242,9 @@ SYM_FUNC_END(efi_enter32)
 /*
  * This is the common EFI stub entry point for mixed mode.
  *
- * Arguments:	%ecx	image handle
- * 		%edx	EFI system table pointer
+ * Arguments:	0(%esp)	image handle
+ * 		4(%esp)	EFI system table pointer
+ *		%edi	struct boot_params pointer (or NULL)
  *
  * Since this is the point of no return for ordinary execution, no registers
  * are considered live except for the function parameters. [Note that the EFI
@@ -261,31 +263,58 @@ SYM_FUNC_START_LOCAL(efi32_entry)
 	/* Store firmware IDT descriptor */
 	sidtl	(efi32_boot_idt - 1b)(%ebx)
 
-	/* Store firmware stack pointer */
-	movl	%esp, (efi32_boot_sp - 1b)(%ebx)
-
-	/* Store boot arguments */
-	leal	(efi32_boot_args - 1b)(%ebx), %ebx
-	movl	%ecx, 0(%ebx)
-	movl	%edx, 4(%ebx)
-	movb	$0x0, 12(%ebx)          // efi_is64
-
-	/*
-	 * Allocate some memory for a temporary struct boot_params, which only
-	 * needs the minimal pieces that startup_32() relies on.
-	 */
-	subl	$PARAM_SIZE, %esp
-	movl	%esp, %esi
-	movl	$PAGE_SIZE, BP_kernel_alignment(%esi)
-	movl	$_end - 1b, BP_init_size(%esi)
-	subl	$startup_32 - 1b, BP_init_size(%esi)
+	/* Record mixed mode entry */
+	movb	$0x0, (efi_is64 - 1b)(%ebx)
 
 	/* Disable paging */
 	movl	%cr0, %eax
 	btrl	$X86_CR0_PG_BIT, %eax
 	movl	%eax, %cr0
 
-	jmp	startup_32
+	/* Set up 1:1 mapping */
+	leal	(pte - 1b)(%ebx), %eax
+	movl	$_PAGE_PRESENT | _PAGE_RW | _PAGE_PSE, %ecx
+	leal	(_PAGE_PRESENT | _PAGE_RW)(%eax), %edx
+2:	movl	%ecx, (%eax)
+	addl	$8, %eax
+	addl	$PMD_SIZE, %ecx
+	jnc	2b
+
+	xor	%ecx, %ecx
+	movl	$PAGE_SIZE, %esi
+3:	movl	%edx, (%eax,%ecx,8)
+	addl	%esi, %edx
+	inc	%ecx
+	cmp	$4, %ecx
+	jl	3b
+
+	addl	%esi, %eax
+	movl	%edx, (%eax)
+	movl	%eax, %cr3
+
+	movl	%cr4, %eax
+	orl	$X86_CR4_PAE, %eax
+	movl	%eax, %cr4
+
+	movl	$MSR_EFER, %ecx
+	rdmsr
+	btsl	$_EFER_LME, %eax
+	wrmsr
+
+	leal	(gdt - 1b)(%ebx), %ecx
+	pushl	%ecx
+	pushw	$gdt_size - 1
+	lgdtl	(%esp)
+	lea	6(%esp), %esp
+
+	/* Enable paging and jump to long mode */
+	leal	(startup_64_mixed_mode - 1b)(%ebx), %ecx
+	pushl	$__KERNEL_CS
+	pushl	%ecx
+	movl	%cr0, %eax
+	btsl	$X86_CR0_PG_BIT, %eax
+	movl	%eax, %cr0
+	lret
 SYM_FUNC_END(efi32_entry)
 
 /*
@@ -301,10 +330,8 @@ SYM_FUNC_START(efi32_pe_entry)
 	btl	$29, %edx			// check long mode bit
 	jnc	1f
 	leal	8(%esp), %esp			// preserve stack alignment
-	movl	(%esp), %ecx			// image_handle
-	movl	4(%esp), %edx			// sys_table
-	jmp	efi32_entry			// pass %ecx, %edx
-						// no other registers remain live
+	xor	%edi, %edi			// no struct boot_params in EDI
+	jmp	efi32_entry			// only ESP and EDI remain live
 1:	movl	$0x80000003, %eax		// EFI_UNSUPPORTED
 	popl	%ebx
 	RET
@@ -318,8 +345,10 @@ SYM_FUNC_START_NOALIGN(efi64_stub_entry)
 SYM_FUNC_END(efi64_stub_entry)
 #endif
 
-	.data
-	.balign	8
+	.bss
+	.balign PAGE_SIZE
+SYM_DATA_LOCAL(pte, .fill 6 * PAGE_SIZE, 1, 0)
+
 SYM_DATA_START_LOCAL(efi32_boot_gdt)
 	.word	0
 	.quad	0
@@ -330,8 +359,8 @@ SYM_DATA_START_LOCAL(efi32_boot_idt)
 	.quad	0
 SYM_DATA_END(efi32_boot_idt)
 
+	.data
+	.balign	4
 SYM_DATA_LOCAL(efi32_boot_cs, .word 0)
 SYM_DATA_LOCAL(efi32_boot_ds, .word 0)
-SYM_DATA_LOCAL(efi32_boot_sp, .long 0)
-SYM_DATA_LOCAL(efi32_boot_args, .long 0, 0, 0)
 SYM_DATA(efi_is64, .byte 1)
diff --git a/arch/x86/boot/compressed/head_64.S b/arch/x86/boot/compressed/head_64.S
index 1dcb794c5479..5db6495a3bb9 100644
--- a/arch/x86/boot/compressed/head_64.S
+++ b/arch/x86/boot/compressed/head_64.S
@@ -263,13 +263,6 @@ SYM_FUNC_START(startup_32)
 	 * used to perform that far jump.
 	 */
 	leal	rva(startup_64)(%ebp), %eax
-#ifdef CONFIG_EFI_MIXED
-	cmpb	$1, rva(efi_is64)(%ebp)
-	je	1f
-	leal	rva(startup_64_mixed_mode)(%ebp), %eax
-1:
-#endif
-
 	pushl	$__KERNEL_CS
 	pushl	%eax
 
-- 
2.47.1.613.gc27f4b7a9f-goog

[PATCH 3/6] x86/efi/mixed: Don't bother preserving 64-bit mode segment selectors

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

Commit

  e10848a26a96 ("x86/efi: Preserve segment registers in mixed mode")

added code to preserve and restore the caller's data segment selector
values when temporarily switching back to the firmware's GDT and segment
layout to perform an EFI runtime call.

At that point, this code was shared between the boot-time and runtime
mixed mode implementations, and the issue this commit aimed to address
only affected the latter.

Since commit

  96738c69a7fc ("x86/efi: Avoid triple faults during EFI mixed mode calls")

the runtime version of the mixed mode handling is entirely separate, and
relies on IA-32e compatibility mode to call into the 32-bit firmware
while retaining the OS's GDT/IDT and segment layout.

At boot-time, preserving/restoring the values is pointless, and the data
segment selectors can now simply be zeroed after (re)entering 64-bit
mode.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/x86/boot/compressed/efi_mixed.S | 24 ++++++--------------
 1 file changed, 7 insertions(+), 17 deletions(-)

diff --git a/arch/x86/boot/compressed/efi_mixed.S b/arch/x86/boot/compressed/efi_mixed.S
index c010aba7a0d0..066f4365af4f 100644
--- a/arch/x86/boot/compressed/efi_mixed.S
+++ b/arch/x86/boot/compressed/efi_mixed.S
@@ -83,17 +83,10 @@ SYM_FUNC_START(__efi64_thunk)
 	push	%rbp
 	push	%rbx
 
-	movl	%ds, %eax
-	push	%rax
-	movl	%es, %eax
-	push	%rax
-	movl	%ss, %eax
-	push	%rax
-
 	/* Copy args passed on stack */
-	movq	0x30(%rsp), %rbp
-	movq	0x38(%rsp), %rbx
-	movq	0x40(%rsp), %rax
+	movq	0x18(%rsp), %rbp
+	movq	0x20(%rsp), %rbx
+	movq	0x28(%rsp), %rax
 
 	/*
 	 * Convert x86-64 ABI params to i386 ABI
@@ -135,14 +128,11 @@ SYM_FUNC_START(__efi64_thunk)
 1:	addq	$64, %rsp
 	movq	%rdi, %rax
 
-	pop	%rbx
-	movl	%ebx, %ss
-	pop	%rbx
-	movl	%ebx, %es
-	pop	%rbx
-	movl	%ebx, %ds
-	/* Clear out 32-bit selector from FS and GS */
+	/* Clear out 32-bit segment selectors */
 	xorl	%ebx, %ebx
+	movl	%ebx, %ds
+	movl	%ebx, %es
+	movl	%ebx, %ss
 	movl	%ebx, %fs
 	movl	%ebx, %gs
 
-- 
2.47.1.613.gc27f4b7a9f-goog

[PATCH 4/6] x86/efi/mixed: Simplify and document thunking logic

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

The current boot-time version of the thunk into 32-bit mode, to invoke
EFI services and protocols using the 32-bit calling convention, is a
jumble of argument marshalling code and GDT/IDT and segment register
handling, with an undocumented de facto calling convention that passes
the data segment descriptor, the return address and the GDT/IDT base
pointer for the return in registers EDX, EBP and EBX respectively.

Let's clean this up, and replace this with documented logic that
separates the handling of the segment registers into the 64-bit caller,
and the marshalling of the arguments into the 32-bit callee. Also,
replace the open coded far call with an actual one, removing the need to
pass the return address via a register.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/x86/boot/compressed/efi_mixed.S | 92 ++++++++++----------
 1 file changed, 44 insertions(+), 48 deletions(-)

diff --git a/arch/x86/boot/compressed/efi_mixed.S b/arch/x86/boot/compressed/efi_mixed.S
index 066f4365af4f..a44e522bbbed 100644
--- a/arch/x86/boot/compressed/efi_mixed.S
+++ b/arch/x86/boot/compressed/efi_mixed.S
@@ -80,52 +80,36 @@ SYM_FUNC_START_LOCAL_NOALIGN(startup_64_mixed_mode)
 SYM_FUNC_END(startup_64_mixed_mode)
 
 SYM_FUNC_START(__efi64_thunk)
-	push	%rbp
 	push	%rbx
 
-	/* Copy args passed on stack */
-	movq	0x18(%rsp), %rbp
-	movq	0x20(%rsp), %rbx
-	movq	0x28(%rsp), %rax
-
-	/*
-	 * Convert x86-64 ABI params to i386 ABI
-	 */
-	subq	$64, %rsp
-	movl	%esi, 0x0(%rsp)
-	movl	%edx, 0x4(%rsp)
-	movl	%ecx, 0x8(%rsp)
-	movl	%r8d, 0xc(%rsp)
-	movl	%r9d, 0x10(%rsp)
-	movl	%ebp, 0x14(%rsp)
-	movl	%ebx, 0x18(%rsp)
-	movl	%eax, 0x1c(%rsp)
-
-	leaq	0x20(%rsp), %rbx
-	sgdt	(%rbx)
-	sidt	16(%rbx)
-
-	leaq	1f(%rip), %rbp
+	/* Store live GDT and IDT descriptors */
+	subq	$16, %rsp
+	sgdt	(%rsp)
+	sidt	6(%rsp)
 
 	/*
 	 * Switch to IDT and GDT with 32-bit segments. These are the firmware
 	 * GDT and IDT that were installed when the kernel started executing.
 	 * The pointers were saved by the efi32_entry() routine below.
-	 *
-	 * Pass the saved DS selector to the 32-bit code, and use far return to
-	 * restore the saved CS selector.
 	 */
 	lidt	efi32_boot_idt(%rip)
 	lgdt	efi32_boot_gdt(%rip)
 
-	movzwl	efi32_boot_ds(%rip), %edx
-	movzwq	efi32_boot_cs(%rip), %rax
-	pushq	%rax
-	leaq	efi_enter32(%rip), %rax
-	pushq	%rax
-	lretq
+	/* Reload firmware's data segment selectors */
+	movw	efi32_boot_ds(%rip), %bx
+	movl	%ebx, %ds
+	movl	%ebx, %es
+	movl	%ebx, %ss
+	movl	%ebx, %fs
+	movl	%ebx, %gs
+
+	/* Move args #5 and #6 into 32-bit accessible registers */
+	movl	%r8d, %eax
+	movl	%r9d, %ebx
 
-1:	addq	$64, %rsp
+	lcalll	*efi32_thunk(%rip)
+
+	addq	$16, %rsp
 	movq	%rdi, %rax
 
 	/* Clear out 32-bit segment selectors */
@@ -137,7 +121,6 @@ SYM_FUNC_START(__efi64_thunk)
 	movl	%ebx, %gs
 
 	pop	%rbx
-	pop	%rbp
 	RET
 SYM_FUNC_END(__efi64_thunk)
 
@@ -163,17 +146,27 @@ SYM_FUNC_END(efi32_stub_entry)
 #endif
 
 /*
- * EFI service pointer must be in %edi.
+ * Called using a far call from 64-bit code, using the x86_64 SysV ABI (except
+ * for R8/R9 which are inaccessible to 32-bit code - EAX/EBX are used instead).
+ *
+ * The first argument (EDI) is a pointer to the boot service or protocol, to
+ * which the remaining arguments are passed, each truncated to 32 bits.
  *
- * The stack should represent the 32-bit calling convention.
+ * Entered with ESP+40 pointing to the arguments passed via the stack, and with
+ * the 64-bit mode GDT and IDT descriptors at ESP+8 and ESP+14, respectively.
  */
 SYM_FUNC_START_LOCAL(efi_enter32)
-	/* Load firmware selector into data and stack segment registers */
-	movl	%edx, %ds
-	movl	%edx, %es
-	movl	%edx, %fs
-	movl	%edx, %gs
-	movl	%edx, %ss
+	/*
+	 * Convert x86-64 SysV ABI params to i386 ABI
+	 */
+	pushl	56(%esp)	/* Up to 3 args passed via the caller's stack */
+	pushl	52(%esp)
+	pushl	48(%esp)
+	pushl	%ebx		/* R9 */
+	pushl	%eax		/* R8 */
+	pushl	%ecx
+	pushl	%edx
+	pushl	%esi
 
 	/* Reload pgtables */
 	movl	%cr3, %eax
@@ -201,8 +194,9 @@ SYM_FUNC_START_LOCAL(efi_enter32)
 	 */
 	cli
 
-	lidtl	16(%ebx)
-	lgdtl	(%ebx)
+	addl	$32, %esp
+	lidtl	14(%esp)
+	lgdtl	8(%esp)
 
 	movl	%cr4, %eax
 	btsl	$(X86_CR4_PAE_BIT), %eax
@@ -219,9 +213,6 @@ SYM_FUNC_START_LOCAL(efi_enter32)
 	xorl	%eax, %eax
 	lldt	%ax
 
-	pushl	$__KERNEL_CS
-	pushl	%ebp
-
 	/* Enable paging */
 	movl	%cr0, %eax
 	btsl	$X86_CR0_PG_BIT, %eax
@@ -250,6 +241,10 @@ SYM_FUNC_START_LOCAL(efi32_entry)
 	movw	%cs, (efi32_boot_cs - 1b)(%ebx)
 	movw	%ds, (efi32_boot_ds - 1b)(%ebx)
 
+	/* Fix up absolute reference */
+	leal	(efi32_thunk - 1b)(%ebx), %eax
+	addl	%eax, (%eax)
+
 	/* Store firmware IDT descriptor */
 	sidtl	(efi32_boot_idt - 1b)(%ebx)
 
@@ -351,6 +346,7 @@ SYM_DATA_END(efi32_boot_idt)
 
 	.data
 	.balign	4
+SYM_DATA_LOCAL(efi32_thunk, .long efi_enter32 - .)
 SYM_DATA_LOCAL(efi32_boot_cs, .word 0)
 SYM_DATA_LOCAL(efi32_boot_ds, .word 0)
 SYM_DATA(efi_is64, .byte 1)
-- 
2.47.1.613.gc27f4b7a9f-goog

[PATCH 5/6] x86/efi/mixed: Reduce padding by moving some code around

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

The 32-bit and 64-bit entrypoints for the EFI handover protocol need to
be exactly 0x200 bytes apart, and so the 64-bit version is placed at the
end, using padding to move it into place.

This padding can be reduced now, since the code in between has reduced
in size as well. So move efi32_stub_entry() to the beginning of the
source file.

Note that this code is outside of the compressed kernel image, and so
the padding is not compressed.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/x86/boot/compressed/efi_mixed.S | 51 ++++++++++----------
 1 file changed, 26 insertions(+), 25 deletions(-)

diff --git a/arch/x86/boot/compressed/efi_mixed.S b/arch/x86/boot/compressed/efi_mixed.S
index a44e522bbbed..b5a46a4ce083 100644
--- a/arch/x86/boot/compressed/efi_mixed.S
+++ b/arch/x86/boot/compressed/efi_mixed.S
@@ -21,12 +21,32 @@
 #include <asm/processor-flags.h>
 #include <asm/segment.h>
 
-	.code64
 	.text
 	.balign	8
 SYM_DATA_LOCAL(gdt, .quad 0x0, 0x0, 0xaf9a000000ffff)	/* __KERNEL_CS */
 	.set	gdt_size, . - gdt
 
+#ifdef CONFIG_EFI_HANDOVER_PROTOCOL
+	.code32
+SYM_FUNC_START_NOALIGN(efi32_stub_entry)
+	call	1f
+1:	popl	%ecx
+
+	/* Clear BSS */
+	xorl	%eax, %eax
+	leal	(_bss - 1b)(%ecx), %edi
+	leal	(_ebss - 1b)(%ecx), %ecx
+	subl	%edi, %ecx
+	shrl	$2, %ecx
+	cld
+	rep	stosl
+
+	add	$0x4, %esp		/* Discard return address */
+	movl	8(%esp), %edi		/* struct boot_params pointer */
+	jmp	efi32_entry
+SYM_FUNC_END(efi32_stub_entry)
+#endif
+
 /*
  * When booting in 64-bit mode on 32-bit EFI firmware, startup_64_mixed_mode()
  * is the first thing that runs after switching to long mode. Depending on
@@ -50,6 +70,7 @@ SYM_DATA_LOCAL(gdt, .quad 0x0, 0x0, 0xaf9a000000ffff)	/* __KERNEL_CS */
  *                          |   vmlinux   |<----| efi_stub_entry |<--------+
  *                          +-------------+     +----------------+
  */
+	.code64
 SYM_FUNC_START_LOCAL_NOALIGN(startup_64_mixed_mode)
 	xorl	%eax, %eax
 	movl	%eax, %ds
@@ -79,7 +100,7 @@ SYM_FUNC_START_LOCAL_NOALIGN(startup_64_mixed_mode)
 	jmp	efi_pe_entry
 SYM_FUNC_END(startup_64_mixed_mode)
 
-SYM_FUNC_START(__efi64_thunk)
+SYM_FUNC_START_NOALIGN(__efi64_thunk)
 	push	%rbx
 
 	/* Store live GDT and IDT descriptors */
@@ -125,26 +146,6 @@ SYM_FUNC_START(__efi64_thunk)
 SYM_FUNC_END(__efi64_thunk)
 
 	.code32
-#ifdef CONFIG_EFI_HANDOVER_PROTOCOL
-SYM_FUNC_START(efi32_stub_entry)
-	call	1f
-1:	popl	%ecx
-
-	/* Clear BSS */
-	xorl	%eax, %eax
-	leal	(_bss - 1b)(%ecx), %edi
-	leal	(_ebss - 1b)(%ecx), %ecx
-	subl	%edi, %ecx
-	shrl	$2, %ecx
-	cld
-	rep	stosl
-
-	add	$0x4, %esp		/* Discard return address */
-	movl	8(%esp), %edi		/* struct boot_params pointer */
-	jmp	efi32_entry
-SYM_FUNC_END(efi32_stub_entry)
-#endif
-
 /*
  * Called using a far call from 64-bit code, using the x86_64 SysV ABI (except
  * for R8/R9 which are inaccessible to 32-bit code - EAX/EBX are used instead).
@@ -155,7 +156,7 @@ SYM_FUNC_END(efi32_stub_entry)
  * Entered with ESP+40 pointing to the arguments passed via the stack, and with
  * the 64-bit mode GDT and IDT descriptors at ESP+8 and ESP+14, respectively.
  */
-SYM_FUNC_START_LOCAL(efi_enter32)
+SYM_FUNC_START_LOCAL_NOALIGN(efi_enter32)
 	/*
 	 * Convert x86-64 SysV ABI params to i386 ABI
 	 */
@@ -232,7 +233,7 @@ SYM_FUNC_END(efi_enter32)
  * stub may still exit and return to the firmware using the Exit() EFI boot
  * service.]
  */
-SYM_FUNC_START_LOCAL(efi32_entry)
+SYM_FUNC_START_LOCAL_NOALIGN(efi32_entry)
 	call	1f
 1:	pop	%ebx
 
@@ -306,7 +307,7 @@ SYM_FUNC_END(efi32_entry)
  * efi_status_t efi32_pe_entry(efi_handle_t image_handle,
  *			       efi_system_table_32_t *sys_table)
  */
-SYM_FUNC_START(efi32_pe_entry)
+SYM_FUNC_START_NOALIGN(efi32_pe_entry)
 	pushl	%ebx				// save callee-save registers
 
 	/* Check whether the CPU supports long mode */
-- 
2.47.1.613.gc27f4b7a9f-goog

[PATCH 6/6] x86/efi/mixed: Move mixed mode startup code into libstub

[Ard Biesheuvel]

From: Ard Biesheuvel <ardb@kernel.org>

The EFI mixed mode code has been decoupled from the legacy decompressor,
in order to be able to reuse it with generic EFI zboot images for x86.

Move the source file into the libstub source directory to facilitate
this.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/x86/boot/compressed/Makefile                                                | 1 -
 drivers/firmware/efi/libstub/Makefile                                            | 3 +++
 arch/x86/boot/compressed/efi_mixed.S => drivers/firmware/efi/libstub/x86-mixed.S | 0
 3 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index f2051644de94..fc5563704466 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -104,7 +104,6 @@ vmlinux-objs-$(CONFIG_INTEL_TDX_GUEST) += $(obj)/tdx.o $(obj)/tdcall.o $(obj)/td
 vmlinux-objs-$(CONFIG_UNACCEPTED_MEMORY) += $(obj)/mem.o
 
 vmlinux-objs-$(CONFIG_EFI) += $(obj)/efi.o
-vmlinux-objs-$(CONFIG_EFI_MIXED) += $(obj)/efi_mixed.o
 vmlinux-libs-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
 
 $(obj)/vmlinux: $(vmlinux-objs-y) $(vmlinux-libs-y) FORCE
diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile
index e04285a7a6b9..f8cbd4557131 100644
--- a/drivers/firmware/efi/libstub/Makefile
+++ b/drivers/firmware/efi/libstub/Makefile
@@ -62,6 +62,8 @@ KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_LTO), $(KBUILD_CFLAGS))
 # `-fdata-sections` flag from KBUILD_CFLAGS_KERNEL
 KBUILD_CFLAGS_KERNEL := $(filter-out -fdata-sections, $(KBUILD_CFLAGS_KERNEL))
 
+KBUILD_AFLAGS			:= $(KBUILD_CFLAGS) -D__ASSEMBLY__
+
 lib-y				:= efi-stub-helper.o gop.o secureboot.o tpm.o \
 				   file.o mem.o random.o randomalloc.o pci.o \
 				   skip_spaces.o lib-cmdline.o lib-ctype.o \
@@ -83,6 +85,7 @@ lib-$(CONFIG_EFI_GENERIC_STUB)	+= efi-stub.o string.o intrinsics.o systable.o \
 lib-$(CONFIG_ARM)		+= arm32-stub.o
 lib-$(CONFIG_ARM64)		+= kaslr.o arm64.o arm64-stub.o smbios.o
 lib-$(CONFIG_X86)		+= x86-stub.o smbios.o
+lib-$(CONFIG_EFI_MIXED)		+= x86-mixed.o
 lib-$(CONFIG_X86_64)		+= x86-5lvl.o
 lib-$(CONFIG_RISCV)		+= kaslr.o riscv.o riscv-stub.o
 lib-$(CONFIG_LOONGARCH)		+= loongarch.o loongarch-stub.o
diff --git a/arch/x86/boot/compressed/efi_mixed.S b/drivers/firmware/efi/libstub/x86-mixed.S
similarity index 100%
rename from arch/x86/boot/compressed/efi_mixed.S
rename to drivers/firmware/efi/libstub/x86-mixed.S
-- 
2.47.1.613.gc27f4b7a9f-goog

Re: [PATCH 0/6] x86/efi/mixed: Decouple from legacy decompressor

[Ingo Molnar]

* Ard Biesheuvel <ardb+git@google.com> wrote:

> From: Ard Biesheuvel <ardb@kernel.org>
> 
> Since commit
> 
>   a1b87d54f4e4 ("x86/efistub: Avoid legacy decompressor when doing EFI boot")
> 
> booting via the EFI stub no longer relies on the legacy decompressor,
> and instead, the kernel proper is decompressed by code executing in the
> context of the EFI boot services, and subsequently invoked directly.
> 
> The only remaining dependency is the EFI mixed mode startup code, which
> makes a detour via the legacy decompressor's 32-bit entrypoint, in order
> to obtain a 1:1 mapping of memory, which is a prerequisite for 64-bit
> execution on x86.
> 
> This detour requires some fiddly setup on the part of the mixed mode
> startup code, which has to stash the firmware stack pointer and boot
> arguments in memory, and create a fake struct boot_params to trick the
> code in startup_32 to behave as intended.
> 
> This dependency also impedes reuse of the EFI stub code in other
> contexts, such as generic EFI zboot, which will reuse the EFI stub but
> not the legacy decompressor.
> 
> So remove this dependency, by replacing this detour with a minimal
> reimplementation of the 1:1 mapping code. With some further cleanup
> applied on top, the line count barely changes, but the resulting code
> can operate independently from the legacy decompressor, and is therefore
> moved out of arch/x86/boot/compressed and into the EFI libstub/
> directory.
> 
> Ard Biesheuvel (6):
>   x86/efi/mixed: Check CPU compatibility without relying on verify_cpu()
>   x86/efi/mixed: Remove dependency on legacy startup_32 code
>   x86/efi/mixed: Don't bother preserving 64-bit mode segment selectors
>   x86/efi/mixed: Simplify and document thunking logic
>   x86/efi/mixed: Reduce padding by moving some code around
>   x86/efi/mixed: Move mixed mode startup code into libstub
> 
>  arch/x86/boot/compressed/Makefile                                                |   1 -
>  arch/x86/boot/compressed/head_64.S                                               |   7 -
>  drivers/firmware/efi/libstub/Makefile                                            |   3 +
>  arch/x86/boot/compressed/efi_mixed.S => drivers/firmware/efi/libstub/x86-mixed.S | 320 ++++++++++----------
>  4 files changed, 169 insertions(+), 162 deletions(-)
>  rename arch/x86/boot/compressed/efi_mixed.S => drivers/firmware/efi/libstub/x86-mixed.S (54%)

A belated:

    Acked-by: Ingo Molnar <mingo@kernel.org>

Thanks,

	Ingo