[U-Boot] bootefi failures on armv7

Heinrich Schuchardt
2019-04-10T18:27:47Z

On 4/10/19 6:57 PM, Ilias Apalodimas wrote:
> Hi Alexander, Heinrich,
>
> (Resending since i was too quick on the trigger forgot to cc the public
> mailing list. Sorry for the mess)
>
> I can't get an armv7 board (STM32MP157C-DK2) to boot with bootefi on
> specific kernel/uboot configurations.
> u-boot version: U-Boot 2019.04-rc3
> kernel version: 5.0.0-rc3
>
> What works:
> - bootefi + 'dtb=' on kernel cmdline regardless of kernel config
> - bootefi + fdtcontroladdr if CONFIG_ARM_LPAE=y on my kernel config
>
> What doesn't work:
> bootefi + fdtcontroladdr if CONFIG_ARM_LPAE is not set.
>
> Once this config is selected the kernel hangs on a BUG_ON() in arm_pte_alloc()
>
> Adding a dump_stack() there throws the relevant information.
> [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.0.0-rc3-29427-g769f1f8f9b56-dirty #127
> [ 0.000000] Hardware name: STM32 (Device Tree Support)
> [ 0.000000] [<c03123ec>] (unwind_backtrace) from [<c030ce0c>] (show_stack+0x10/0x14)
> [ 0.000000] [<c030ce0c>] (show_stack) from [<c0e27250>] (dump_stack+0x8c/0xa0)
> [ 0.000000] [<c0e27250>] (dump_stack) from [<c1507e9c>] (arm_pte_alloc+0x74/0xb8)
> [ 0.000000] [<c1507e9c>] (arm_pte_alloc) from [<c15081ec>] (__create_mapping+0x30c/0x36c)
> [ 0.000000] [<c15081ec>] (__create_mapping) from [<c15089f8>] (paging_init+0x234/0x648)
> [ 0.000000] [<c15089f8>] (paging_init) from [<c1504950>] (setup_arch+0x660/0xcac)
> [ 0.000000] [<c1504950>] (setup_arch) from [<c1500a4c>] (start_kernel+0x70/0x458)
> [ 0.000000] [<c1500a4c>] (start_kernel) from [<00000000>] ( (null))
>
> Kernel's EFI map:
> [ 0.000000] efi: 0x0000c0000000-0x0000c1ffffff [Boot Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c2000000-0x0000c2860fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c2861000-0x0000c7cfffff [Conventional Memory| | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c7d00000-0x0000c7efffff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c7f00000-0x0000c7f0bfff [Runtime Data |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c7f0c000-0x0000dc705fff [Conventional Memory| | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dc706000-0x0000dc709fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dc70a000-0x0000dcf6afff [Loader Code | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf6b000-0x0000dcf6bfff [Runtime Data |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf6c000-0x0000dcf72fff [Reserved | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf73000-0x0000dcf73fff [Runtime Data |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf74000-0x0000dcf75fff [Reserved | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf76000-0x0000dff80fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dff81000-0x0000dff81fff [Runtime Code |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dff82000-0x0000dfffffff [Loader Data | | | | | | | | |WB| | | ]
>
> I've found some hacky workarounds on that and i'd like some input on what's
> actually wrong
>
> - Workaround 1, add a bogus bootefi cmd
>
> instead of
> load mmc 0:5 $kernel_addr_r efi/BOOT/zImage;
> bootefi $kernel_addr_r $fdtcontroladdr
>
> doing
> bootefi $kernel_addr_r $fdtcontroladdr
> load mmc 0:5 $kernel_addr_r efi/BOOT/zImage
> bootefi $kernel_addr_r $fdtcontroladdr' works
>
> The first bootefi command will fail since there's nothing loaded on
> that address yet. What happens is that u-boot adds an extra
> memory_map, via efi_add_memory_map() and the kernel boots
>
> Kernel's EFI map:
> [ 0.000000] efi: Processing EFI memory map:
> [ 0.000000] efi: 0x0000c0000000-0x0000c1ffffff [Boot Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c2000000-0x0000c2860fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c2861000-0x0000c7cf3fff [Conventional Memory| | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c7cf4000-0x0000c7ef3fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c7ef4000-0x0000c7f0bfff [Runtime Data |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c7f0c000-0x0000dc704fff [Conventional Memory| | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dc705000-0x0000dc708fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dc709000-0x0000dcf69fff [Loader Code | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf6a000-0x0000dcf6cfff [Reserved | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf6d000-0x0000dcf6dfff [Runtime Data |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf6e000-0x0000dcf74fff [Reserved | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf75000-0x0000dcf75fff [Runtime Data |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf76000-0x0000dff80fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dff81000-0x0000dff81fff [Runtime Code |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dff82000-0x0000dfffffff [Loader Data | | | | | | | | |WB| | | ]
>
>
> - Workaround 2, copy fdt one page before the 127mb limit
>
> diff --git a/cmd/bootefi.c b/cmd/bootefi.c
> index 3619a20e6433..a0b6156845da 100644
> --- a/cmd/bootefi.c
> +++ b/cmd/bootefi.c
> @@ -108,11 +108,13 @@ static efi_status_t copy_fdt(void **fdtp)
> * Safe fdt location is at 127 MiB.

Hello Ilias,

thanks for reporting the issue.

@Simon, @Alex
Why did we ever introduce this value?
127 MB in baf70c02107 ("efi: Relocate FDT to 127MB instead of 128MB")
128 MB in ad0c1a3d2ce ("efi_loader: Put fdt into convenient location")

baf70c02107 says that the Sandbox has only 128 MiB. But why should we
need to hard code this here. When the Sandbox has initialized the memory
map efi_allocate_pool() will take care of finding an address in the
available memory anyway.

ad0c1a3d2ce mentions 128 MB as recommended value but does not provide
any reference. I think we should provide the source of this recommendation.

According to
https://www.st.com/en/microcontrollers-microprocessors/stm32mp157.html
STM32MP157 comes with support for TrustZone and SecureBoot.

https://www.st.com/en/microcontrollers-microprocessors/stm32mp157.html#tools-software:

"The developer package comes with

Boot chain based on TF-A and U-Boot
OP-TEE secure OS running on the Arm® Cortex®-A in secure mode."

The Linux device tree does not reserve a memory region for the trusted
firmware (TF-A) and the trusted execution environment (OP-TEE) Shouldn't
there be a reserved memory area for both.

If TF-A or OP-TEE reside near 0xC8000000, this would explain the
observed behavior.

@Ilias
Do you use TF-A or OP-TEE in your setup?

Best regards

Heinrich

> * On the sandbox convert from the sandbox address space.
> */
> - new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000 +
> - fdt_size, 0);
> + //new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000 +
> + //fdt_size, 0);
> + new_fdt_addr = (uintptr_t)map_sysmem(fdt_ram_start + 0x7f00000, fdt_size);
> ret = efi_allocate_pages(EFI_ALLOCATE_MAX_ADDRESS,
> EFI_RUNTIME_SERVICES_DATA, fdt_pages,
> &new_fdt_addr);
> + printf("FDTADDR: %llx pages %lu sz %x\n", new_fdt_addr, fdt_pages, fdt_size);
> if (ret != EFI_SUCCESS) {
> /* If we can't put it there, put it somewhere */
> new_fdt_addr = (ulong)memalign(EFI_PAGE_SIZE, fdt_size);
>
> This loads the fdt on c7ef4000 (which is more than a page). Changing
> the address from 0x7f00000 to 7EFF000, on the original code, works as
> well
>
> Kernel's EFI map (with the patch) :
> [ 0.000000] efi: Processing EFI memory map:
> [ 0.000000] efi: 0x0000c0000000-0x0000c1ffffff [Boot Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c2000000-0x0000c2860fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c2861000-0x0000c7cf3fff [Conventional Memory| | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c7cf4000-0x0000c7ef3fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c7ef4000-0x0000c7efffff [Runtime Data |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c7f00000-0x0000dc705fff [Conventional Memory| | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dc706000-0x0000dc709fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dc70a000-0x0000dcf6afff [Loader Code | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf6b000-0x0000dcf6bfff [Runtime Data |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf6c000-0x0000dcf72fff [Reserved | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf73000-0x0000dcf73fff [Runtime Data |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf74000-0x0000dcf75fff [Reserved | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf76000-0x0000dff80fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dff81000-0x0000dff81fff [Runtime Code |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dff82000-0x0000dfffffff [Loader Data | | | | | | | | |WB| | | ]
>
> - Workaround 3, align fdt on 1mb boundries
>
> diff --git a/cmd/bootefi.c b/cmd/bootefi.c
> index 3619a20e6433..37e849fad773 100644
> --- a/cmd/bootefi.c
> +++ b/cmd/bootefi.c
> @@ -101,7 +101,8 @@ static efi_status_t copy_fdt(void **fdtp)
> * needs to be expanded later.
> */
> fdt = *fdtp;
> - fdt_pages = efi_size_in_pages(fdt_totalsize(fdt) + 0x3000);
> + //fdt_pages = efi_size_in_pages(fdt_totalsize(fdt) + 0x3000);
> + fdt_pages = efi_size_in_pages(0x100000);
> fdt_size = fdt_pages << EFI_PAGE_SHIFT;
>
> /*
>
> Kernel's EFI map:
> [ 0.000000] efi: Processing EFI memory map:
> [ 0.000000] efi: 0x0000c0000000-0x0000c1ffffff [Boot Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c2000000-0x0000c2860fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c2861000-0x0000c7cfffff [Conventional Memory| | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c7d00000-0x0000c7efffff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c7f00000-0x0000c7ffffff [Runtime Data |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000c8000000-0x0000dc705fff [Conventional Memory| | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dc706000-0x0000dc709fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dc70a000-0x0000dcf6afff [Loader Code | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf6b000-0x0000dcf6bfff [Runtime Data |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf6c000-0x0000dcf72fff [Reserved | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf73000-0x0000dcf73fff [Runtime Data |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf74000-0x0000dcf75fff [Reserved | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dcf76000-0x0000dff80fff [Loader Data | | | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dff81000-0x0000dff81fff [Runtime Code |RUN| | | | | | | |WB| | | ]
> [ 0.000000] efi: 0x0000dff82000-0x0000dfffffff [Loader Data | | | | | | | | |WB| | | ]
>
>
> Thanks
> /Ilias
>