[REGRESSION] boot fails for EFI boot stub loaded by u-boot

[REGRESSION] boot fails for EFI boot stub loaded by u-boot

[Ben Schneider]

Hi Ard,

I have an ESPRESSObin Ultra (aarch64) that uses U-Boot as its bootloader. It shipped from the manufacturer with with v5.10, and I've been trying to upgrade. U-Boot supports booting Image directly via EFI (https://u-boot.readthedocs.io/en/latest/usage/cmd/bootefi.html), and I have been using it that way to successfully boot the system up to and including v6.0.19. However, v6.1 and v6.5 kernels fail to boot.

When booting successfully, the following messages are displayed:

EFI stub: Booting Linux Kernel...EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value
EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
EFI stub: Using DTB from configuration table
EFI stub: ERROR: Failed to install memreserve config table!
EFI stub: Exiting boot services...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]

I suspect many of the above error messages are simply attributable to using U-Boot to load an EFI stub and can be safely ignored given that the system boots and runs fine.

When boot fails (v6.5), the following messages are displayed:

EFI stub: Booting Linux Kernel...
EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value
EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
EFI stub: ERROR: Failed to install memreserve config table!
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
EFI stub: ERROR: Unable to construct new device tree.
EFI stub: ERROR: Failed to update FDT and exit boot services

In case it's relevant, the device tree for this device is arch/arm64/boot/marvell/armada-3720-espressobin-ultra.dts

Hopefully I've reported this in the correct place or that the information provided is sufficient to get it where it needs to be. Let me know if there is additional information I can provide. I am also able to use the device to test.

Sincerely,

Ben Schneider

#regzbot introduced v6.0..v6.1

Re: [REGRESSION] boot fails for EFI boot stub loaded by u-boot

[Ard Biesheuvel]

(cc Heinrich)

Hello Ben,

Thanks for the report.

On Wed, 18 Oct 2023 at 03:19, Ben Schneider <ben@bens.haus> wrote:
>
> Hi Ard,
>
> I have an ESPRESSObin Ultra (aarch64) that uses U-Boot as its bootloader. It shipped from the manufacturer with with v5.10, and I've been trying to upgrade. U-Boot supports booting Image directly via EFI (https://u-boot.readthedocs.io/en/latest/usage/cmd/bootefi.html), and I have been using it that way to successfully boot the system up to and including v6.0.19. However, v6.1 and v6.5 kernels fail to boot.
>
> When booting successfully, the following messages are displayed:
>
> EFI stub: Booting Linux Kernel...EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value
> EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
> EFI stub: Using DTB from configuration table
> EFI stub: ERROR: Failed to install memreserve config table!
> EFI stub: Exiting boot services...
> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
>
> I suspect many of the above error messages are simply attributable to using U-Boot to load an EFI stub and can be safely ignored given that the system boots and runs fine.
>

I suspect that these are not as harmless as you think. How old is the
u-boot build on this platform?

> When boot fails (v6.5), the following messages are displayed:
>
> EFI stub: Booting Linux Kernel...
> EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value
> EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
> EFI stub: ERROR: Failed to install memreserve config table!
> EFI stub: Using DTB from configuration table
> EFI stub: Exiting boot services...
> EFI stub: ERROR: Unable to construct new device tree.
> EFI stub: ERROR: Failed to update FDT and exit boot services
>
> In case it's relevant, the device tree for this device is arch/arm64/boot/marvell/armada-3720-espressobin-ultra.dts
>

This is a uboot path, right? Not a linux path? Are you sure this DTS
is compatible with the v6.5 kernel?

> Hopefully I've reported this in the correct place or that the information provided is sufficient to get it where it needs to be. Let me know if there is additional information I can provide. I am also able to use the device to test.
>

Please add some efi_warn() message inside the update_fdt() routine in
drivers/firmware/efi/libstub/fdt.c to narrow down which call is
causing it to return an error. Nothing in that code jumps out to me,
but we regularly update libfdt in the kernel as well, so it might be a
change in there that triggers this.

Re: [REGRESSION] boot fails for EFI boot stub loaded by u-boot

[Heinrich Schuchardt]

On 10/18/23 10:34, Ard Biesheuvel wrote:
> (cc Heinrich)
> 
> Hello Ben,
> 
> Thanks for the report.
> 
> On Wed, 18 Oct 2023 at 03:19, Ben Schneider <ben@bens.haus> wrote:
>>
>> Hi Ard,
>>
>> I have an ESPRESSObin Ultra (aarch64) that uses U-Boot as its bootloader. It shipped from the manufacturer with with v5.10, and I've been trying to upgrade. U-Boot supports booting Image directly via EFI (https://u-boot.readthedocs.io/en/latest/usage/cmd/bootefi.html), and I have been using it that way to successfully boot the system up to and including v6.0.19. However, v6.1 and v6.5 kernels fail to boot.
>>
>> When booting successfully, the following messages are displayed:
>>
>> EFI stub: Booting Linux Kernel...EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value
>> EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
>> EFI stub: Using DTB from configuration table
>> EFI stub: ERROR: Failed to install memreserve config table!
>> EFI stub: Exiting boot services...
>> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
>>
>> I suspect many of the above error messages are simply attributable to using U-Boot to load an EFI stub and can be safely ignored given that the system boots and runs fine.

These messages are not typical for launching a kernel via the EFI stub 
from U-Boot. It should look like this:

=> load mmc 0:1 $fdt_addr_r boot/dtb
28846 bytes read in 6 ms (4.6 MiB/s)
=> load mmc 0:1 $kernel_addr_r boot/vmlinuz
53686664 bytes read in 2223 ms (23 MiB/s)
=> setenv bootargs root=/dev/mmcblk0p1 efi=debug earlyprintk 
initrd=boot/initrd.img
=> bootefi $kernel_addr_r $fdt_addr_r
Card did not respond to voltage select! : -110
Failed to load EFI variables
Booting /boot\vmlinuz
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable
EFI stub: Loaded initrd from command line option
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]

>>
> 
> I suspect that these are not as harmless as you think. How old is the
> u-boot build on this platform?
> 
>> When boot fails (v6.5), the following messages are displayed:
>>
>> EFI stub: Booting Linux Kernel...
>> EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value
>> EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
>> EFI stub: ERROR: Failed to install memreserve config table!
>> EFI stub: Using DTB from configuration table
>> EFI stub: Exiting boot services...
>> EFI stub: ERROR: Unable to construct new device tree.
>> EFI stub: ERROR: Failed to update FDT and exit boot services
>>
>> In case it's relevant, the device tree for this device is arch/arm64/boot/marvell/armada-3720-espressobin-ultra.dts
>>
> 
> This is a uboot path, right? Not a linux path? Are you sure this DTS
> is compatible with the v6.5 kernel?

There is no arch/arm64/ in U-Boot. Maybe 
arch/arm64/boot/dts/marvell/armada-3720-espressobin-ultra.dts?

Best regards

Heinrich

> 
>> Hopefully I've reported this in the correct place or that the information provided is sufficient to get it where it needs to be. Let me know if there is additional information I can provide. I am also able to use the device to test.
>>
> 
> Please add some efi_warn() message inside the update_fdt() routine in
> drivers/firmware/efi/libstub/fdt.c to narrow down which call is
> causing it to return an error. Nothing in that code jumps out to me,
> but we regularly update libfdt in the kernel as well, so it might be a
> change in there that triggers this.

Re: [REGRESSION] boot fails for EFI boot stub loaded by u-boot

[Ilias Apalodimas]

Hi all,

On Wed, 18 Oct 2023 at 12:17, Heinrich Schuchardt
<heinrich.schuchardt@canonical.com> wrote:
>
> On 10/18/23 10:34, Ard Biesheuvel wrote:
> > (cc Heinrich)
> >
> > Hello Ben,
> >
> > Thanks for the report.
> >
> > On Wed, 18 Oct 2023 at 03:19, Ben Schneider <ben@bens.haus> wrote:
> >>
> >> Hi Ard,
> >>
> >> I have an ESPRESSObin Ultra (aarch64) that uses U-Boot as its bootloader. It shipped from the manufacturer with with v5.10, and I've been trying to upgrade. U-Boot supports booting Image directly via EFI (https://u-boot.readthedocs.io/en/latest/usage/cmd/bootefi.html), and I have been using it that way to successfully boot the system up to and including v6.0.19. However, v6.1 and v6.5 kernels fail to boot.
> >>
> >> When booting successfully, the following messages are displayed:
> >>
> >> EFI stub: Booting Linux Kernel...EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value
> >> EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
> >> EFI stub: Using DTB from configuration table
> >> EFI stub: ERROR: Failed to install memreserve config table!
> >> EFI stub: Exiting boot services...
> >> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
> >>
> >> I suspect many of the above error messages are simply attributable to using U-Boot to load an EFI stub and can be safely ignored given that the system boots and runs fine.
>
> These messages are not typical for launching a kernel via the EFI stub
> from U-Boot. It should look like this:

The alignment one might be true depending on the U-Boot version that's used.
The alignment was fixed on commit ebdea88d57d5e ("efi_loader: Fix
loaded image alignment")
The rest indeed make little sense for now

>
> => load mmc 0:1 $fdt_addr_r boot/dtb
> 28846 bytes read in 6 ms (4.6 MiB/s)
> => load mmc 0:1 $kernel_addr_r boot/vmlinuz
> 53686664 bytes read in 2223 ms (23 MiB/s)
> => setenv bootargs root=/dev/mmcblk0p1 efi=debug earlyprintk
> initrd=boot/initrd.img
> => bootefi $kernel_addr_r $fdt_addr_r
> Card did not respond to voltage select! : -110
> Failed to load EFI variables
> Booting /boot\vmlinuz
> EFI stub: Booting Linux Kernel...
> EFI stub: EFI_RNG_PROTOCOL unavailable
> EFI stub: Loaded initrd from command line option
> EFI stub: Using DTB from configuration table
> EFI stub: Exiting boot services...
> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
>

[...]

Cheers
/Ilias

Re: [REGRESSION] boot fails for EFI boot stub loaded by u-boot

[Ben Schneider]

> How old is the u-boot build on this platform?

U-Boot 2018.03-devel-18.12.3-g926d08c7ce (Apr 11 2022 - 15:48:13 +0800)

This appears to be the most recent version the manufacturer has released for this device. Source: https://github.com/globalscaletechnologies/u-boot-marvell.

>> arch/arm64/boot/marvell/armada-3720-espressobin-ultra.dts
> This is a uboot path, right? Not a linux path? Are you sure this DTS is compatible with the v6.5 kernel?

Sorry for the confusion; that is the path in the linux source to the DTS used to compile the DTB that I am using to boot the device. I booted v5.15.135 using the DTB compiled from v6.5.7 source and that works fine. I also tried to boot v6.5.7 with the factory DTB and that failed.

> Please add message inside the update_fdt() routine...

I added a bunch and here's what I got back:

EFI stub: Booting Linux Kernel...
EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value
EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
EFI stub: ERROR: Failed to install memreserve config table!
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services...
EFI stub: Starting update_fdt()...
EFI stub: fdt_num_mem_rsv() returned 1
EFI stub: fdt_subnode_offset() returned 8944
EFI stub: Setting bootargs=console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000 root=/dev/sda1 rw rootwait
EFI stub: Adding FDT entries...
EFI stub: fdt_setprop_var() for linux,uefi-system-table returned 0
EFI stub: fdt_setprop_var() for linux,uefi-mmap-start returned -11
EFI stub: update_fdt() failed with status -11
EFI stub: ERROR: Unable to construct new device tree.
EFI stub: ERROR: Failed to update FDT and exit boot services

That's as far as I could get today but hopefully that starts to narrow it down. Appreciate the help!

Sincerely,

Ben

Re: [REGRESSION] boot fails for EFI boot stub loaded by u-boot

[Heinrich Schuchardt]

On 10/19/23 03:51, Ben Schneider wrote:
>> How old is the u-boot build on this platform?
> 
> U-Boot 2018.03-devel-18.12.3-g926d08c7ce (Apr 11 2022 - 15:48:13 +0800)
> 
> This appears to be the most recent version the manufacturer has released for this device. Source: https://github.com/globalscaletechnologies/u-boot-marvell.
> 
>>> arch/arm64/boot/marvell/armada-3720-espressobin-ultra.dts
>> This is a uboot path, right? Not a linux path? Are you sure this DTS is compatible with the v6.5 kernel?
> 
> Sorry for the confusion; that is the path in the linux source to the DTS used to compile the DTB that I am using to boot the device. I booted v5.15.135 using the DTB compiled from v6.5.7 source and that works fine. I also tried to boot v6.5.7 with the factory DTB and that failed.

To which kernel and device-tree are the messages below related?

> 
>> Please add message inside the update_fdt() routine...
> 
> I added a bunch and here's what I got back:
> 
> EFI stub: Booting Linux Kernel...
> EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value
> EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
> EFI stub: ERROR: Failed to install memreserve config table!
> EFI stub: Using DTB from configuration table
> EFI stub: Exiting boot services...
> EFI stub: Starting update_fdt()...
> EFI stub: fdt_num_mem_rsv() returned 1
> EFI stub: fdt_subnode_offset() returned 8944
> EFI stub: Setting bootargs=console=ttyMV0,115200 earlycon=ar3700_uart,0xd0012000 root=/dev/sda1 rw rootwait
> EFI stub: Adding FDT entries...
> EFI stub: fdt_setprop_var() for linux,uefi-system-table returned 0
> EFI stub: fdt_setprop_var() for linux,uefi-mmap-start returned -11

11 = FDT_ERR_BADSTRUCTURE
This is probably set in scripts/dtc/libfdt/fdt_ro.c.
Something in the structure of your device-tree is invalid.

Please, check the load addresses in U-Boot. Is something overwriting the 
tail of the device-tree?

Compiling upstream U-Boot's qemu_arm64_defconfig yields 
lib/efi_loader/dtbdump.efi. If you run this instead of the kernel, you 
can write the device-tree as it is passed in a configuration table to 
the ESP.

Best regards

Heinrich

> EFI stub: update_fdt() failed with status -11
> EFI stub: ERROR: Unable to construct new device tree.
> EFI stub: ERROR: Failed to update FDT and exit boot services
> 
> That's as far as I could get today but hopefully that starts to narrow it down. Appreciate the help!
> 
> Sincerely,
> 
> Ben

Re: [REGRESSION] boot fails for EFI boot stub loaded by u-boot

[Ben Schneider]

Oct 19, 2023, 07:21 by heinrich.schuchardt@canonical.com:

> To which kernel and device-tree are the messages below related?
>
Sorry. I'm building/testing with v6.5.7 right now, but I have not been able to boot any kernel version >= 6.1. The device tree I used also came from linux v6.5.7. Its source is at arch/arm64/boot/dts/marvell/armada-3720-espressobin-ultra.dts.

> Something in the structure of your device-tree is invalid.
>
The device does boot using kernel v5.15 and the device tree compiled from linux v6.5.7. The device tree could still very well be a problem. I just didn't start there because whatever problems the device tree may have don't prevent the device from booting for kernels prior to v6.1.

> Please, check the load addresses in U-Boot. Is something overwriting the tail of the device-tree?
>
I have always loaded the device tree to 0x1000000 (16MiB) and the kernel to 0x2000000 (32MiB) for no particular reason except that's what the manufacturer did. armada-3720-espressobin-ultra.dtb is only 14K. I don't load an initramfs or anything else to memory with u-boot and the address used to load u-boot environment is 0x6000000.

> Compiling upstream U-Boot's qemu_arm64_defconfig yields lib/efi_loader/dtbdump.efi. If you run this instead of the kernel, you can write the device-tree as it is passed in a configuration table to the ESP.
>
I compiled and ran this fine, but I was unable to save the device tree. I suspect this is because the program searches for an ESP, and there is none on the device. U-boot was compiled with support to load directly from an ext4 filesystem so I didn't bother setting one up. I will work on it. I can convert the .dtb on disk back to a human-readable .dts easily with dtc if that is helpful.

Thanks!

Ben

Re: [REGRESSION] boot fails for EFI boot stub loaded by u-boot

[Ben Schneider]

Oct 20, 2023, 01:25 by ben@bens.haus:

> Oct 19, 2023, 07:21 by heinrich.schuchardt@canonical.com:
>
>> Compiling upstream U-Boot's qemu_arm64_defconfig yields lib/efi_loader/dtbdump.efi. If you run this instead of the kernel, you can write the device-tree as it is passed in a configuration table to the ESP.
>>
> I compiled and ran this fine, but I was unable to save the device tree. I suspect this is because the program searches for an ESP, and there is none on the device. U-boot was compiled with support to load directly from an ext4 filesystem so I didn't bother setting one up. I will work on it. 
>
Hi Heinrich, I loaded dtbdump.efi from a FAT32 formatted partition with type EFI System, but attempts to run the save command return "Failed to open simple file system protocol". Sorry if there is something else I am missing.

Ben

Re: [REGRESSION] boot fails for EFI boot stub loaded by u-boot

[Heinrich Schuchardt]

On 10/21/23 02:07, Ben Schneider wrote:
> Oct 20, 2023, 01:25 by ben@bens.haus:
> 
>> Oct 19, 2023, 07:21 by heinrich.schuchardt@canonical.com:
>>
>>> Compiling upstream U-Boot's qemu_arm64_defconfig yields lib/efi_loader/dtbdump.efi. If you run this instead of the kernel, you can write the device-tree as it is passed in a configuration table to the ESP.
>>>
>> I compiled and ran this fine, but I was unable to save the device tree. I suspect this is because the program searches for an ESP, and there is none on the device. U-boot was compiled with support to load directly from an ext4 filesystem so I didn't bother setting one up. I will work on it.
>>
> Hi Heinrich, I loaded dtbdump.efi from a FAT32 formatted partition with type EFI System, but attempts to run the save command return "Failed to open simple file system protocol". Sorry if there is something else I am missing.

On upstream U-Boot I cannot see this problem.

Best regards

Heinrich

Re: [REGRESSION] boot fails for EFI boot stub loaded by u-boot

[Ben Schneider]

Oct 18, 2023, 09:17 by heinrich.schuchardt@canonical.com:

> On 10/18/23 10:34, Ard Biesheuvel wrote:
>
>> (cc Heinrich)
>>
>> Hello Ben,
>>
>> Thanks for the report.
>>
>> On Wed, 18 Oct 2023 at 03:19, Ben Schneider <ben@bens.haus> wrote:
>>
>>>
>>> Hi Ard,
>>>
>>> I have an ESPRESSObin Ultra (aarch64) that uses U-Boot as its bootloader. It shipped from the manufacturer with with v5.10, and I've been trying to upgrade. U-Boot supports booting Image directly via EFI (https://u-boot.readthedocs.io/en/latest/usage/cmd/bootefi.html), and I have been using it that way to successfully boot the system up to and including v6.0.19. However, v6.1 and v6.5 kernels fail to boot.
>>>
>>> When booting successfully, the following messages are displayed:
>>>
>>> EFI stub: Booting Linux Kernel...EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value
>>> EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary
>>> EFI stub: Using DTB from configuration table
>>> EFI stub: ERROR: Failed to install memreserve config table!
>>> EFI stub: Exiting boot services...
>>> [    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
>>>
>>> I suspect many of the above error messages are simply attributable to using U-Boot to load an EFI stub and can be safely ignored given that the system boots and runs fine.
>>>
>
> These messages are not typical for launching a kernel via the EFI stub from U-Boot. It should look like this:
>

All, I can confirm that this issue is not present using upstream U-Boot (v2024.01). That is, EFI stub booting works as expected and does not display any error messages. Thanks!

Ben

#regzbot invalid: issue is with device firmware (u-boot)