From mboxrd@z Thu Jan 1 00:00:00 1970 From: Cristian Ciocaltea Date: Sun, 29 Dec 2019 23:50:06 +0200 Subject: [PATCH v4 5/5] test/py: Create a test for launching UEFI binaries from FIT images In-Reply-To: <2c790471-f921-1f00-e00a-55753e01d3a8@gmx.de> References: <879e87d8-7af0-730f-af01-6fcf09a3d3c8@gmx.de> <20191229183920.GA28605@BV030612LT> <2c790471-f921-1f00-e00a-55753e01d3a8@gmx.de> Message-ID: <20191229215006.GA31348@BV030612LT> List-Id: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: u-boot@lists.denx.de On Sun, Dec 29, 2019 at 08:11:10PM +0100, Heinrich Schuchardt wrote: > On 12/29/19 7:39 PM, Cristian Ciocaltea wrote: > > On Sun, Dec 29, 2019 at 11:22:08AM +0100, Heinrich Schuchardt wrote: > > > On 12/24/19 5:05 PM, Cristian Ciocaltea wrote: > > > > This test verifies the implementation of the 'bootm' extension that > > > > handles UEFI binaries inside FIT images (enabled via CONFIG_BOOTM_EFI). > > > > > > > > Signed-off-by: Cristian Ciocaltea > > > > > > Thanks a lot for devising this test. > > > > > > --- > > > > > > You are using variable env__efi_fit_tftp_file. To run the test on Gitlab > > > and Travis CI a patch will be needed for: > > > > > > https://github.com/swarren/uboot-test-hooks.git > > > > > > I hope > > > > > > https://github.com/xypron/uboot-test-hooks/commit/20dcd721437dd5f7d7d3d235f7112246f43305d2 > > > > > > will do the job. > > > > > > Once we have this applied we will have to adjust the config files for QEMU. > > > > > > --- > > > > > > I have been trying to run the test on qemu_arm64_defconfig using the > > > following lines in u_boot_boardenv_qemu_arm64.py: > > > > > > env__efi_fit_tftp_file = { > > > "fn": "helloworld.efi", > > > "size": 4480, > > > "crc32": "19f9c0ab", > > > } > > > > > > I got an error: > > > > > > test/py/tests/test_efi_fit.py:417: in launch_efi > > > addr = load_fit_from_host(fit) if is_sandbox else > > > load_fit_from_tftp(fit) > > > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > > > addr = f.get('addr', None) > > > if not addr: > > > > addr = u_boot_utils.find_ram_base(cons) > > > E NameError: name 'u_boot_utils' is not defined > > > > > > > > > When I provided addr: > > > > > > env__efi_fit_tftp_file = { > > > "fn": "helloworld.efi", > > > "size": 4480, > > > "crc32": "19f9c0ab", > > > "addr": 0x40400000, > > > } > > > > > > I got the following error: > > > > > > => tftpboot 1073741824 helloworld.efi > > > TFTP error: trying to overwrite reserved memory... > > > > > > I would have expected a command > > > > > > tftpboot 40400000 helloworld.efi > > > > > > to be issued. > > > > > > Same error with bootm: > > > > > > => bootm 1077936128#config-efi-nofdt > > > "Synchronous Abort" handler, esr 0x96000010 > > > elr: 000000000001c36c lr : 00000000000140f4 (reloc) > > > > > > Please, fix the lines indicated below and verify that you can actually > > > execute this test on the QEMU platform. > > > > Thank you for the detailed report! > > > > Unfortunately I have only tested on qemu_arm and somehow I missed the > > check of having the address computed by the test suite. It used to work > > before I changed the address data type to string - the reason was to > > allow for more flexibility, e.g. providing values like '$kernel_addr_r' > > instead of just precomputed numbers. > > > > I'm going to extend the tests on qemu_arm64 as well. > > > > > > > > https://github.com/xypron/u-boot-build/tree/qemu-arm64/u-boot-test > > > > > > contains the files I use to run Python tests on qemu_arm64_defconfig. > > > > > > > --- > > > > test/py/tests/test_efi_fit.py | 459 ++++++++++++++++++++++++++++++++++ > > > > 1 file changed, 459 insertions(+) > > > > create mode 100644 test/py/tests/test_efi_fit.py > > > > > > > > diff --git a/test/py/tests/test_efi_fit.py b/test/py/tests/test_efi_fit.py > > > > new file mode 100644 > > > > index 0000000000..e1f0e42694 > > > > --- /dev/null > > > > +++ b/test/py/tests/test_efi_fit.py > > > > @@ -0,0 +1,459 @@ > > > > +# SPDX-License-Identifier: GPL-2.0 > > > > +# Copyright (c) 2019, Cristian Ciocaltea > > > > +# > > > > +# Work based on: > > > > +# - test_net.py > > > > +# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved. > > > > +# - test_fit.py > > > > +# Copyright (c) 2013, Google Inc. > > > > +# > > > > +# Test launching UEFI binaries from FIT images. > > > > + > > > > +import os.path > > > > +import pytest > > > > +import u_boot_utils as util > > > > > > "as util" causes an error if you use u_boot_utils.* below. Below I > > > indicate the places to change. > > > > Fixed, thanks. > > > > > > + > > > > +""" > > > > +Note: This test relies on boardenv_* containing configuration values to define > > > > +which network environment is available for testing. Without this, the parts > > > > +that rely on network will be automatically skipped. > > > > + > > > > +For example: > > > > + > > > > +# Boolean indicating whether the Ethernet device is attached to USB, and hence > > > > +# USB enumeration needs to be performed prior to network tests. > > > > +# This variable may be omitted if its value is False. > > > > +env__net_uses_usb = False > > > > + > > > > +# Boolean indicating whether the Ethernet device is attached to PCI, and hence > > > > +# PCI enumeration needs to be performed prior to network tests. > > > > +# This variable may be omitted if its value is False. > > > > +env__net_uses_pci = True > > > > + > > > > +# True if a DHCP server is attached to the network, and should be tested. > > > > +# If DHCP testing is not possible or desired, this variable may be omitted or > > > > +# set to False. > > > > +env__net_dhcp_server = True > > > > + > > > > +# A list of environment variables that should be set in order to configure a > > > > +# static IP. If solely relying on DHCP, this variable may be omitted or set to > > > > +# an empty list. > > > > +env__net_static_env_vars = [ > > > > + ('ipaddr', '10.0.0.100'), > > > > + ('netmask', '255.255.255.0'), > > > > + ('serverip', '10.0.0.1'), > > > > +] > > > > + > > > > +# Details regarding a file that may be read from a TFTP server. This variable > > > > +# may be omitted or set to None if TFTP testing is not possible or desired. > > > > +# Additionally, when the 'size' is not available, the file will be generated > > > > +# automatically in the TFTP root directory, as specified by the 'dn' field. > > > > +env__efi_fit_tftp_file = { > > > > + 'fn': 'test-efi-fit.img', # File path relative to TFTP root > > > > + 'size': 3831, # File size > > > > + 'crc32': '9fa3f79c', # Checksum using CRC-32 algorithm, optional > > > > + 'addr': '$kernel_addr_r', # Loading address, optional > > > > + 'dn': 'tftp/root/dir', # TFTP root directory path, optional > > > > +} > > > > +""" > > > > + > > > > +# Define the parametrized ITS data to be used for FIT images generation. > > > > +its_data = ''' > > > > +/dts-v1/; > > > > + > > > > +/ { > > > > + description = "EFI image with FDT blob"; > > > > + #address-cells = <1>; > > > > + > > > > + images { > > > > + efi { > > > > + description = "Test EFI"; > > > > + data = /incbin/("%(efi-bin)s"); > > > > + type = "%(kernel-type)s"; > > > > + arch = "%(sys-arch)s"; > > > > + os = "efi"; > > > > + compression = "%(efi-comp)s"; > > > > + load = <0x0>; > > > > + entry = <0x0>; > > > > + }; > > > > + fdt { > > > > + description = "Test FDT"; > > > > + data = /incbin/("%(fdt-bin)s"); > > > > + type = "flat_dt"; > > > > + arch = "%(sys-arch)s"; > > > > + compression = "%(fdt-comp)s"; > > > > + }; > > > > + }; > > > > + > > > > + configurations { > > > > + default = "config-efi-fdt"; > > > > + config-efi-fdt { > > > > + description = "EFI FIT w/ FDT"; > > > > + kernel = "efi"; > > > > + fdt = "fdt"; > > > > + }; > > > > + config-efi-nofdt { > > > > + description = "EFI FIT w/o FDT"; > > > > + kernel = "efi"; > > > > + }; > > > > + }; > > > > +}; > > > > +''' > > > > + > > > > +# Define the parametrized FDT data to be used for DTB images generation. > > > > +fdt_data = ''' > > > > +/dts-v1/; > > > > + > > > > +/ { > > > > + #address-cells = <1>; > > > > + #size-cells = <0>; > > > > + > > > > + model = "%(sys-arch)s %(fdt_type)s EFI FIT Boot Test"; > > > > + compatible = "%(sys-arch)s"; > > > > + > > > > + reset at 0 { > > > > + compatible = "%(sys-arch)s,reset"; > > > > + reg = <0>; > > > > + }; > > > > +}; > > > > +''' > > > > + > > > > + at pytest.mark.buildconfigspec('bootm_efi') > > > > + at pytest.mark.buildconfigspec('cmd_bootefi_hello_compile') > > > > + at pytest.mark.buildconfigspec('fit') > > > > + at pytest.mark.notbuildconfigspec('generate_acpi_table') > > > > + at pytest.mark.requiredtool('dtc') > > > > +def test_efi_fit_launch(u_boot_console): > > > > + """Test handling of UEFI binaries inside FIT images. > > > > + > > > > + The tests are trying to launch U-Boot's helloworld.efi embedded into > > > > + FIT images, in uncompressed or gzip compressed format. > > > > + > > > > + Additionally, a sample FDT blob is created and embedded into the above > > > > + mentioned FIT images, in uncompressed or gzip compressed format. > > > > + > > > > + For more details, see launch_efi(). > > > > + > > > > + The following test cases are currently defined and enabled: > > > > + - Launch uncompressed FIT EFI & internal FDT > > > > + - Launch uncompressed FIT EFI & FIT FDT > > > > + - Launch compressed FIT EFI & internal FDT > > > > + - Launch compressed FIT EFI & FIT FDT > > > > + """ > > > > + > > > > + def net_pre_commands(): > > > > + """Execute any commands required to enable network hardware. > > > > + > > > > + These commands are provided by the boardenv_* file; see the comment > > > > + at the beginning of this file. > > > > + """ > > > > + > > > > + init_usb = cons.config.env.get('env__net_uses_usb', False) > > > > + if init_usb: > > > > + cons.run_command('usb start') > > > > + > > > > + init_pci = cons.config.env.get('env__net_uses_pci', False) > > > > + if init_pci: > > > > + cons.run_command('pci enum') > > > > + > > > > + def net_dhcp(): > > > > + """Execute the dhcp command. > > > > + > > > > + The boardenv_* file may be used to enable/disable DHCP; see the > > > > + comment at the beginning of this file. > > > > + """ > > > > + > > > > + has_dhcp = cons.config.buildconfig.get('config_cmd_dhcp', 'n') == 'y' > > > > + if not has_dhcp: > > > > + cons.log.warning('CONFIG_CMD_DHCP != y: Skipping DHCP network setup') > > > > + return False > > > > + > > > > + test_dhcp = cons.config.env.get('env__net_dhcp_server', False) > > > > + if not test_dhcp: > > > > + cons.log.info('No DHCP server available') > > > > + return False > > > > + > > > > + cons.run_command('setenv autoload no') > > > > + output = cons.run_command('dhcp') > > > > + assert 'DHCP client bound to address ' in output > > > > + return True > > > > + > > > > + def net_setup_static(): > > > > + """Set up a static IP configuration. > > > > + > > > > + The configuration is provided by the boardenv_* file; see the comment at > > > > + the beginning of this file. > > > > + """ > > > > + > > > > + has_dhcp = cons.config.buildconfig.get('config_cmd_dhcp', 'n') == 'y' > > > > + if not has_dhcp: > > > > + cons.log.warning('CONFIG_NET != y: Skipping static network setup') > > > > + return False > > > > + > > > > + env_vars = cons.config.env.get('env__net_static_env_vars', None) > > > > + if not env_vars: > > > > + cons.log.info('No static network configuration is defined') > > > > + return False > > > > + > > > > + for (var, val) in env_vars: > > > > + cons.run_command('setenv %s %s' % (var, val)) > > > > + return True > > > > + > > > > + def make_fpath(fname): > > > > + """Compute the path of a given (temporary) file. > > > > + > > > > + Args: > > > > + fname: The name of a file within U-Boot build dir. > > > > + Return: > > > > + The computed file path. > > > > + """ > > > > + > > > > + return os.path.join(cons.config.build_dir, fname) > > > > + > > > > + def make_efi(fname, comp): > > > > + """Create an UEFI binary. > > > > + > > > > + This simply copies lib/efi_loader/helloworld.efi into U-Boot > > > > + build dir and, optionally, compresses the file using gzip. > > > > + > > > > + Args: > > > > + fname: The target file name within U-Boot build dir. > > > > + comp: Flag to enable gzip compression. > > > > + Return: > > > > + The path of the created file. > > > > + """ > > > > + > > > > + bin_path = make_fpath(fname) > > > > + util.run_and_log(cons, > > > > + ['cp', make_fpath('lib/efi_loader/helloworld.efi'), bin_path]) > > > > + if comp: > > > > + util.run_and_log(cons, ['gzip', '-f', bin_path]) > > > > + bin_path += '.gz' > > > > + return bin_path > > > > + > > > > + def make_dtb(fdt_type, comp): > > > > + """Create a sample DTB file. > > > > + > > > > + Creates a DTS file and compiles it to a DTB. > > > > + > > > > + Args: > > > > + fdt_type: The type of the FDT, i.e. internal, user. > > > > + comp: Flag to enable gzip compression. > > > > + Return: > > > > + The path of the created file. > > > > + """ > > > > + > > > > + # Generate resources referenced by FDT. > > > > + fdt_params = { > > > > + 'sys-arch': sys_arch, > > > > + 'fdt_type' : fdt_type, > > > > + } > > > > + > > > > + # Generate a test FDT file. > > > > + dts = make_fpath('test-efi-fit-%s.dts' % fdt_type) > > > > + with open(dts, 'w') as fd: > > > > + fd.write(fdt_data % fdt_params) > > > > + > > > > + # Build the test FDT. > > > > + dtb = make_fpath('test-efi-fit-%s.dtb' % fdt_type) > > > > + util.run_and_log(cons, ['dtc', '-I', 'dts', '-O', 'dtb', '-o', dtb, dts]) > > > > + if comp: > > > > + util.run_and_log(cons, ['gzip', '-f', dtb]) > > > > + dtb += '.gz' > > > > + return dtb > > > > + > > > > + def make_fit(comp): > > > > + """Create a sample FIT image. > > > > + > > > > + Runs 'mkimage' to create a FIT image within U-Boot build dir. > > > > + Args: > > > > + comp: Enable gzip compression for the EFI binary and FDT blob. > > > > + Return: > > > > + The path of the created file. > > > > + """ > > > > + > > > > + # Generate resources referenced by ITS. > > > > + its_params = { > > > > + 'sys-arch': sys_arch, > > > > + 'efi-bin': os.path.basename(make_efi('test-efi-fit-helloworld.efi', comp)), > > > > + 'kernel-type': 'kernel' if comp else 'kernel_noload', > > > > + 'efi-comp': 'gzip' if comp else 'none', > > > > + 'fdt-bin': os.path.basename(make_dtb('user', comp)), > > > > + 'fdt-comp': 'gzip' if comp else 'none', > > > > + } > > > > + > > > > + # Generate a test ITS file. > > > > + its_path = make_fpath('test-efi-fit-helloworld.its') > > > > + with open(its_path, 'w') as fd: > > > > + fd.write(its_data % its_params) > > > > + > > > > + # Build the test ITS. > > > > + fit_path = make_fpath('test-efi-fit-helloworld.fit') > > > > + util.run_and_log( > > > > + cons, [make_fpath('tools/mkimage'), '-f', its_path, fit_path]) > > > > + return fit_path > > > > + > > > > + def load_fit_from_host(f): > > > > + """Load the FIT image using the 'host load' command and return its address. > > > > + > > > > + Args: > > > > + f: Dictionary describing the FIT image to load, see env__efi_fit_test_file > > > > + in the comment at the beginning of this file. > > > > + Return: > > > > + The address where the file has been loaded. > > > > + """ > > > > + > > > > + addr = f.get('addr', None) > > > > + if not addr: > > > > + addr = u_boot_utils.find_ram_base(cons) > > > > > > %s/u_boot_utils/util/ > > > > > > > > > > + > > > > + output = cons.run_command( > > > > + 'host load hostfs - %s %s/%s' % (addr, f['dn'], f['fn'])) > > > > + expected_text = ' bytes read' > > > > + sz = f.get('size', None) > > > > + if sz: > > > > + expected_text = '%d' % sz + expected_text > > > > + assert(expected_text in output) > > > > + > > > > + return addr > > > > + > > > > + def load_fit_from_tftp(f): > > > > + """Load the FIT image using the tftpboot command and return its address. > > > > + > > > > + The file is downloaded from the TFTP server, its size and optionally its > > > > + CRC32 are validated. > > > > + > > > > + Args: > > > > + f: Dictionary describing the FIT image to load, see env__efi_fit_tftp_file > > > > + in the comment at the beginning of this file. > > > > + Return: > > > > + The address where the file has been loaded. > > > > + """ > > > > + > > > > + addr = f.get('addr', None) > > > > + if not addr: > > > > + addr = u_boot_utils.find_ram_base(cons) > > > > > > %s/u_boot_utils/util/ > > > > > > > > > > + > > > > + fn = f['fn'] > > > > + output = cons.run_command('tftpboot %s %s' % (addr, fn)) > > > > > > You have to pass addr as hexadecimal number. > > > > > > output = cons.run_command('tftpboot %x %s' % (addr, fn)) > > > > As explained before, I eventually converted 'addr' to string > > in order to allow values like '$kernel_addr_r' to be passed from the > > environment (this is what I actually used during my local tests). > > What I missed was the following: > > > > addr = '%x' % util.find_ram_base(cons) > > > > If this is not desired, I can revert to numbers only. > > I suggest to use the following instead of referring to $kernel_addr_r: > > with cons.log.section('FDT=%s;COMP=%s' % (enable_fdt, > enable_comp)): > if is_sandbox: > addr = util.find_ram_base(cons) > fit = { > 'dn': cons.config.build_dir, > 'addr': addr, > } > else: Actually this is already performed by 'load_fit_from_host()', all we need here is to avoid setting 'addr' in the dictionary. Thanks again for the whole feedback, I'll send a new patch series with all the changes agreed so far. > Best regards > > Heinrich > > > > > > > > + expected_text = 'Bytes transferred = ' > > > > + sz = f.get('size', None) > > > > + if sz: > > > > + expected_text += '%d' % sz > > > > + assert expected_text in output > > > > + > > > > + expected_crc = f.get('crc32', None) > > > > + if not expected_crc: > > > > + return addr > > > > + > > > > + if cons.config.buildconfig.get('config_cmd_crc32', 'n') != 'y': > > > > + return addr > > > > + > > > > + output = cons.run_command('crc32 $fileaddr $filesize') > > > > + assert expected_crc in output > > > > + > > > > + return addr > > > > + > > > > + def launch_efi(enable_fdt, enable_comp): > > > > + """Launch U-Boot's helloworld.efi binary from a FIT image. > > > > + > > > > + An external image file can be downloaded from TFTP, when related > > > > + details are provided by the boardenv_* file; see the comment at the > > > > + beginning of this file. > > > > + > > > > + If the size of the TFTP file is not provided within env__efi_fit_tftp_file, > > > > + the test image is generated automatically and placed in the TFTP root > > > > + directory specified via the 'dn' field. > > > > + > > > > + When running the tests on Sandbox, the image file is loaded directly > > > > + from the host filesystem. > > > > + > > > > + Once the load address is available on U-Boot console, the 'bootm' > > > > + command is executed for either 'config-efi-fdt' or 'config-efi-nofdt' > > > > + FIT configuration, depending on the value of the 'enable_fdt' function > > > > + argument. > > > > + > > > > + Eventually the 'Hello, world' message is expected in the U-Boot console. > > > > + > > > > + Args: > > > > + enable_fdt: Flag to enable using the FDT blob inside FIT image. > > > > + enable_comp: Flag to enable GZIP compression on EFI and FDT > > > > + generated content. > > > > + """ > > > > + > > > > + with cons.log.section('FDT=%s;COMP=%s' % (enable_fdt, enable_comp)): > > > > + if is_sandbox: > > > > + fit = { > > > > + 'dn': cons.config.build_dir, > > > > + 'addr': '${kernel_addr_r}', > > > > + } > > > > + else: > > > > + # Init networking. > > > > + net_pre_commands() > > > > + net_set_up = net_dhcp() > > > > + net_set_up = net_setup_static() or net_set_up > > > > + if not net_set_up: > > > > + pytest.skip('Network not initialized') > > > > + > > > > + fit = cons.config.env.get('env__efi_fit_tftp_file', None) > > > > + if not fit: > > > > + pytest.skip('No env__efi_fit_tftp_file binary specified in environment') > > > > + > > > > + sz = fit.get('size', None) > > > > + if not sz: > > > > + if not fit.get('dn', None): > > > > + pytest.skip('Neither "size", nor "dn" info provided in env__efi_fit_tftp_file') > > > > + > > > > + # Create test FIT image. > > > > + fit_path = make_fit(enable_comp) > > > > + fit['fn'] = os.path.basename(fit_path) > > > > + fit['size'] = os.path.getsize(fit_path) > > > > + > > > > + # Copy image to TFTP root directory. > > > > + if fit['dn'] != cons.config.build_dir: > > > > + util.run_and_log(cons, ['mv', '-f', fit_path, '%s/' % fit['dn']]) > > > > + > > > > + # Load FIT image. > > > > + addr = load_fit_from_host(fit) if is_sandbox else load_fit_from_tftp(fit) > > > > + > > > > + # Select boot configuration. > > > > + fit_config = 'config-efi-fdt' if enable_fdt else 'config-efi-nofdt' > > > > + > > > > + # Try booting. > > > > + cons.run_command( > > > > + 'bootm %s#%s' % (addr, fit_config), wait_for_prompt=False) > > > > > > You have to pass the address as hexadecimal number. > > > > > > 'bootm %x#%s' % (addr, fit_config), wait_for_prompt=False) > > > > Please see comments above. > > > > > Best regards > > > > > > Heinrich > > > > > > > + if enable_fdt: > > > > + cons.wait_for('Booting using the fdt blob') > > > > + cons.wait_for('Hello, world') > > > > + cons.wait_for('## Application terminated, r = 0') > > > > + cons.restart_uboot(); > > > > + > > > > + cons = u_boot_console > > > > + # Array slice removes leading/trailing quotes. > > > > + sys_arch = cons.config.buildconfig.get('config_sys_arch', '"sandbox"')[1:-1] > > > > + is_sandbox = sys_arch == 'sandbox' > > > > + > > > > + try: > > > > + if is_sandbox: > > > > + # Use our own device tree file, will be restored afterwards. > > > > + control_dtb = make_dtb('internal', False) > > > > + old_dtb = cons.config.dtb > > > > + cons.config.dtb = control_dtb > > > > + > > > > + # Run tests > > > > + # - fdt OFF, gzip OFF > > > > + launch_efi(False, False) > > > > + # - fdt ON, gzip OFF > > > > + launch_efi(True, False) > > > > + > > > > + if is_sandbox: > > > > + # - fdt OFF, gzip ON > > > > + launch_efi(False, True) > > > > + # - fdt ON, gzip ON > > > > + launch_efi(True, True) > > > > + > > > > + finally: > > > > + if is_sandbox: > > > > + # Go back to the original U-Boot with the correct dtb. > > > > + cons.config.dtb = old_dtb > > > > + cons.restart_uboot() > > > > > > > > > >