[U-Boot] [PATCH 14/14] test: Convert the vboot test to test/py

Teddy Reed
2016-07-03T21:38:48Z

Hi Simon,

On Sun, Jul 3, 2016 at 8:40 AM, Simon Glass <sjg@chromium.org> wrote:
> Now that we have a suitable test framework we should move all tests into it.
> The vboot test is a suitable candidate. Rewrite it in Python and move the
> data files into an appropriate directory.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
>
> test/README | 1 -
> test/py/tests/test_vboot.py | 185 ++++++++++++++++++++++
> test/{ => py/tests}/vboot/sandbox-kernel.dts | 0
> test/{ => py/tests}/vboot/sandbox-u-boot.dts | 0
> test/{ => py/tests}/vboot/sign-configs-sha1.its | 0
> test/{ => py/tests}/vboot/sign-configs-sha256.its | 0
> test/{ => py/tests}/vboot/sign-images-sha1.its | 0
> test/{ => py/tests}/vboot/sign-images-sha256.its | 0
> test/vboot/.gitignore | 3 -
> test/vboot/vboot_test.sh | 151 ------------------
> 10 files changed, 185 insertions(+), 155 deletions(-)
> create mode 100644 test/py/tests/test_vboot.py
> rename test/{ => py/tests}/vboot/sandbox-kernel.dts (100%)
> rename test/{ => py/tests}/vboot/sandbox-u-boot.dts (100%)
> rename test/{ => py/tests}/vboot/sign-configs-sha1.its (100%)
> rename test/{ => py/tests}/vboot/sign-configs-sha256.its (100%)
> rename test/{ => py/tests}/vboot/sign-images-sha1.its (100%)
> rename test/{ => py/tests}/vboot/sign-images-sha256.its (100%)
> delete mode 100644 test/vboot/.gitignore
> delete mode 100755 test/vboot/vboot_test.sh
>
> diff --git a/test/README b/test/README
> index 6cfee05..ee55972 100644
> --- a/test/README
> +++ b/test/README
> @@ -58,7 +58,6 @@ There are several ad-hoc tests which run outside the pytest environment:
> test/image - FIT and lagacy image tests (shell script and Python)
> test/stdint - A test that stdint.h can be used in U-Boot (shell script)
> trace - Test for the tracing feature (shell script)
> - vboot - Test for verified boot (shell script)
>
> The above should be converted to run as part of the pytest suite.
>
> diff --git a/test/py/tests/test_vboot.py b/test/py/tests/test_vboot.py
> new file mode 100644
> index 0000000..c779895
> --- /dev/null
> +++ b/test/py/tests/test_vboot.py
> @@ -0,0 +1,185 @@
> +# Copyright (c) 2013, Google Inc.
> +#
> +# SPDX-License-Identifier: GPL-2.0+
> +#
> +# U-Boot Verified Boot Test
> +
> +"""
> +This tests verified boot in the following ways:
> +
> +For image verification:
> +- Create FIT (unsigned) with mkimage
> +- Check that verification shows that no keys are verified
> +- Sign image
> +- Check that verification shows that a key is now verified
> +
> +For configuration verification:
> +- Corrupt signature and check for failure
> +- Create FIT (with unsigned configuration) with mkimage
> +- Check that image veriication works

verification

> +- Sign the FIT and mark the key as 'required' for verification
> +- Check that image verification works
> +- Corrupt the signature
> +- Check that image verification no-longer works
> +
> +Tests run with both SHA1 and SHA256 hashing.
> +"""
> +
> +import pytest
> +import sys
> +import u_boot_utils as util
> +
> + at pytest.mark.buildconfigspec('fit_signature')
> +def test_vboot(u_boot_console):
> + """Test verified boot signing with mkimage and verification with 'bootm'.
> +
> + This works using sandbox only as it needs to update the device tree used
> + by U-Boot to hold public keys from the signing process.
> +
> + The SHA1 and SHA256 tests are combined into a single test since the
> + key-generation process is quite slow and we want to avoid doing it twice.
> + """
> + def dtc(dts):
> + """Run the device-tree compiler to compile a .dts file

In a few other places it reads: "device tree" without a dash. ;)

> +
> + The output file will be the same as the input file but with a .dtb
> + extension.
> +
> + Args:
> + dts: Device tree file to compile.
> + """
> + dtb = dts.replace('.dts', '.dtb')
> + util.cmd(cons, 'dtc %s %s%s -O dtb '
> + '-o %s%s' % (dtc_args, datadir, dts, tmpdir, dtb))
> +
> + def run_bootm(test_type, expect_string):
> + """Run a 'bootm' command U-Boot.
> +
> + This always starts a fresh U-Boot instance since the device tree may
> + contain a new public key.
> +
> + Args:
> + test_type: A string identifying the test type
> + expect_string: A string which is expected in the output

nit, punctuation.

> + """
> + cons.cleanup_spawn()
> + cons.ensure_spawned()
> + cons.log.action('%s: Test Verified Boot Run: %s' % (algo, test_type))
> + output = cons.run_command_list(
> + ['sb load hostfs - 100 %stest.fit' % tmpdir,
> + 'fdt addr 100',
> + 'bootm 100'])
> + assert(expect_string in output)
> +
> + def make_fit(its):
> + """Make a new FIT from the .its source file
> +
> + This runs 'mkimage -f' to create a new FIT.
> +
> + Args:
> + its: Filename containing .its source

nit, same

> + """
> + util.run_and_log(cons, [mkimage, '-D', dtc_args, '-f',
> + '%s%s' % (datadir, its), fit])
> +
> + def sign_fit():
> + """Sign the FIT
> +
> + Signs the FIT and writes the signature into it. It also writes the
> + public key into the dtb.
> + """
> + cons.log.action('%s: Sign images' % algo)
> + util.run_and_log(cons, [mkimage, '-F', '-k', tmpdir, '-K', dtb,
> + '-r', fit])
> +
> + def test_with_algo(sha):

nit, maybe sha_algo?

> + """Test verified boot with the given hash algorithm

nit, same

> +
> + This is the main part of the test code. The same procedure is followed
> + for both hashing algorithms.
> +
> + Args:
> + sha: Either 'sha1' or 'sha256', to select the algorithm to use
> + """
> + global algo
> +
> + algo = sha
> +
> + # Compile our device tree files for kernel and U-Boot
> + dtc('sandbox-kernel.dts')
> + dtc('sandbox-u-boot.dts')
> +
> + # Build the FIT, but don't sign anything yet
> + cons.log.action('%s: Test FIT with signed images' % algo)
> + make_fit('sign-images-%s.its' % algo)
> + run_bootm('unsigned images', 'dev-')
> +
> + # Sign images with our dev keys
> + sign_fit()
> + run_bootm('signed images', 'dev+')
> +
> + # Create a fresh .dtb without the public keys
> + dtc('sandbox-u-boot.dts')
> +
> + cons.log.action('%s: Test FIT with signed configuration' % algo)
> + make_fit('sign-configs-%s.its' % algo)
> + run_bootm('unsigned config', '%s+ OK' % algo)
> +
> + # Sign images with our dev keys
> + sign_fit()
> + run_bootm('signed config', 'dev+')
> +
> + cons.log.action('%s: Check signed config on the host' % algo)
> +
> + util.run_and_log(cons, [fit_check_sign, '-f', fit, '-k', tmpdir,
> + '-k', dtb])
> +
> + # Increment the first byte of the signature, which should cause failure
> + sig = util.cmd(cons, 'fdtget -t bx %s %s value' % (fit, sig_node))
> + byte_list = sig.split()
> + byte = int(byte_list[0], 16)
> + byte_list = ['%x' % (byte + 1)] + byte_list[1:]
> + sig = ' '.join(byte_list)
> + util.cmd(cons, 'fdtput -t bx %s %s value %s' % (fit, sig_node, sig))
> +
> + run_bootm('Signed config with bad hash', 'Bad Data Hash')
> +
> + cons.log.action('%s: Check bad config on the host' % algo)
> + util.run_and_log_expect_exception(cons, [fit_check_sign, '-f', fit,
> + '-k', dtb], 1, 'Failed to verify required signature')
> +
> + cons = u_boot_console
> + tmpdir = cons.config.result_dir + '/'
> + tmp = tmpdir + 'vboot.tmp'

Is there a need to clean these up afterward?

Python's tempfile might be helpful, you can supply the result_dir as
the directory prefix.

> + datadir = 'test/py/tests/vboot/'
> + fit = '%stest.fit' % tmpdir
> + mkimage = cons.config.build_dir + '/tools/mkimage'
> + fit_check_sign = cons.config.build_dir + '/tools/fit_check_sign'
> + dtc_args = '-I dts -O dtb -i %s' % tmpdir
> + dtb = '%ssandbox-u-boot.dtb' % tmpdir
> + sig_node = '/configurations/conf at 1/signature at 1'

If these variables are used throughout the tests like globals, should
they be DATADIR, MKIMAGE, etc?

> +
> + # Create an RSA key pair
> + public_exponent = 65537
> + util.cmd(cons, 'openssl genpkey -algorithm RSA -out %sdev.key '
> + '-pkeyopt rsa_keygen_bits:2048 '
> + '-pkeyopt rsa_keygen_pubexp:%d '
> + '2>/dev/null' % (tmpdir, public_exponent))
> +
> + # Create a certificate containing the public key
> + util.cmd(cons, 'openssl req -batch -new -x509 -key %sdev.key -out '
> + '%sdev.crt' % (tmpdir, tmpdir))
> +
> + # Create a number kernel image with zeroes
> + with open('%stest-kernel.bin' % tmpdir, 'w') as fd:
> + fd.write(5000 * chr(0))
> +

Any need to clean these up afterward or place them into
test-run-unique directories? Is the expectation that subsequent tests
will overwrite existing test data (also that vboot tests cannot run
concurrently)?

> + try:
> + # We need to use our own device tree file. Remember to restore it
> + # afterwards.
> + old_dtb = cons.config.dtb
> + cons.config.dtb = dtb
> + test_with_algo('sha1')
> + test_with_algo('sha256')
> + finally:
> + cons.config.dtb = old_dtb
> diff --git a/test/vboot/sandbox-kernel.dts b/test/py/tests/vboot/sandbox-kernel.dts
> similarity index 100%
> rename from test/vboot/sandbox-kernel.dts
> rename to test/py/tests/vboot/sandbox-kernel.dts
> diff --git a/test/vboot/sandbox-u-boot.dts b/test/py/tests/vboot/sandbox-u-boot.dts
> similarity index 100%
> rename from test/vboot/sandbox-u-boot.dts
> rename to test/py/tests/vboot/sandbox-u-boot.dts
> diff --git a/test/vboot/sign-configs-sha1.its b/test/py/tests/vboot/sign-configs-sha1.its
> similarity index 100%
> rename from test/vboot/sign-configs-sha1.its
> rename to test/py/tests/vboot/sign-configs-sha1.its
> diff --git a/test/vboot/sign-configs-sha256.its b/test/py/tests/vboot/sign-configs-sha256.its
> similarity index 100%
> rename from test/vboot/sign-configs-sha256.its
> rename to test/py/tests/vboot/sign-configs-sha256.its
> diff --git a/test/vboot/sign-images-sha1.its b/test/py/tests/vboot/sign-images-sha1.its
> similarity index 100%
> rename from test/vboot/sign-images-sha1.its
> rename to test/py/tests/vboot/sign-images-sha1.its
> diff --git a/test/vboot/sign-images-sha256.its b/test/py/tests/vboot/sign-images-sha256.its
> similarity index 100%
> rename from test/vboot/sign-images-sha256.its
> rename to test/py/tests/vboot/sign-images-sha256.its
> diff --git a/test/vboot/.gitignore b/test/vboot/.gitignore
> deleted file mode 100644
> index 4631242..0000000
> --- a/test/vboot/.gitignore
> +++ /dev/null
> @@ -1,3 +0,0 @@
> -/*.dtb
> -/test.fit
> -/dev-keys
> diff --git a/test/vboot/vboot_test.sh b/test/vboot/vboot_test.sh
> deleted file mode 100755
> index 6d7abb8..0000000
> --- a/test/vboot/vboot_test.sh
> +++ /dev/null
> @@ -1,151 +0,0 @@
> -#!/bin/bash
> -#
> -# Copyright (c) 2013, Google Inc.
> -#
> -# Simple Verified Boot Test Script
> -#
> -# SPDX-License-Identifier: GPL-2.0+
> -
> -set -e
> -
> -# Run U-Boot and report the result
> -# Args:
> -# $1: Test message
> -run_uboot() {
> - echo -n "Test Verified Boot Run: $1: "
> - ${uboot} -d sandbox-u-boot.dtb >${tmp} -c '
> -sb load hostfs - 100 test.fit;
> -fdt addr 100;
> -bootm 100;
> -reset'
> - if ! grep -q "$2" ${tmp}; then
> - echo
> - echo "Verified boot key check failed, output follows:"
> - cat ${tmp}
> - false
> - else
> - echo "OK"
> - fi
> -}
> -
> -echo "Simple Verified Boot Test"
> -echo "========================="
> -echo
> -echo "Please see doc/uImage.FIT/verified-boot.txt for more information"
> -echo
> -
> -err=0
> -tmp=/tmp/vboot_test.$$
> -
> -dir=$(dirname $0)
> -
> -if [ -z ${O} ]; then
> - O=.
> -fi
> -O=$(readlink -f ${O})
> -
> -dtc="-I dts -O dtb -p 2000"
> -uboot="${O}/u-boot"
> -mkimage="${O}/tools/mkimage"
> -fit_check_sign="${O}/tools/fit_check_sign"
> -keys="${dir}/dev-keys"
> -echo ${mkimage} -D "${dtc}"
> -
> -echo "Build keys"
> -mkdir -p ${keys}
> -
> -PUBLIC_EXPONENT=${1}
> -
> -if [ -z "${PUBLIC_EXPONENT}" ]; then
> - PUBLIC_EXPONENT=65537
> -fi
> -
> -# Create an RSA key pair
> -openssl genpkey -algorithm RSA -out ${keys}/dev.key \
> - -pkeyopt rsa_keygen_bits:2048 \
> - -pkeyopt rsa_keygen_pubexp:${PUBLIC_EXPONENT} 2>/dev/null
> -
> -# Create a certificate containing the public key
> -openssl req -batch -new -x509 -key ${keys}/dev.key -out ${keys}/dev.crt
> -
> -pushd ${dir} >/dev/null
> -
> -function do_test {
> - echo do $sha test
> - # Compile our device tree files for kernel and U-Boot
> - dtc -p 0x1000 sandbox-kernel.dts -O dtb -o sandbox-kernel.dtb
> - dtc -p 0x1000 sandbox-u-boot.dts -O dtb -o sandbox-u-boot.dtb
> -
> - # Create a number kernel image with zeroes
> - head -c 5000 /dev/zero >test-kernel.bin
> -
> - # Build the FIT, but don't sign anything yet
> - echo Build FIT with signed images
> - ${mkimage} -D "${dtc}" -f sign-images-$sha.its test.fit >${tmp}
> -
> - run_uboot "unsigned signatures:" "dev-"
> -
> - # Sign images with our dev keys
> - echo Sign images
> - ${mkimage} -D "${dtc}" -F -k dev-keys -K sandbox-u-boot.dtb \
> - -r test.fit >${tmp}
> -
> - run_uboot "signed images" "dev+"
> -
> -
> - # Create a fresh .dtb without the public keys
> - dtc -p 0x1000 sandbox-u-boot.dts -O dtb -o sandbox-u-boot.dtb
> -
> - echo Build FIT with signed configuration
> - ${mkimage} -D "${dtc}" -f sign-configs-$sha.its test.fit >${tmp}
> -
> - run_uboot "unsigned config" $sha"+ OK"
> -
> - # Sign images with our dev keys
> - echo Sign images
> - ${mkimage} -D "${dtc}" -F -k dev-keys -K sandbox-u-boot.dtb \
> - -r test.fit >${tmp}
> -
> - run_uboot "signed config" "dev+"
> -
> - echo check signed config on the host
> - if ! ${fit_check_sign} -f test.fit -k sandbox-u-boot.dtb >${tmp}; then
> - echo
> - echo "Verified boot key check on host failed, output follows:"
> - cat ${tmp}
> - false
> - else
> - if ! grep -q "dev+" ${tmp}; then
> - echo
> - echo "Verified boot key check failed, output follows:"
> - cat ${tmp}
> - false
> - else
> - echo "OK"
> - fi
> - fi
> -
> - run_uboot "signed config" "dev+"
> -
> - # Increment the first byte of the signature, which should cause failure
> - sig=$(fdtget -t bx test.fit /configurations/conf at 1/signature at 1 value)
> - newbyte=$(printf %x $((0x${sig:0:2} + 1)))
> - sig="${newbyte} ${sig:2}"
> - fdtput -t bx test.fit /configurations/conf at 1/signature at 1 value ${sig}
> -
> - run_uboot "signed config with bad hash" "Bad Data Hash"
> -}
> -
> -sha=sha1
> -do_test
> -sha=sha256
> -do_test
> -
> -popd >/dev/null
> -
> -echo
> -if ${ok}; then
> - echo "Test passed"
> -else
> - echo "Test failed"
> -fi
> --
> 2.8.0.rc3.226.g39d4020
>
> _______________________________________________
> U-Boot mailing list
> U-Boot at lists.denx.de
> http://lists.denx.de/mailman/listinfo/u-boot

Thanks for this refactor! If the comments related to the sh to Python
are too nit-picky we can certainly change and expand the test harness
within additional patches later.

--
Teddy Reed V