[U-Boot] [PATCH 8/8] test/py: add DFU test

[Lukasz Majewski <l.majewski@samsung.com>]

Hi Stephen,

> From: Stephen Warren <swarren@nvidia.com>
> 
> Add a test of DFU functionality to the Python test suite. The test
> starts DFU in U-Boot, waits for USB device enumeration on the host,
> executes dfu-util multiple times to test various transfer sizes, many
> of which trigger USB driver edge cases, and finally aborts the DFU
> command in U-Boot.
> 
> This test mirrors the functionality previously available via the shell
> scripts in test/dfu, and hence those are removed too.
> 
> Cc: Lukasz Majewski <l.majewski@majess.pl>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
>  test/dfu/README                  |  44 -------
>  test/dfu/dfu_gadget_test.sh      | 108 ----------------
>  test/dfu/dfu_gadget_test_init.sh |  45 -------
>  test/py/tests/test_dfu.py        | 257
> +++++++++++++++++++++++++++++++++++++++ 4 files changed, 257
> insertions(+), 197 deletions(-) delete mode 100644 test/dfu/README
>  delete mode 100755 test/dfu/dfu_gadget_test.sh
>  delete mode 100755 test/dfu/dfu_gadget_test_init.sh
>  create mode 100644 test/py/tests/test_dfu.py
> 
> diff --git a/test/dfu/README b/test/dfu/README
> deleted file mode 100644
> index 408d5594219a..000000000000
> --- a/test/dfu/README
> +++ /dev/null
> @@ -1,44 +0,0 @@
> -DFU TEST CASE DESCRIPTION:
> -
> -The prerequisites for running this script are assured by
> -dfu_gadget_test_init.sh, which is automatically invoked by
> dfu_gadget_test.sh. -In this file user is able to generate their own
> set of test files by altering -the default set of TEST_FILES_SIZES
> variable. -The dfu_gadget_test_init.sh would generate test images
> only if they are not -already generated.
> -
> -On the target device, environment variable "dfu_alt_info" must
> contain at -least:
> -
> -    dfu_test.bin fat 0 6;dfudummy.bin fat 0 6
> -
> -Depending on your device, you may need to replace "fat" with
> -"ext4", and "6" with the relevant partition number. For reference
> please -consult the config file for TRATS/TRATS2 devices
> -(../../include/configs/trats{2}.h)
> -
> -One can use fat, ext4 or any other supported file system supported
> by U-Boot. -These can be created by exporting storage devices via UMS
> (ums 0 mmc 0) and -using standard tools on host (like mkfs.ext4).
> -
> -Example usage:
> -1. On the target:
> -   setenv dfu_alt_info dfu_test.bin fat 0 6\;dfudummy.bin fat 0 6
> -   dfu 0 mmc 0
> -2. On the host:
> -   test/dfu/dfu_gadget_test.sh X Y [test file name] [usb device
> vendor:product]
> -   e.g. test/dfu/dfu_gadget_test.sh 0 1
> -   or
> -   e.g. test/dfu/dfu_gadget_test.sh 0 1 ./dat_960.img
> -   or
> -   e.g. test/dfu/dfu_gadget_test.sh 0 1 0451:d022
> -   or
> -   e.g. test/dfu/dfu_gadget_test.sh 0 1 ./dat_960.img 0451:d022
> -
> -... where X and Y are dfu_test.bin's and dfudummy.bin's alt setting
> numbers. -They can be obtained from dfu-util -l or $dfu_alt_info.
> -It is also possible to pass optional [test file name] to force the
> script to -test one particular file.
> -If many DFU devices are connected, it may be useful to filter on USB
> -vendor/product ID (0451:d022).
> -One can get them by running "lsusb" command on a host PC.
> diff --git a/test/dfu/dfu_gadget_test.sh b/test/dfu/dfu_gadget_test.sh
> deleted file mode 100755
> index 9c7942257b44..000000000000
> --- a/test/dfu/dfu_gadget_test.sh
> +++ /dev/null
> @@ -1,108 +0,0 @@
> -#! /bin/bash
> -
> -# Copyright (C) 2014 Samsung Electronics
> -# Lukasz Majewski <l.majewski@samsung.com>
> -#
> -# Script fixes, enhancements and testing:
> -# Stephen Warren <swarren@nvidia.com>
> -#
> -# DFU operation test script
> -#
> -# SPDX-License-Identifier:	GPL-2.0+
> -
> -set -e # any command return if not equal to zero
> -clear
> -
> -COLOUR_RED="\33[31m"
> -COLOUR_GREEN="\33[32m"
> -COLOUR_DEFAULT="\33[0m"
> -
> -DIR=./
> -SUFFIX=img
> -RCV_DIR=rcv/
> -LOG_FILE=./log/log-`date +%d-%m-%Y_%H-%M-%S`
> -
> -cd `dirname $0`
> -./dfu_gadget_test_init.sh
> -
> -cleanup () {
> -    rm -rf $DIR$RCV_DIR
> -}
> -
> -die () {
> -	printf "   $COLOUR_RED FAILED $COLOUR_DEFAULT \n"
> -	cleanup
> -	exit 1
> -}
> -
> -calculate_md5sum () {
> -    MD5SUM=`md5sum $1`
> -    MD5SUM=`echo $MD5SUM | cut -d ' ' -f1`
> -    echo "md5sum:"$MD5SUM
> -}
> -
> -dfu_test_file () {
> -    printf "$COLOUR_GREEN
> =========================================================================================
> $COLOUR_DEFAULT\n"
> -    printf "File:$COLOUR_GREEN %s $COLOUR_DEFAULT\n" $1
> -
> -    dfu-util $USB_DEV -D $1 -a $TARGET_ALT_SETTING >> $LOG_FILE 2>&1
> || die $? -
> -    echo -n "TX: "
> -    calculate_md5sum $1
> -
> -    MD5_TX=$MD5SUM
> -
> -    dfu-util $USB_DEV -D ${DIR}/dfudummy.bin -a
> $TARGET_ALT_SETTING_B >> $LOG_FILE 2>&1 || die $? -
> -    N_FILE=$DIR$RCV_DIR${1:2}"_rcv"
> -
> -    dfu-util $USB_DEV -U $N_FILE -a $TARGET_ALT_SETTING >> $LOG_FILE
> 2>&1 || die $? -
> -    echo -n "RX: "
> -    calculate_md5sum $N_FILE
> -    MD5_RX=$MD5SUM
> -
> -    if [ "$MD5_TX" == "$MD5_RX" ]; then
> -	printf "   $COLOUR_GREEN -------> OK $COLOUR_DEFAULT \n"
> -    else
> -	printf "   $COLOUR_RED -------> FAILED $COLOUR_DEFAULT \n"
> -	cleanup
> -	exit 1
> -    fi
> -
> -}
> -
> -printf
> "$COLOUR_GREEN=========================================================================================
> $COLOUR_DEFAULT\n" -echo "DFU EP0 transmission test program" -echo
> "Trouble shoot -> disable DBG (even the KERN_DEBUG) in the UDC
> driver" -echo "@ -> TRATS2 # dfu 0 mmc 0" -cleanup
> -mkdir -p $DIR$RCV_DIR
> -touch $LOG_FILE
> -
> -if [ $# -eq 0 ]
> -then
> -	printf "   $COLOUR_RED Please pass alt setting number!!
> $COLOUR_DEFAULT \n"
> -	exit 0
> -fi
> -
> -TARGET_ALT_SETTING=$1
> -TARGET_ALT_SETTING_B=$2
> -
> -file=$3
> -[[ $3 == *':'* ]] && USB_DEV="-d $3" && file=""
> -[ $# -eq 4 ] && USB_DEV="-d $4"
> -
> -if [ -n "$file" ]
> -then
> -	dfu_test_file $file
> -else
> -	for f in $DIR*.$SUFFIX
> -	do
> -	    dfu_test_file $f
> -	done
> -fi
> -
> -cleanup
> -
> -exit 0
> diff --git a/test/dfu/dfu_gadget_test_init.sh
> b/test/dfu/dfu_gadget_test_init.sh deleted file mode 100755
> index 640628eecb7a..000000000000
> --- a/test/dfu/dfu_gadget_test_init.sh
> +++ /dev/null
> @@ -1,45 +0,0 @@
> -#! /bin/bash
> -
> -# Copyright (C) 2014 Samsung Electronics
> -# Lukasz Majewski <l.majewski@samsung.com>
> -#
> -# Script fixes, enhancements and testing:
> -# Stephen Warren <swarren@nvidia.com>
> -#
> -# Script for test files generation
> -#
> -# SPDX-License-Identifier:	GPL-2.0+
> -
> -set -e # any command return if not equal to zero
> -clear
> -
> -COLOUR_RED="\33[31m"
> -COLOUR_GREEN="\33[32m"
> -COLOUR_DEFAULT="\33[0m"
> -
> -LOG_DIR="./log"
> -
> -if [ $# -eq 0 ]; then
> -    TEST_FILES_SIZES="63 64 65 127 128 129 4095 4096 4097 959 960
> 961 1048575 1048576 8M" -else
> -    TEST_FILES_SIZES=$@
> -fi
> -
> -printf "Init script for generating data necessary for DFU test
> script" -
> -if [ ! -d $LOG_DIR ]; then
> -    `mkdir $LOG_DIR`
> -fi
> -
> -for size in $TEST_FILES_SIZES
> -do
> -    FILE="./dat_$size.img"
> -    if [ ! -f $FILE ]; then
> -	dd if=/dev/urandom of="./dat_$size.img" bs=$size count=1
> > /dev/null 2>&1 || exit $?
> -    fi
> -done
> -dd if=/dev/urandom of="./dfudummy.bin" bs=1024 count=1 > /dev/null
> 2>&1 || exit $? -
> -printf "$COLOUR_GREEN OK $COLOUR_DEFAULT \n"
> -
> -exit 0
> diff --git a/test/py/tests/test_dfu.py b/test/py/tests/test_dfu.py
> new file mode 100644
> index 000000000000..e86ea9a1887c
> --- /dev/null
> +++ b/test/py/tests/test_dfu.py
> @@ -0,0 +1,257 @@
> +# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
> +#
> +# SPDX-License-Identifier: GPL-2.0
> +
> +# Test U-Boot's "ums" command. At present, this test only ensures
> that a UMS +# device can be enumerated by the host/test machine. In
> the future, this test +# should be enhanced to validate disk IO.

Please update this comment to regard dfu, not ums.

> +
> +import os
> +import os.path
> +import pytest
> +import u_boot_utils
> +
> +'''
> +Note: This test relies on:
> +
> +a) boardenv_* to contain configuration values to define which USB
> ports are +available for testing. Without this, this test will be
> automatically skipped. +For example:
> +
> +env__usb_dev_ports = (
> +    {
> +        "tgt_usb_ctlr": "0",
> +        "host_usb_dev_node": "/dev/usbdev-p2371-2180",
> +        # This parameter is optional /if/ you only have a single
> board
> +        # attached to your host at a time.
> +        "host_usb_port_path": "3-13",
> +    },
> +)
> +
> +env__dfu_configs = (
> +    # eMMC, partition 1
> +    {
> +        "alt_info": "/dfu_test.bin ext4 0 1;/dfu_dummy.bin ext4 0 1",
> +        "cmd_params": "mmc 0",
> +    },
> +)
> +b) udev rules to set permissions on devices nodes, so that sudo is
> not +required. For example:
> +
> +ACTION=="add", SUBSYSTEM=="block", SUBSYSTEMS=="usb",
> KERNELS=="3-13", MODE:="666" +
> +(You may wish to change the group ID instead of setting the
> permissions wide +open. All that matters is that the user ID running
> the test can access the +device.)
> +'''
> +
> +# The set of file sizes to test. These values trigger various
> edge-cases such +# as one less than, equal to, and one greater than
> typical USB max packet +# sizes, and similar boundary conditions.
> +test_sizes = (
> +    64 - 1,
> +    64,
> +    64 + 1,
> +    128 - 1,
> +    128,
> +    128 + 1,
> +    960 - 1,
> +    960,
> +    960 + 1,
> +    4096 - 1,
> +    4096,
> +    4096 + 1,
> +    1024 * 1024 - 1,
> +    1024 * 1024,
> +    8 * 1024 * 1024,
> +)
> +
> +first_usb_dev_port = None
> +
> + at pytest.mark.buildconfigspec('cmd_dfu')
> +def test_dfu(u_boot_console, env__usb_dev_port, env__dfu_config):
> +    '''Test DFU functionality, using numerous file sizes.
> +
> +    Args:
> +        u_boot_console: A U-Boot console connection.
> +        env__usb_dev_port: The single USB device-mode port
> specification on
> +            which to run the test.
> +        env__dfu_config: The single DFU (memory region)
> configuration on which
> +            to run the test.
> +
> +    Returns:
> +        Nothing.
> +    '''
> +
> +    def start_dfu():
> +        '''Start U-Boot's dfu shell command.
> +
> +        This also waits for the host-side USB enumeration process to
> complete. +
> +        Args:
> +            None.
> +
> +        Returns:
> +            Nothing.
> +        '''
> +
> +        u_boot_console.log.action(
> +            'Starting long-running U-Boot dfu shell command')
> +
> +        cmd = 'setenv dfu_alt_info "%s"' %
> env__dfu_config['alt_info']
> +        u_boot_console.run_command(cmd)
> +
> +        cmd = 'dfu 0 ' + env__dfu_config['cmd_params']
> +        u_boot_console.run_command(cmd, wait_for_prompt=False)
> +        u_boot_console.log.action('Waiting for DFU USB device to
> appear')
> +        fh = u_boot_utils.wait_until_open_succeeds(
> +            env__usb_dev_port['host_usb_dev_node'])
> +        fh.close()
> +
> +    def stop_dfu(ignore_errors):
> +        '''Stop U-Boot's dfu shell command from executing.
> +
> +        This also waits for the host-side USB de-enumeration process
> to
> +        complete.
> +
> +        Args:
> +            ignore_errors: Ignore any errors. This is useful if an
> error has
> +                already been detected, and the code is performing
> best-effort
> +                cleanup. In this case, we do not want to mask the
> original
> +                error by "honoring" any new errors.
> +
> +        Returns:
> +            Nothing.
> +        '''
> +
> +        try:
> +            u_boot_console.log.action(
> +                'Stopping long-running U-Boot dfu shell command')
> +            u_boot_console.ctrlc()
> +            u_boot_console.log.action(
> +                'Waiting for DFU USB device to disappear')
> +            u_boot_utils.wait_until_file_open_fails(
> +                env__usb_dev_port['host_usb_dev_node'],
> ignore_errors)
> +        except:
> +            if not ignore_errors:
> +                raise
> +
> +    def run_dfu_util(alt_setting, fn, up_dn_load_arg):
> +        '''Invoke dfu-util on the host.
> +
> +        Args:
> +            alt_setting: The DFU "alternate setting" identifier to
> interact
> +                with.
> +            fn: The host-side file name to transfer.
> +            up_dn_load_arg: '-U' or '-D' depending on whether a DFU
> upload or
> +                download operation should be performed.
> +
> +        Returns:
> +            Nothing.
> +        '''
> +
> +        cmd = ['dfu-util', '-a', str(alt_setting), up_dn_load_arg,
> fn]
> +        if 'host_usb_port_path' in env__usb_dev_port:
> +            cmd += ['-p', env__usb_dev_port['host_usb_port_path']]
> +        u_boot_utils.run_and_log(u_boot_console, cmd)
> +        u_boot_console.wait_for('Ctrl+C to exit ...')
> +
> +    def dfu_write(alt_setting, fn):
> +        '''Write a file to the target board using DFU.
> +
> +        Args:
> +            alt_setting: The DFU "alternate setting" identifier to
> interact
> +                with.
> +            fn: The host-side file name to transfer.
> +
> +        Returns:
> +            Nothing.
> +        '''
> +
> +        run_dfu_util(alt_setting, fn, '-D')
> +
> +    def dfu_read(alt_setting, fn):
> +        '''Read a file from the target board using DFU.
> +
> +        Args:
> +            alt_setting: The DFU "alternate setting" identifier to
> interact
> +                with.
> +            fn: The host-side file name to transfer.
> +
> +        Returns:
> +            Nothing.
> +        '''
> +
> +        # dfu-util fails reads/uploads if the host file already
> exists
> +        if os.path.exists(fn):
> +            os.remove(fn)
> +        run_dfu_util(alt_setting, fn, '-U')
> +
> +    def dfu_write_read_check(size):
> +        '''Test DFU transfers of a specific size of data
> +
> +        This function first writes data to the board then reads it
> back and
> +        compares the written and read back data. Measures are taken
> to avoid
> +        certain types of false positives.
> +
> +        Args:
> +            size: The data size to test.
> +
> +        Returns:
> +            Nothing.
> +        '''
> +
> +        test_f = u_boot_utils.PersistentRandomFile(u_boot_console,
> +            'dfu_%d.bin' % size, size)
> +        readback_fn = u_boot_console.config.result_dir +
> '/dfu_readback.bin' +
> +        u_boot_console.log.action('Writing test data to DFU primary
> ' +
> +            'altsetting')
> +        dfu_write(0, test_f.abs_fn)
> +
> +        u_boot_console.log.action('Writing dummy data to DFU
> secondary ' +
> +            'altsetting to clear DFU buffers')
> +        dfu_write(1, dummy_f.abs_fn)
> +
> +        u_boot_console.log.action('Reading DFU primary altsetting
> for ' +
> +            'comparison')
> +        dfu_read(0, readback_fn)
> +
> +        u_boot_console.log.action('Comparing written and read data')
> +        written_hash = test_f.content_hash
> +        read_back_hash = u_boot_utils.md5sum_file(readback_fn, size)
> +        assert(written_hash == read_back_hash)
> +
> +    # This test may be executed against multiple USB ports. The test
> takes a
> +    # long time, so we don't want to do the whole thing each time.
> Instead,
> +    # execute the full test on the first USB port, and perform a
> very limited
> +    # test on other ports. In the limited case, we solely validate
> that the
> +    # host PC can enumerate the U-Boot USB device.
> +    global first_usb_dev_port
> +    if not first_usb_dev_port:
> +        first_usb_dev_port = env__usb_dev_port
> +    if env__usb_dev_port == first_usb_dev_port:
> +        sizes = test_sizes
> +    else:
> +        sizes = []
> +
> +    dummy_f = u_boot_utils.PersistentRandomFile(u_boot_console,
> +        'dfu_dummy.bin', 1024)
> +
> +    ignore_cleanup_errors = True
> +    try:
> +        start_dfu()
> +
> +        u_boot_console.log.action(
> +            'Overwriting DFU primary altsetting with dummy data')
> +        dfu_write(0, dummy_f.abs_fn)
> +
> +        for size in sizes:
> +            with u_boot_console.log.section("Data size %d" % size):
> +                dfu_write_read_check(size)
> +                # Make the status of each sub-test obvious. If the
> test didn't
> +                # pass, an exception was thrown so this code isn't
> executed.
> +                u_boot_console.log.status_pass('OK')
> +        ignore_cleanup_errors = False
> +    finally:
> +        stop_dfu(ignore_cleanup_errors)

Acked-by: Lukasz Majewski <l.majewski@samsung.com>

Great work Stephen, Thanks !

-- 
Best regards,

Lukasz Majewski

Samsung R&D Institute Poland (SRPOL) | Linux Platform Group