[U-Boot] [PATCH v2 0/7] test: Include Python tools in test coverage

[U-Boot] [PATCH v2 0/7] test: Include Python tools in test coverage

[Simon Glass]

At present there are a number of tools in U-Boot which have tests which
are not widely used:

  patman   - prepare, check and send patches
  buildman - multi-threaded multi-commit builder
  dtoc     - convert device tree file to C
  binman   - produce firmware images

In fact many tests are broken and this has not been noticed.

This series fixes up the tests and updates 'make tests' to run them. This
should help ensure that the tests stay current.

Note: This series is available at u-boot-dm/binman-working. It requires
Masahiro's pylibfdt series.

Changes in v2:
- Add new patch to correct operation when tests pass
- Adjust PYTHONPATH for new locations
- Adjust PYTHONPATH for new locations
- Drop patches already applied
- Run tests with travis-ci also
- Drop RFC tag from patch for binman code-coverage tests

Simon Glass (7):
  test: Correct operation when tests pass
  test: Run binman tests
  test: Run patman tests
  test: Run buildman tests
  test: Run dtoc tests
  travis.yml: Run tests for tools
  binman: Run code coverage tests

 .travis.yml | 15 +++++++++++++++
 test/run    | 16 +++++++++++++++-
 2 files changed, 30 insertions(+), 1 deletion(-)

-- 
2.15.0.417.g466bffb3ac-goog

[U-Boot] [PATCH v2 1/7] test: Correct operation when tests pass

[Simon Glass]

When tests pass an error message is printed because of a variable that is
not initialised. Fix this.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Add new patch to correct operation when tests pass

 test/run | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/test/run b/test/run
index caee4f83f2c..56160977a3f 100755
--- a/test/run
+++ b/test/run
@@ -3,9 +3,10 @@
 run_test() {
 	$@
 	[ $? -ne 0 ] && result=$((result+1))
-	echo "result $result"
 }
 
+result=0
+
 # Run all tests that the standard sandbox build can support
 run_test ./test/py/test.py --bd sandbox --build
 
-- 
2.15.0.417.g466bffb3ac-goog

[U-Boot] [U-Boot, v2, 1/7] test: Correct operation when tests pass

[Tom Rini]

On Sun, Nov 26, 2017 at 08:25:55PM -0700, Simon Glass wrote:

> When tests pass an error message is printed because of a variable that is
> not initialised. Fix this.
> 
> Signed-off-by: Simon Glass <sjg@chromium.org>

Applied to u-boot/master, thanks!

-- 
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20171128/54e11a35/attachment.sig>

[U-Boot] [PATCH v2 2/7] test: Run binman tests

[Simon Glass]

Update the test script to run the binman tests also.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Adjust PYTHONPATH for new locations

 test/run | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/test/run b/test/run
index 56160977a3f..a12ea31bf23 100755
--- a/test/run
+++ b/test/run
@@ -17,6 +17,9 @@ run_test ./test/py/test.py --bd sandbox_spl --build -k \
 # Run tests for the flat DT version of sandbox
 ./test/py/test.py --bd sandbox_flattree --build
 
+PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \
+	./tools/binman/binman -t
+
 if [ $result == 0 ]; then
 	echo "Tests passed!"
 else
-- 
2.15.0.417.g466bffb3ac-goog

[U-Boot] [PATCH v2 2/7] test: Run binman tests

[Simon Glass]

On 26 November 2017 at 20:25, Simon Glass <sjg@chromium.org> wrote:
> Update the test script to run the binman tests also.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
>
> Changes in v2:
> - Adjust PYTHONPATH for new locations
>
>  test/run | 3 +++
>  1 file changed, 3 insertions(+)

Applied to u-boot-dm

[U-Boot] [PATCH v2 3/7] test: Run patman tests

[Simon Glass]

Update the test script to run the patman tests also.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 test/run | 1 +
 1 file changed, 1 insertion(+)

diff --git a/test/run b/test/run
index a12ea31bf23..8f72a1f7b29 100755
--- a/test/run
+++ b/test/run
@@ -19,6 +19,7 @@ run_test ./test/py/test.py --bd sandbox_spl --build -k \
 
 PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \
 	./tools/binman/binman -t
+run_test ./tools/patman/patman --test
 
 if [ $result == 0 ]; then
 	echo "Tests passed!"
-- 
2.15.0.417.g466bffb3ac-goog

[U-Boot] [PATCH v2 3/7] test: Run patman tests

[Simon Glass]

On 26 November 2017 at 20:25, Simon Glass <sjg@chromium.org> wrote:
> Update the test script to run the patman tests also.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
>
> Changes in v2: None
>
>  test/run | 1 +
>  1 file changed, 1 insertion(+)

Applied to u-boot-dm

[U-Boot] [PATCH v2 4/7] test: Run buildman tests

[Simon Glass]

Update the test script to run the buildman tests also.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 test/run | 1 +
 1 file changed, 1 insertion(+)

diff --git a/test/run b/test/run
index 8f72a1f7b29..a53383bb8e8 100755
--- a/test/run
+++ b/test/run
@@ -20,6 +20,7 @@ run_test ./test/py/test.py --bd sandbox_spl --build -k \
 PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \
 	./tools/binman/binman -t
 run_test ./tools/patman/patman --test
+run_test ./tools/buildman/buildman -t
 
 if [ $result == 0 ]; then
 	echo "Tests passed!"
-- 
2.15.0.417.g466bffb3ac-goog

[U-Boot] [PATCH v2 4/7] test: Run buildman tests

[Simon Glass]

On 26 November 2017 at 20:25, Simon Glass <sjg@chromium.org> wrote:
> Update the test script to run the buildman tests also.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
>
> Changes in v2: None
>
>  test/run | 1 +
>  1 file changed, 1 insertion(+)

Applied to u-boot-dm

[U-Boot] [PATCH v2 5/7] test: Run dtoc tests

[Simon Glass]

Update the test script to run the dtoc tests also.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Adjust PYTHONPATH for new locations

 test/run | 1 +
 1 file changed, 1 insertion(+)

diff --git a/test/run b/test/run
index a53383bb8e8..af1219fdda9 100755
--- a/test/run
+++ b/test/run
@@ -21,6 +21,7 @@ PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \
 	./tools/binman/binman -t
 run_test ./tools/patman/patman --test
 run_test ./tools/buildman/buildman -t
+PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test ./tools/dtoc/dtoc -t
 
 if [ $result == 0 ]; then
 	echo "Tests passed!"
-- 
2.15.0.417.g466bffb3ac-goog

[U-Boot] [PATCH v2 5/7] test: Run dtoc tests

[Simon Glass]

On 26 November 2017 at 20:25, Simon Glass <sjg@chromium.org> wrote:
> Update the test script to run the dtoc tests also.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
>
> Changes in v2:
> - Adjust PYTHONPATH for new locations
>
>  test/run | 1 +
>  1 file changed, 1 insertion(+)

Applied to u-boot-dm

[U-Boot] [PATCH v2 6/7] travis.yml: Run tests for tools

[Simon Glass]

Run tests for the Python tools used by U-Boot.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 .travis.yml | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index 0b7a0622bfd..5afed98bf6b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -113,6 +113,20 @@ script:
      ./test/py/test.py --bd ${TEST_PY_BD} ${TEST_PY_ID}
        -k "${TEST_PY_TEST_SPEC:-not a_test_which_does_not_exist}"
        --build-dir "$UBOOT_TRAVIS_BUILD_DIR";
+     ret=$?;
+     if [[ $ret -ne 0 ]]; then
+       exit $ret;
+     fi;
+   fi;
+   if [[ -n "${TEST_PY_TOOLS}" ]]; then
+     PYTHONPATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc/pylibfdt"
+     PATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc:${PATH}"
+     ./tools/binman/binman -t &&
+     ./tools/patman/patman --test &&
+     ./tools/buildman/buildman -t &&
+     PYTHONPATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc/pylibfdt"
+     PATH="${UBOOT_TRAVIS_BUILD_DIR}/scripts/dtc:${PATH}"
+     ./tools/dtoc/dtoc -t;
    fi
 
 matrix:
@@ -283,6 +297,7 @@ matrix:
           TEST_PY_TEST_SPEC="test_ofplatdata"
           BUILDMAN="^sandbox$"
           TOOLCHAIN="x86_64"
+          TEST_PY_TOOLS="yes"
     - env:
         - TEST_PY_BD="sandbox_flattree"
           BUILDMAN="^sandbox_flattree$"
-- 
2.15.0.417.g466bffb3ac-goog

[U-Boot] [PATCH v2 6/7] travis.yml: Run tests for tools

[sjg at google.com]

Run tests for the Python tools used by U-Boot.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2: None

 .travis.yml | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

Applied to u-boot-dm thanks!

[U-Boot] [PATCH v2 7/7] binman: Run code coverage tests

[Simon Glass]

Binman has 100% test coverage for the code as it is at present. To
encourage it to stay that way, run the code-coverage test as part of the
normal U-Boot tests.

This is RFC because it requires the Python code coverage tools to be
available.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Drop patches already applied
- Run tests with travis-ci also
- Drop RFC tag from patch for binman code-coverage tests

 test/run | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/test/run b/test/run
index af1219fdda9..d5631b802af 100755
--- a/test/run
+++ b/test/run
@@ -23,6 +23,13 @@ run_test ./tools/patman/patman --test
 run_test ./tools/buildman/buildman -t
 PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test ./tools/dtoc/dtoc -t
 
+# This needs you to set up Python test coverage tools.
+# To enable Python test coverage on Debian-type distributions (e.g. Ubuntu):
+#   $ sudo apt-get install python-pip python-pytest
+#   $ sudo pip install coverage
+PYTHONPATH=build-sandbox_spl/scripts/dtc/pylibfdt run_test \
+	./tools/binman/binman -T
+
 if [ $result == 0 ]; then
 	echo "Tests passed!"
 else
-- 
2.15.0.417.g466bffb3ac-goog

[U-Boot] [PATCH v2 7/7] binman: Run code coverage tests

[sjg at google.com]

Binman has 100% test coverage for the code as it is at present. To
encourage it to stay that way, run the code-coverage test as part of the
normal U-Boot tests.

This is RFC because it requires the Python code coverage tools to be
available.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

Changes in v2:
- Drop patches already applied
- Run tests with travis-ci also
- Drop RFC tag from patch for binman code-coverage tests

 test/run | 7 +++++++
 1 file changed, 7 insertions(+)

Applied to u-boot-dm thanks!