Re: [PATCH v3 1/4] tests: Check the message when dtc fails

[Herve Codina <herve.codina@bootlin.com>]

Hi Simon,

On Thu, 29 Oct 2026 03:30:54 -0600
Simon Glass <sjg@chromium.org> wrote:

> dtc-fatal.sh only checks that dtc exits with status 1, so a test which
> expects a particular error passes just as well if dtc fails for some
> other reason, such as a mistake in the test's dts file.
> 
> Use run_wrap_error_msg_test() instead, which also checks that dtc's
> output contains the expected message.
> 
> dtc-fatal.sh also catches a valgrind error, since valgrind exits with
> a status other than 1. Make wrap_error_msg() report a valgrind error as
> wrap_test() does, so this is not lost.
> 
> Suggested-by: Herve Codina <herve.codina@bootlin.com>
> Assisted-by: Claude:claude-opus-5-5
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
> 
> Changes in v3:
> - Use run_wrap_error_msg_test() instead of adding -m to dtc-fatal.sh
> - Check the message in all five tests
> 
> Changes in v2:
> - Add new patch to check the error message in dtc-fatal.sh
> 
>  tests/run_tests.sh | 18 +++++++++++++-----
>  1 file changed, 13 insertions(+), 5 deletions(-)
> 
> diff --git a/tests/run_tests.sh b/tests/run_tests.sh
> index 5bc7817..d3134f2 100755
> --- a/tests/run_tests.sh
> +++ b/tests/run_tests.sh
> @@ -193,6 +193,9 @@ wrap_error_msg () {
>  	FAIL_IF_SIGNAL $ret
>  	if [ "$ret" -eq 0 ]; then
>  	    FAIL "Expected non-zero return code"
> +	elif [ "$ret" -eq $VGCODE ]; then
> +	    echo "VALGRIND ERROR"
> +	    exit $VGCODE
>  	fi
>  	if ! grep -qF -- "$msg" "$log"; then
>  	    FAIL "Expected message '$msg'"
> @@ -319,7 +322,8 @@ libfdt_overlay_tests () {
>  	run_dtc_test -I dts -O dtb -o $tree.test.dtb "$SRCDIR/$tree.dts"
>  	run_test overlay_bad_fixup overlay_base_manual_symbols.test.dtb $tree.test.dtb
>      done
> -    run_sh_test "$SRCDIR/dtc-fatal.sh" -I dts -O dtb -o /dev/null "$SRCDIR/fixup-ref-to-path.dts"
> +    run_wrap_error_msg_test "Can't generate fixup for reference to path" \
> +	$VALGRIND $DTC -I dts -O dtb -o /dev/null "$SRCDIR/fixup-ref-to-path.dts"

I don't think that the $VALGRIND here is a good idea. Also this is not consistent
with other dtc test calls. Those tests calls (other than using a script) use
run_dtc_test(). run_dtc_test() use $VALGRIND internally.

Could we imagine run_dtc_fatal_test() (or similar name) to call here in order
to be consistent with other calls?

This will look like the following:
--- 8< ---
	run_dtc_fatal_test "Can't generate fixup for reference to path" \
			   -I dts -O dtb -o /dev/null "$SRCDIR/fixup-ref-to-path.dts"
--- 8< ---

At some point, also dtc-checkfails.sh script could be replace by
run_dtc_checkfails_test wrapper and dtc-fails.sh by run_dtc_checkfails_test
wrapper.

Best regards,
Hervé