[PATCH 0/2] selftests: firmware: skip testing unsupported features

[PATCH 0/2] selftests: firmware: skip testing unsupported features

[Amit Pundir]

Hi,

At Linaro we run mainline/linux-next selftests on LTS releases and
run into few test failures due to kernel mismatch or missing upstream
functionality in older kernels. Discussed at length here:
https://lkml.org/lkml/2017/6/15/652

This patch series is an attempt to modify selftest firmware test scripts.
The proposed changes skip/ignore testing the upstream functionality
missing in the older kernel releases.

Regards,
Amit Pundir


Amit Pundir (2):
  selftests: firmware: skip unsupported async loading tests
  selftests: firmware: skip unsupported custom firmware fallback tests

 tools/testing/selftests/firmware/fw_fallback.sh   | 38 ++++++++++++++++-------
 tools/testing/selftests/firmware/fw_filesystem.sh | 34 ++++++++++++--------
 2 files changed, 47 insertions(+), 25 deletions(-)

-- 
2.7.4

[PATCH 1/2] selftests: firmware: skip unsupported async loading tests

[Amit Pundir]

Ignore async firmware loading tests on older kernel releases,
which do not support this feature.

Signed-off-by: Amit Pundir <amit.pundir@linaro.org>
---
 tools/testing/selftests/firmware/fw_filesystem.sh | 34 ++++++++++++++---------
 1 file changed, 21 insertions(+), 13 deletions(-)

diff --git a/tools/testing/selftests/firmware/fw_filesystem.sh b/tools/testing/selftests/firmware/fw_filesystem.sh
index 62f2d6f54929..b1f20fef36c7 100755
--- a/tools/testing/selftests/firmware/fw_filesystem.sh
+++ b/tools/testing/selftests/firmware/fw_filesystem.sh
@@ -70,9 +70,13 @@ if printf '\000' >"$DIR"/trigger_request 2> /dev/null; then
 	exit 1
 fi
 
-if printf '\000' >"$DIR"/trigger_async_request 2> /dev/null; then
-	echo "$0: empty filename should not succeed (async)" >&2
-	exit 1
+if [ ! -e "$DIR"/trigger_async_request ]; then
+	echo "$0: empty filename: async trigger not supported" >&2
+else
+	if printf '\000' >"$DIR"/trigger_async_request 2> /dev/null; then
+		echo "$0: empty filename should not succeed (async)" >&2
+		exit 1
+	fi
 fi
 
 # Request a firmware that doesn't exist, it should fail.
@@ -105,17 +109,21 @@ else
 fi
 
 # Try the asynchronous version too
-if ! echo -n "$NAME" >"$DIR"/trigger_async_request ; then
-	echo "$0: could not trigger async request" >&2
-	exit 1
-fi
-
-# Verify the contents are what we expect.
-if ! diff -q "$FW" /dev/test_firmware >/dev/null ; then
-	echo "$0: firmware was not loaded (async)" >&2
-	exit 1
+if [ ! -e "$DIR"/trigger_async_request ]; then
+	echo "$0: firmware loading: async trigger not supported" >&2
 else
-	echo "$0: async filesystem loading works"
+	if ! echo -n "$NAME" >"$DIR"/trigger_async_request ; then
+		echo "$0: could not trigger async request" >&2
+		exit 1
+	fi
+
+	# Verify the contents are what we expect.
+	if ! diff -q "$FW" /dev/test_firmware >/dev/null ; then
+		echo "$0: firmware was not loaded (async)" >&2
+		exit 1
+	else
+		echo "$0: async filesystem loading works"
+	fi
 fi
 
 ### Batched requests tests
-- 
2.7.4

Re: [PATCH 1/2] selftests: firmware: skip unsupported async loading tests

[Greg Kroah-Hartman]

On Wed, Nov 08, 2017 at 05:17:39PM +0530, Amit Pundir wrote:
> Ignore async firmware loading tests on older kernel releases,
> which do not support this feature.
> 
> Signed-off-by: Amit Pundir <amit.pundir@linaro.org>

Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

Re: [PATCH 1/2] selftests: firmware: skip unsupported async loading tests

[Shuah Khan]

On 11/08/2017 04:47 AM, Amit Pundir wrote:
> Ignore async firmware loading tests on older kernel releases,
> which do not support this feature.
> 
> Signed-off-by: Amit Pundir <amit.pundir@linaro.org>
> ---
>  tools/testing/selftests/firmware/fw_filesystem.sh | 34 ++++++++++++++---------
>  1 file changed, 21 insertions(+), 13 deletions(-)
> 
> diff --git a/tools/testing/selftests/firmware/fw_filesystem.sh b/tools/testing/selftests/firmware/fw_filesystem.sh
> index 62f2d6f54929..b1f20fef36c7 100755
> --- a/tools/testing/selftests/firmware/fw_filesystem.sh
> +++ b/tools/testing/selftests/firmware/fw_filesystem.sh
> @@ -70,9 +70,13 @@ if printf '\000' >"$DIR"/trigger_request 2> /dev/null; then
>  	exit 1
>  fi
>  
> -if printf '\000' >"$DIR"/trigger_async_request 2> /dev/null; then
> -	echo "$0: empty filename should not succeed (async)" >&2
> -	exit 1
> +if [ ! -e "$DIR"/trigger_async_request ]; then
> +	echo "$0: empty filename: async trigger not supported" >&2
> +else
> +	if printf '\000' >"$DIR"/trigger_async_request 2> /dev/null; then
> +		echo "$0: empty filename should not succeed (async)" >&2
> +		exit 1
> +	fi
>  fi
>  
>  # Request a firmware that doesn't exist, it should fail.
> @@ -105,17 +109,21 @@ else
>  fi
>  
>  # Try the asynchronous version too
> -if ! echo -n "$NAME" >"$DIR"/trigger_async_request ; then
> -	echo "$0: could not trigger async request" >&2
> -	exit 1
> -fi
> -
> -# Verify the contents are what we expect.
> -if ! diff -q "$FW" /dev/test_firmware >/dev/null ; then
> -	echo "$0: firmware was not loaded (async)" >&2
> -	exit 1
> +if [ ! -e "$DIR"/trigger_async_request ]; then
> +	echo "$0: firmware loading: async trigger not supported" >&2

Please make this message consistent with the rest when a test is ignored
as in the following tomake it clear what is being done.

fw_fallback.sh:	echo "usermode helper disabled so ignoring test"
fw_filesystem.sh:		echo "Configuration triggers not present, ignoring test"


thanks,
-- Shuah

[PATCH 2/2] selftests: firmware: skip unsupported custom firmware fallback tests

[Amit Pundir]

Ignore custom firmware loading and cancellation tests on older
kernel releases, which do not support this feature.

Signed-off-by: Amit Pundir <amit.pundir@linaro.org>
---
 tools/testing/selftests/firmware/fw_fallback.sh | 38 +++++++++++++++++--------
 1 file changed, 26 insertions(+), 12 deletions(-)

diff --git a/tools/testing/selftests/firmware/fw_fallback.sh b/tools/testing/selftests/firmware/fw_fallback.sh
index a52a3bab532b..72858c921eed 100755
--- a/tools/testing/selftests/firmware/fw_fallback.sh
+++ b/tools/testing/selftests/firmware/fw_fallback.sh
@@ -86,6 +86,11 @@ load_fw_cancel()
 
 load_fw_custom()
 {
+	if [ ! -e "$DIR"/trigger_custom_fallback ]; then
+		echo "$0: custom fallback loading trigger not supported" >&2
+		return 1
+	fi
+
 	local name="$1"
 	local file="$2"
 
@@ -108,11 +113,17 @@ load_fw_custom()
 
 	# Wait for request to finish.
 	wait
+	return 0
 }
 
 
 load_fw_custom_cancel()
 {
+	if [ ! -e "$DIR"/trigger_custom_fallback ]; then
+		echo "$0: cancelling custom fallback trigger not supported" >&2
+		return 1
+	fi
+
 	local name="$1"
 	local file="$2"
 
@@ -133,6 +144,7 @@ load_fw_custom_cancel()
 
 	# Wait for request to finish.
 	wait
+	return 0
 }
 
 load_fw_fallback_with_child()
@@ -227,20 +239,22 @@ else
 	echo "$0: cancelling fallback mechanism works"
 fi
 
-load_fw_custom "$NAME" "$FW"
-if ! diff -q "$FW" /dev/test_firmware >/dev/null ; then
-	echo "$0: firmware was not loaded" >&2
-	exit 1
-else
-	echo "$0: custom fallback loading mechanism works"
+if load_fw_custom "$NAME" "$FW" ; then
+	if ! diff -q "$FW" /dev/test_firmware >/dev/null ; then
+		echo "$0: firmware was not loaded" >&2
+		exit 1
+	else
+		echo "$0: custom fallback loading mechanism works"
+	fi
 fi
 
-load_fw_custom_cancel "nope-$NAME" "$FW"
-if diff -q "$FW" /dev/test_firmware >/dev/null ; then
-	echo "$0: firmware was expected to be cancelled" >&2
-	exit 1
-else
-	echo "$0: cancelling custom fallback mechanism works"
+if load_fw_custom_cancel "nope-$NAME" "$FW" ; then
+	if diff -q "$FW" /dev/test_firmware >/dev/null ; then
+		echo "$0: firmware was expected to be cancelled" >&2
+		exit 1
+	else
+		echo "$0: cancelling custom fallback mechanism works"
+	fi
 fi
 
 set +e
-- 
2.7.4

Re: [PATCH 2/2] selftests: firmware: skip unsupported custom firmware fallback tests

[Greg Kroah-Hartman]

On Wed, Nov 08, 2017 at 05:17:40PM +0530, Amit Pundir wrote:
> Ignore custom firmware loading and cancellation tests on older
> kernel releases, which do not support this feature.
> 
> Signed-off-by: Amit Pundir <amit.pundir@linaro.org>

Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

Re: [PATCH 2/2] selftests: firmware: skip unsupported custom firmware fallback tests

[Shuah Khan]

On 11/08/2017 04:47 AM, Amit Pundir wrote:
> Ignore custom firmware loading and cancellation tests on older
> kernel releases, which do not support this feature.
> 
> Signed-off-by: Amit Pundir <amit.pundir@linaro.org>
> ---
>  tools/testing/selftests/firmware/fw_fallback.sh | 38 +++++++++++++++++--------
>  1 file changed, 26 insertions(+), 12 deletions(-)
> 
> diff --git a/tools/testing/selftests/firmware/fw_fallback.sh b/tools/testing/selftests/firmware/fw_fallback.sh
> index a52a3bab532b..72858c921eed 100755
> --- a/tools/testing/selftests/firmware/fw_fallback.sh
> +++ b/tools/testing/selftests/firmware/fw_fallback.sh
> @@ -86,6 +86,11 @@ load_fw_cancel()
>  
>  load_fw_custom()
>  {
> +	if [ ! -e "$DIR"/trigger_custom_fallback ]; then
> +		echo "$0: custom fallback loading trigger not supported" >&2

Please make this message consistent with the rest when a test is ignored
as in the following to make it clear what is being done.

fw_fallback.sh:	echo "usermode helper disabled so ignoring test"
fw_filesystem.sh:		echo "Configuration triggers not present, ignoring test"

> +		return 1
> +	fi
> +
>  	local name="$1"
>  	local file="$2"
>  
> @@ -108,11 +113,17 @@ load_fw_custom()
>  
>  	# Wait for request to finish.
>  	wait
> +	return 0
>  }
>  
>  
>  load_fw_custom_cancel()
>  {
> +	if [ ! -e "$DIR"/trigger_custom_fallback ]; then
> +		echo "$0: cancelling custom fallback trigger not supported" >&2

Please make this message consistent with the rest when a test is ignored
as in the following to make it clear what is being done.

fw_fallback.sh:	echo "usermode helper disabled so ignoring test"
fw_filesystem.sh:		echo "Configuration triggers not present, ignoring test"

If I missed ponting out any other instances of ignoring, please do the
same for them.

> +		return 1
> +	fi
> +
>  	local name="$1"
>  	local file="$2"
>  
> @@ -133,6 +144,7 @@ load_fw_custom_cancel()
>  
>  	# Wait for request to finish.
>  	wait
> +	return 0
>  }
>  
>  load_fw_fallback_with_child()
> @@ -227,20 +239,22 @@ else
>  	echo "$0: cancelling fallback mechanism works"
>  fi
>  
> -load_fw_custom "$NAME" "$FW"
> -if ! diff -q "$FW" /dev/test_firmware >/dev/null ; then
> -	echo "$0: firmware was not loaded" >&2
> -	exit 1
> -else
> -	echo "$0: custom fallback loading mechanism works"
> +if load_fw_custom "$NAME" "$FW" ; then
> +	if ! diff -q "$FW" /dev/test_firmware >/dev/null ; then
> +		echo "$0: firmware was not loaded" >&2
> +		exit 1
> +	else
> +		echo "$0: custom fallback loading mechanism works"
> +	fi
>  fi
>  
> -load_fw_custom_cancel "nope-$NAME" "$FW"
> -if diff -q "$FW" /dev/test_firmware >/dev/null ; then
> -	echo "$0: firmware was expected to be cancelled" >&2
> -	exit 1
> -else
> -	echo "$0: cancelling custom fallback mechanism works"
> +if load_fw_custom_cancel "nope-$NAME" "$FW" ; then
> +	if diff -q "$FW" /dev/test_firmware >/dev/null ; then
> +		echo "$0: firmware was expected to be cancelled" >&2
> +		exit 1
> +	else
> +		echo "$0: cancelling custom fallback mechanism works"
> +	fi
>  fi
>  
>  set +e
> 

thanks,
-- Shuah

Re: [PATCH 0/2] selftests: firmware: skip testing unsupported features

[Sumit Semwal]

Hi Amit,

Thanks for your patches.

On 8 November 2017 at 17:17, Amit Pundir <amit.pundir@linaro.org> wrote:
> Hi,
>
> At Linaro we run mainline/linux-next selftests on LTS releases and
> run into few test failures due to kernel mismatch or missing upstream
> functionality in older kernels. Discussed at length here:
> https://lkml.org/lkml/2017/6/15/652
>
> This patch series is an attempt to modify selftest firmware test scripts.
> The proposed changes skip/ignore testing the upstream functionality
> missing in the older kernel releases.
>
> Regards,
> Amit Pundir
>
>
> Amit Pundir (2):
>   selftests: firmware: skip unsupported async loading tests
>   selftests: firmware: skip unsupported custom firmware fallback tests
Please feel free to add my
Reviewed-by: Sumit Semwal <sumit.semwal@linaro.org>
to both these patches.
>
>  tools/testing/selftests/firmware/fw_fallback.sh   | 38 ++++++++++++++++-------
>  tools/testing/selftests/firmware/fw_filesystem.sh | 34 ++++++++++++--------
>  2 files changed, 47 insertions(+), 25 deletions(-)
>
> --
> 2.7.4
>
Best,
Sumit.

Re: [PATCH 0/2] selftests: firmware: skip testing unsupported features

[Shuah Khan]

On 11/08/2017 04:47 AM, Amit Pundir wrote:
> Hi,
> 
> At Linaro we run mainline/linux-next selftests on LTS releases and
> run into few test failures due to kernel mismatch or missing upstream
> functionality in older kernels. Discussed at length here:
> https://lkml.org/lkml/2017/6/15/652
> 
> This patch series is an attempt to modify selftest firmware test scripts.
> The proposed changes skip/ignore testing the upstream functionality
> missing in the older kernel releases.
> 
> Regards,
> Amit Pundir
> 
> 
> Amit Pundir (2):
>   selftests: firmware: skip unsupported async loading tests
>   selftests: firmware: skip unsupported custom firmware fallback tests
> 
>  tools/testing/selftests/firmware/fw_fallback.sh   | 38 ++++++++++++++++-------
>  tools/testing/selftests/firmware/fw_filesystem.sh | 34 ++++++++++++--------
>  2 files changed, 47 insertions(+), 25 deletions(-)
> 

Thanks for the patches. I have couple of minor comments on the patches.
Please see the individual responses to the patches.

-- Shuah

Re: [PATCH 0/2] selftests: firmware: skip testing unsupported features

[Luis R. Rodriguez]

On Wed, Nov 08, 2017 at 05:17:38PM +0530, Amit Pundir wrote:
> Hi,
> 
> At Linaro we run mainline/linux-next selftests on LTS releases and
> run into few test failures due to kernel mismatch or missing upstream
> functionality in older kernels. Discussed at length here:
> https://lkml.org/lkml/2017/6/15/652
> 
> This patch series is an attempt to modify selftest firmware test scripts.
> The proposed changes skip/ignore testing the upstream functionality
> missing in the older kernel releases.
> 
> Regards,
> Amit Pundir
> 
> 
> Amit Pundir (2):
>   selftests: firmware: skip unsupported async loading tests
>   selftests: firmware: skip unsupported custom firmware fallback tests

For both, if you re-send with a Fixes tag that would be appreciated, there
is tooling now in place so that when a Fixes tag is used maintainers in
a distro kernel merge these patches as well when relevant. If you can
resend with that tag, or if Shuah is willing to amend the commit log
with them if you provide it:

Acked-by: Luis R. Rodriguez <mcgrof@kernel.org>

  Luis