[PATCH v3 0/5] selftests: ftrace: ftracetest improvements

[PATCH v3 0/5] selftests: ftrace: ftracetest improvements

[Masami Hiramatsu]

Hello,

Here is v3 of ftracetest improvements, including test
return code change and immediate logging features.

This version adds 1 patch which changes test command
options to check whether given string is zero-length
or not.

Changes in v3:
 - [4/5]: Replace [ "$VAR" ] with [ ! -z "$VAR" ]
 - [5/5]: (new) Replace [ "$VAR" ] with [ ! -z "$VAR" ]
          in ftracetest

Thank you,

---

Masami Hiramatsu (5):
      selftests: ftrace: Do not failure if there is unsupported tests
      selftests: ftrace: Add --fail-unsupported option
      selftests: ftrace: Add more verbosity for immediate log
      selftests: ftrace: Output only to console with "--logdir -"
      selftests: ftrace: Check given string is not zero-length


 tools/testing/selftests/ftrace/ftracetest |   51 +++++++++++++++++++++--------
 1 file changed, 37 insertions(+), 14 deletions(-)

--
Masami Hiramatsu (Linaro) <mhiramat@kernel.org>

[PATCH v3 1/5] selftests: ftrace: Do not failure if there is unsupported tests

[Masami Hiramatsu]

Do not return failure exit code (1) for unsupported testcases,
since it is expected for stable kernels.

Previously, ftracetest is expected to run only on current
release for avoiding regressions. However, nowadays we run
it on stable kernels. This means some test cases must return
unsupported result. In such case, we should NOT exit
ftracetest with error status for unsupported results so that
kselftest (upper tests wrapper) shows it passed correctly.

Note that we continue to treat unresolved results as failure,
if test writers would like to notice user that the test result
should be reviewed, they can use exit_unresolved.

Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 tools/testing/selftests/ftrace/ftracetest |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
index 14a03ea..290cd42 100755
--- a/tools/testing/selftests/ftrace/ftracetest
+++ b/tools/testing/selftests/ftrace/ftracetest
@@ -187,7 +187,7 @@ eval_result() { # sigval
     $UNSUPPORTED)
       prlog "	[UNSUPPORTED]"
       UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO"
-      return 1 # this is not a bug, but the result should be reported.
+      return 0 # this is not a bug.
     ;;
     $XFAIL)
       prlog "	[XFAIL]"

[PATCH v3 2/5] selftests: ftrace: Add --fail-unsupported option

[Masami Hiramatsu]

Add --fail-unsupported option to fail the test result if
ftracetest gets UNSUPPORTED result. UNSUPPORTED usually
happens when the kernel is old (e.g. stable tree) or some
kernel feature is disabled.

However, if newer kernel has any bug or regression, it
can make test results in UNSUPPORTED too. This option
can detect such kernel regression.

Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 tools/testing/selftests/ftrace/ftracetest |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
index 290cd42..e033f54 100755
--- a/tools/testing/selftests/ftrace/ftracetest
+++ b/tools/testing/selftests/ftrace/ftracetest
@@ -15,6 +15,7 @@ echo "		-h|--help  Show help message"
 echo "		-k|--keep  Keep passed test logs"
 echo "		-v|--verbose Increase verbosity of test messages"
 echo "		-vv        Alias of -v -v (Show all results in stdout)"
+echo "		--fail-unsupported Treat UNSUPPORTED as a failure"
 echo "		-d|--debug Debug mode (trace all shell commands)"
 echo "		-l|--logdir <dir> Save logs on the <dir>"
 exit $1
@@ -65,6 +66,10 @@ parse_opts() { # opts
       DEBUG=1
       shift 1
     ;;
+    --fail-unsupported)
+      UNSUPPORTED_RESULT=1
+      shift 1
+    ;;
     --logdir|-l)
       LOG_DIR=$2
       shift 2
@@ -108,6 +113,7 @@ LOG_DIR=$TOP_DIR/logs/`date +%Y%m%d-%H%M%S`/
 KEEP_LOG=0
 DEBUG=0
 VERBOSE=0
+UNSUPPORTED_RESULT=0
 # Parse command-line options
 parse_opts $*
 
@@ -187,7 +193,7 @@ eval_result() { # sigval
     $UNSUPPORTED)
       prlog "	[UNSUPPORTED]"
       UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO"
-      return 0 # this is not a bug.
+      return $UNSUPPORTED_RESULT # depends on use case
     ;;
     $XFAIL)
       prlog "	[XFAIL]"

[PATCH v3 3/5] selftests: ftrace: Add more verbosity for immediate log

[Masami Hiramatsu]

Add 3-level verbosity for showing traced command log
on console immediately. Since some test cases can cause
kernel pacic if there is a probrem (like regression etc.),
we can not know which command caused the problem without
traced command log. This verbosity (-vvv) solves that
because it shows the log on console immediately. User
can get continuous command/error log.

Note that this is a kind of kernel debug mode, if you
don't see any kernel related issue, you don't need this
verbosity.

Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
  Changes in v2:
    - Do not show failure log on console again.
---
 tools/testing/selftests/ftrace/ftracetest |   10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
index e033f54..892ca4e 100755
--- a/tools/testing/selftests/ftrace/ftracetest
+++ b/tools/testing/selftests/ftrace/ftracetest
@@ -15,6 +15,7 @@ echo "		-h|--help  Show help message"
 echo "		-k|--keep  Keep passed test logs"
 echo "		-v|--verbose Increase verbosity of test messages"
 echo "		-vv        Alias of -v -v (Show all results in stdout)"
+echo "		-vvv       Alias of -v -v -v (Show all commands immediately)"
 echo "		--fail-unsupported Treat UNSUPPORTED as a failure"
 echo "		-d|--debug Debug mode (trace all shell commands)"
 echo "		-l|--logdir <dir> Save logs on the <dir>"
@@ -57,9 +58,10 @@ parse_opts() { # opts
       KEEP_LOG=1
       shift 1
     ;;
-    --verbose|-v|-vv)
+    --verbose|-v|-vv|-vvv)
       VERBOSE=$((VERBOSE + 1))
       [ $1 = '-vv' ] && VERBOSE=$((VERBOSE + 1))
+      [ $1 = '-vvv' ] && VERBOSE=$((VERBOSE + 2))
       shift 1
     ;;
     --debug|-d)
@@ -258,7 +260,9 @@ run_test() { # testfile
   testcase $1
   echo "execute$INSTANCE: "$1 > $testlog
   SIG_RESULT=0
-  if [ $VERBOSE -ge 2 ]; then
+  if [ $VERBOSE -ge 3 ]; then
+    __run_test $1 | tee -a $testlog 2>&1
+  elif [ $VERBOSE -eq 2 ]; then
     __run_test $1 2>> $testlog | tee -a $testlog
   else
     __run_test $1 >> $testlog 2>&1
@@ -268,7 +272,7 @@ run_test() { # testfile
     # Remove test log if the test was done as it was expected.
     [ $KEEP_LOG -eq 0 ] && rm $testlog
   else
-    [ $VERBOSE -ge 1 ] && catlog $testlog
+    [ $VERBOSE -eq 1 -o $VERBOSE -eq 2 ] && catlog $testlog
     TOTAL_RESULT=1
   fi
   rm -rf $TMPDIR

[PATCH v3 4/5] selftests: ftrace: Output only to console with "--logdir -"

[Masami Hiramatsu]

Output logs only to console if "-" is given to --logdir
option. In this case, ftracetest doesn't record any log
on the disk, and all logs immediately shown (including
all command logs.) Since there is no "tee" in the middle
of command and console, it outputs the log really soon.

This option is useful only when the console is logged.

Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
  Changes in v3:
   - Use [ ! -z "$VAR" ] instead of [ "$VAR" ].
---
 tools/testing/selftests/ftrace/ftracetest |   29 +++++++++++++++++++++--------
 1 file changed, 21 insertions(+), 8 deletions(-)

diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
index 892ca4e..453f73d 100755
--- a/tools/testing/selftests/ftrace/ftracetest
+++ b/tools/testing/selftests/ftrace/ftracetest
@@ -19,6 +19,7 @@ echo "		-vvv       Alias of -v -v -v (Show all commands immediately)"
 echo "		--fail-unsupported Treat UNSUPPORTED as a failure"
 echo "		-d|--debug Debug mode (trace all shell commands)"
 echo "		-l|--logdir <dir> Save logs on the <dir>"
+echo "		            If <dir> is -, all logs output in console only"
 exit $1
 }
 
@@ -127,14 +128,20 @@ if [ -z "$TRACING_DIR" -o ! -d "$TRACING_DIR" ]; then
 fi
 
 # Preparing logs
-LOG_FILE=$LOG_DIR/ftracetest.log
-mkdir -p $LOG_DIR || errexit "Failed to make a log directory: $LOG_DIR"
-date > $LOG_FILE
+if [ "x$LOG_DIR" = "x-" ]; then
+  LOG_FILE=
+  date
+else
+  LOG_FILE=$LOG_DIR/ftracetest.log
+  mkdir -p $LOG_DIR || errexit "Failed to make a log directory: $LOG_DIR"
+  date > $LOG_FILE
+fi
+
 prlog() { # messages
-  echo "$@" | tee -a $LOG_FILE
+  [ -z "$LOG_FILE" ] && echo "$@" || echo "$@" | tee -a $LOG_FILE
 }
 catlog() { #file
-  cat $1 | tee -a $LOG_FILE
+  [ -z "$LOG_FILE" ] && cat $1 || cat $1 | tee -a $LOG_FILE
 }
 prlog "=== Ftrace unit tests ==="
 
@@ -255,12 +262,18 @@ __run_test() { # testfile
 # Run one test case
 run_test() { # testfile
   local testname=`basename $1`
-  local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX`
+  if [ ! -z "$LOG_FILE" ] ; then
+    local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX`
+  else
+    local testlog=`/proc/self/fd/1`
+  fi
   export TMPDIR=`mktemp -d /tmp/ftracetest-dir.XXXXXX`
   testcase $1
   echo "execute$INSTANCE: "$1 > $testlog
   SIG_RESULT=0
-  if [ $VERBOSE -ge 3 ]; then
+  if [ -z "$LOG_FILE" ]; then
+    __run_test $1 2>&1
+  elif [ $VERBOSE -ge 3 ]; then
     __run_test $1 | tee -a $testlog 2>&1
   elif [ $VERBOSE -eq 2 ]; then
     __run_test $1 2>> $testlog | tee -a $testlog
@@ -270,7 +283,7 @@ run_test() { # testfile
   eval_result $SIG_RESULT
   if [ $? -eq 0 ]; then
     # Remove test log if the test was done as it was expected.
-    [ $KEEP_LOG -eq 0 ] && rm $testlog
+    [ $KEEP_LOG -eq 0 -a ! -z "$LOG_FILE" ] && rm $testlog
   else
     [ $VERBOSE -eq 1 -o $VERBOSE -eq 2 ] && catlog $testlog
     TOTAL_RESULT=1

Re: [PATCH v3 4/5] selftests: ftrace: Output only to console with "--logdir -"

[Steven Rostedt]

On Thu,  6 Jul 2017 18:10:58 +0900
Masami Hiramatsu <mhiramat@kernel.org> wrote:

> @@ -255,12 +262,18 @@ __run_test() { # testfile
>  # Run one test case
>  run_test() { # testfile
>    local testname=`basename $1`
> -  local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX`
> +  if [ ! -z "$LOG_FILE" ] ; then
> +    local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX`
> +  else
> +    local testlog=`/proc/self/fd/1`

As Stafford mentioned. I don't think you want backticks here.

-- Steve

> +  fi
>    export TMPDIR=`mktemp -d /tmp/ftracetest-dir.XXXXXX`
>    testcase $1

Re: [PATCH v3 4/5] selftests: ftrace: Output only to console with "--logdir -"

[Masami Hiramatsu]

On Thu, 6 Jul 2017 09:50:26 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Thu,  6 Jul 2017 18:10:58 +0900
> Masami Hiramatsu <mhiramat@kernel.org> wrote:
> 
> > @@ -255,12 +262,18 @@ __run_test() { # testfile
> >  # Run one test case
> >  run_test() { # testfile
> >    local testname=`basename $1`
> > -  local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX`
> > +  if [ ! -z "$LOG_FILE" ] ; then
> > +    local testlog=`mktemp $LOG_DIR/${testname}-log.XXXXXX`
> > +  else
> > +    local testlog=`/proc/self/fd/1`
> 
> As Stafford mentioned. I don't think you want backticks here.

Oops, right!

-- 
Masami Hiramatsu <mhiramat@kernel.org>

[PATCH v3 5/5] selftests: ftrace: Check given string is not zero-length

[Masami Hiramatsu]

Use [ ! -z "$VAR" ] instead of [ "$VAR" ] to check
whether the given string variable is not zero-length
since it obviously shows what it means.

Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
---
 tools/testing/selftests/ftrace/ftracetest |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
index 453f73d..aab3d5e 100755
--- a/tools/testing/selftests/ftrace/ftracetest
+++ b/tools/testing/selftests/ftrace/ftracetest
@@ -8,7 +8,7 @@
 # Released under the terms of the GPL v2.
 
 usage() { # errno [message]
-[ "$2" ] && echo $2
+[ ! -z "$2" ] && echo $2
 echo "Usage: ftracetest [options] [testcase(s)] [testcase-directory(s)]"
 echo " Options:"
 echo "		-h|--help  Show help message"
@@ -50,7 +50,7 @@ parse_opts() { # opts
   local OPT_TEST_CASES=
   local OPT_TEST_DIR=
 
-  while [ "$1" ]; do
+  while [ ! -z "$1" ]; do
     case "$1" in
     --help|-h)
       usage 0
@@ -96,7 +96,7 @@ parse_opts() { # opts
     ;;
     esac
   done
-  if [ "$OPT_TEST_CASES" ]; then
+  if [ ! -z "$OPT_TEST_CASES" ]; then
     TEST_CASES=$OPT_TEST_CASES
   fi
 }