[RFC PATCH 0/3] ftrace: Add a ftrace test collection

[RFC PATCH 0/3] ftrace: Add a ftrace test collection

[Masami Hiramatsu]

Hi,

I'd like to introduce a collection of testcases for ftrace to
avoid regressions.

For a long time, we've tried to stabilize and extend ftrace
tracing infrastructure. This small test framework is a kind of
stabilizing work for ftrace. For the first step, this series
just introduces a few basic testcases. However, it is easy to
add additional tests. I'd like to ask you, ftrace developers,
to add tests for your features to ensure it will not be broken
by future works.

ftracetest is a tiny bash script so that anyone can easily
understand what it does. I think it is better to share and
discuss this tests before growing it.

 - Is it enough to support bash script? (of course you can
   invoke other commands from the script)
 - What's the good naming method of testcases?
 - Is any dependency check required?

BTW, I decided to put this under tools/testing/ftrace instead
of tools/testing/selftests/, because all tests requires root
privilege. It will be one of discussion points. Anyway,
it is easy to integrate this to the selftests.

Thank you,

---

Masami Hiramatsu (3):
      ftracetest: Initial commit for ftracetest
      ftracetest: Add ftrace basic testcases
      ftracetest: Add kprobe basic testcases


 tools/testing/ftrace/README                        |   38 ++++++
 tools/testing/ftrace/ftracetest                    |  133 ++++++++++++++++++++
 tools/testing/ftrace/test.d/basic1.tc              |    3 
 tools/testing/ftrace/test.d/basic2.tc              |    6 +
 tools/testing/ftrace/test.d/basic3.tc              |    8 +
 .../testing/ftrace/test.d/kprobe/add_and_remove.tc |   11 ++
 tools/testing/ftrace/test.d/kprobe/busy_check.tc   |   14 ++
 tools/testing/ftrace/test.d/template               |    4 +
 8 files changed, 217 insertions(+)
 create mode 100644 tools/testing/ftrace/README
 create mode 100755 tools/testing/ftrace/ftracetest
 create mode 100644 tools/testing/ftrace/test.d/basic1.tc
 create mode 100644 tools/testing/ftrace/test.d/basic2.tc
 create mode 100644 tools/testing/ftrace/test.d/basic3.tc
 create mode 100644 tools/testing/ftrace/test.d/kprobe/add_and_remove.tc
 create mode 100644 tools/testing/ftrace/test.d/kprobe/busy_check.tc
 create mode 100644 tools/testing/ftrace/test.d/template

--

[RFC PATCH 1/3] ftracetest: Initial commit for ftracetest

[Masami Hiramatsu]

ftracetest is a collection of testcase shell-scripts for ftrace.
To avoid regressions of ftrace, these testcases check correct
ftrace behaviors. If someone would like to add any features on
ftrace, the patch series should have at least one testcase for
checking the new behavior.

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 tools/testing/ftrace/README          |   38 ++++++++++
 tools/testing/ftrace/ftracetest      |  133 ++++++++++++++++++++++++++++++++++
 tools/testing/ftrace/test.d/template |    4 +
 3 files changed, 175 insertions(+)
 create mode 100644 tools/testing/ftrace/README
 create mode 100755 tools/testing/ftrace/ftracetest
 create mode 100644 tools/testing/ftrace/test.d/template

diff --git a/tools/testing/ftrace/README b/tools/testing/ftrace/README
new file mode 100644
index 0000000..16de781
--- /dev/null
+++ b/tools/testing/ftrace/README
@@ -0,0 +1,38 @@
+Linux Ftrace Testcases
+
+This is a collection of testcases for ftrace tracing feature in the Linux
+kernel. Since ftrace exports interfaces via the debugfs, we just need
+shell scripts for testing. Feel free to add new test cases.
+
+Running the ftrace testcases
+============================
+
+At first, you need to be the root user to run this script.
+To run all testcases:
+
+  $ sudo ./ftracetest
+
+To run specific testcases:
+
+  # ./ftracetest test.d/basic3.tc
+
+Contributing new testcases
+==========================
+
+Copy test.d/template to your testcase (whose filename must has *.tc extension)
+and rewrite test description line.
+
+ * The working directory of the script is <debugfs>/tracing/.
+
+ * Take care about the side effect, because that is run with root privilege.
+
+ * Don't take too long. It's a kind of unit test.
+
+ * You can add a directory for your testcases under test.d/ if needed.
+
+TODO
+====
+
+ * Fancy colored output :)
+
+ * Integrate with selftest?
diff --git a/tools/testing/ftrace/ftracetest b/tools/testing/ftrace/ftracetest
new file mode 100755
index 0000000..5a7c9f8
--- /dev/null
+++ b/tools/testing/ftrace/ftracetest
@@ -0,0 +1,133 @@
+#!/bin/bash
+# ftracetest - Ftrace test shell scripts
+# Written by Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
+
+usage() { # errno [message]
+[ "$2" ] && echo $2
+echo "Usage: ftracetest [options] [testcase(s)]"
+echo " Options:"
+echo "		-h|--help  Show help message"
+echo "		-k|--keep  Keep passed test logs"
+echo "		-d|--debug Debug mode (trace all shell commands)"
+exit $1
+}
+
+# Utilities
+absdir() { # file_path
+  (cd `dirname $1`; pwd)
+}
+
+errexit() { # message
+  echo "Error: $1" 1>&2
+  exit 1
+}
+
+# Ensuring user privilege
+if [ `id -u` -ne 0 ]; then
+  errexit "this must be run by root user"
+fi
+
+parse_opts() { # opts
+  OPT_TEST_CASES=
+  while [ "$1" ]; do
+    case "$1" in
+    --help|-h)
+      usage 0
+    ;;
+    --keep|-k)
+      KEEP_LOG=1
+      shift 1
+    ;;
+    --debug|-d)
+      DEBUG=1
+      shift 1
+    ;;
+    *)
+      if [ -f "$1" ]; then
+        OPT_TEST_CASES="$OPT_TEST_CASES `absdir $1`/`basename $1`"
+        shift 1
+      else
+        usage 1 "Invalid option ($1)"
+      fi
+    ;;
+    esac
+  done
+  if [ "$OPT_TEST_CASES" ]; then
+    TEST_CASES=$OPT_TEST_CASES
+  fi
+}
+
+# Parameters
+DEBUGFS_DIR=`grep debugfs /proc/mounts | cut -f2 -d' '`
+TRACING_DIR=$DEBUGFS_DIR/tracing
+TOP_DIR=`absdir $0`
+TEST_DIR=$TOP_DIR/test.d
+TEST_CASES=`find $TEST_DIR -name *.tc`
+LOG_DIR=$TOP_DIR/logs/`date +%Y%m%d-%H%M%S`/
+KEEP_LOG=0
+DEBUG=0
+# Parse command-line options
+parse_opts $*
+
+[ $DEBUG -ne 0 ] && set -x
+
+# Verify parameters
+if [ -z "$DEBUGFS_DIR" -o ! -d "$TRACING_DIR" ]; then
+  errexit "No ftrace directory found"
+fi
+
+# Preparing logs
+LOG_FILE=$LOG_DIR/ftracetest.log
+mkdir -p $LOG_DIR || errexit "Failed to make a log directory: $LOG_DIR"
+date > $LOG_FILE
+prlog() { # messages
+  echo $@ | tee -a $LOG_FILE
+}
+catlog() { #file
+  cat $1 | tee -a $LOG_FILE
+}
+
+# Testcase management
+PASSED_CASES=
+FAILED_CASES=
+CASENO=0
+testcase() { # testfile
+  CASENO=$((CASENO+1))
+  prlog -n "[$CASENO]"`grep "^#[ \t]*description:" $1 | cut -f2 -d:`
+}
+failed() {
+  prlog -e "\t[FAILED]"
+  FAILED_CASES="$FAILED_CASES $CASENO"
+}
+passed() {
+  prlog -e "\t[PASSED]"
+  PASSED_CASES="$PASSED_CASES $CASENO"
+}
+
+
+# Run one test case
+function run_test() { # testfile
+  local testname=`basename $1`
+  local testlog=`mktemp --tmpdir=$LOG_DIR ${testname}-XXXXXX.log`
+  testcase $1
+  echo "execute: "$1 > $testlog
+  (cd $TRACING_DIR; set -x ; source $t) >> $testlog 2>&1
+  ret=$?
+  if [ $ret -ne 0 ]; then
+    failed
+    catlog $testlog
+  else
+    passed
+    [ $KEEP_LOG -eq 0 ] && rm $testlog
+  fi
+}
+
+# Main loop
+for t in $TEST_CASES; do
+  run_test $t
+done
+prlog ""
+prlog "Passed: " `echo $PASSED_CASES | wc -w`
+prlog "Failed: " `echo $FAILED_CASES | wc -w`
+
+test -z "$FAILED_CASES" # if no error, return 0
diff --git a/tools/testing/ftrace/test.d/template b/tools/testing/ftrace/test.d/template
new file mode 100644
index 0000000..e855b65
--- /dev/null
+++ b/tools/testing/ftrace/test.d/template
@@ -0,0 +1,4 @@
+#!/bin/bash
+# description: %HERE DESCRIBE WHAT THIS DOES%
+# you have to add ".tc" extention for your testcase file
+exit 0 # Return 0 if the test is passed, otherwise return !0

[RFC PATCH 2/3] ftracetest: Add ftrace basic testcases

[Masami Hiramatsu]

Add ftrace basic testcases. This just checks ftrace debugfs
interface works as it is designed.

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 tools/testing/ftrace/test.d/basic1.tc |    3 +++
 tools/testing/ftrace/test.d/basic2.tc |    6 ++++++
 tools/testing/ftrace/test.d/basic3.tc |    8 ++++++++
 3 files changed, 17 insertions(+)
 create mode 100644 tools/testing/ftrace/test.d/basic1.tc
 create mode 100644 tools/testing/ftrace/test.d/basic2.tc
 create mode 100644 tools/testing/ftrace/test.d/basic3.tc

diff --git a/tools/testing/ftrace/test.d/basic1.tc b/tools/testing/ftrace/test.d/basic1.tc
new file mode 100644
index 0000000..275bd71
--- /dev/null
+++ b/tools/testing/ftrace/test.d/basic1.tc
@@ -0,0 +1,3 @@
+#!/bin/bash
+# description: Basic trace file check
+test -f README -a -f trace -a -f tracing_on -a -f trace_pipe
diff --git a/tools/testing/ftrace/test.d/basic2.tc b/tools/testing/ftrace/test.d/basic2.tc
new file mode 100644
index 0000000..5849922
--- /dev/null
+++ b/tools/testing/ftrace/test.d/basic2.tc
@@ -0,0 +1,6 @@
+#!/bin/bash
+# description: Basic test for tracers
+for t in `cat available_tracers`; do
+  echo $t > current_tracer || exit 1
+done
+echo nop > current_tracer
diff --git a/tools/testing/ftrace/test.d/basic3.tc b/tools/testing/ftrace/test.d/basic3.tc
new file mode 100644
index 0000000..0b9cb2c
--- /dev/null
+++ b/tools/testing/ftrace/test.d/basic3.tc
@@ -0,0 +1,8 @@
+#!/bin/bash
+# description: Basic trace clock test
+[ -f trace_clock ] || exit 1
+for c in `cat trace_clock | tr  -d \[\]`; do
+  echo $c > trace_clock || exit 1
+  grep '\['$c'\]' trace_clock || exit 1
+done
+echo local > trace_clock

[RFC PATCH 3/3] ftracetest: Add kprobe basic testcases

[Masami Hiramatsu]

Add basic testcases for kprobe dynamic events.
This also shows that the ftracetest accepts sub-directory
for new testcases.

Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
 .../testing/ftrace/test.d/kprobe/add_and_remove.tc |   11 +++++++++++
 tools/testing/ftrace/test.d/kprobe/busy_check.tc   |   14 ++++++++++++++
 2 files changed, 25 insertions(+)
 create mode 100644 tools/testing/ftrace/test.d/kprobe/add_and_remove.tc
 create mode 100644 tools/testing/ftrace/test.d/kprobe/busy_check.tc

diff --git a/tools/testing/ftrace/test.d/kprobe/add_and_remove.tc b/tools/testing/ftrace/test.d/kprobe/add_and_remove.tc
new file mode 100644
index 0000000..8e02257
--- /dev/null
+++ b/tools/testing/ftrace/test.d/kprobe/add_and_remove.tc
@@ -0,0 +1,11 @@
+#!/bin/bash
+# description: Kprobe dynamic event - adding and removing
+
+[ -f kprobe_events ] || exit 1
+
+echo 0 > events/enable || exit 1
+echo > kprobe_events || exit 1
+echo p:myevent do_fork > kprobe_events || exit 1
+grep myevent kprobe_events || exit 1
+[ -d events/kprobes/myevent ] || exit 1
+echo > kprobe_events
diff --git a/tools/testing/ftrace/test.d/kprobe/busy_check.tc b/tools/testing/ftrace/test.d/kprobe/busy_check.tc
new file mode 100644
index 0000000..57a8a2c
--- /dev/null
+++ b/tools/testing/ftrace/test.d/kprobe/busy_check.tc
@@ -0,0 +1,14 @@
+#!/bin/bash
+# description: Kprobe dynamic event - busy event check
+
+[ -f kprobe_events ] || exit 1
+
+echo 0 > events/enable || exit 1
+echo > kprobe_events || exit 1
+echo p:myevent do_fork > kprobe_events || exit 1
+[ -d events/kprobes/myevent ] || exit 1
+echo 1 > events/kprobes/myevent/enable || exit 1
+echo > kprobe_events && exit 1 # this must fail
+echo 0 > events/kprobes/myevent/enable || exit 1
+echo > kprobe_events # this must succeed
+

Re: [RFC PATCH 0/3] ftrace: Add a ftrace test collection

[Steven Rostedt]

Hi Masami,

This looks great. I'm a bit busy at the moment (just came back from
vacation, and digging myself out of the hole that left me). But I
definitely want this in. I have a bunch of tests too, that I can put on
top of this. My tests are rather hacky, and hard code a lot of stuff in
them, but they do test a bunch of features of ftrace. It shouldn't be
too hard to include them here.


On Tue, 05 Aug 2014 02:45:44 +0000
Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:

> Hi,
> 
> I'd like to introduce a collection of testcases for ftrace to
> avoid regressions.
> 
> For a long time, we've tried to stabilize and extend ftrace
> tracing infrastructure. This small test framework is a kind of
> stabilizing work for ftrace. For the first step, this series
> just introduces a few basic testcases. However, it is easy to
> add additional tests. I'd like to ask you, ftrace developers,
> to add tests for your features to ensure it will not be broken
> by future works.
> 
> ftracetest is a tiny bash script so that anyone can easily
> understand what it does. I think it is better to share and
> discuss this tests before growing it.
> 
>  - Is it enough to support bash script? (of course you can
>    invoke other commands from the script)
>  - What's the good naming method of testcases?
>  - Is any dependency check required?
> 
> BTW, I decided to put this under tools/testing/ftrace instead
> of tools/testing/selftests/, because all tests requires root
> privilege. It will be one of discussion points. Anyway,
> it is easy to integrate this to the selftests.

I agree. I think having its own directory is a good idea. Lets see what
other people think. When I get time, I'll see if I can start a branch
that pulls this in and start adding my own tests on top of it.

Thanks!

-- Steve

Re: [RFC PATCH 0/3] ftrace: Add a ftrace test collection

[Masami Hiramatsu]

(2014/08/06 6:37), Steven Rostedt wrote:
> Hi Masami,
> 
> This looks great. I'm a bit busy at the moment (just came back from
> vacation, and digging myself out of the hole that left me). But I
> definitely want this in. I have a bunch of tests too, that I can put on
> top of this. My tests are rather hacky, and hard code a lot of stuff in
> them, but they do test a bunch of features of ftrace. It shouldn't be
> too hard to include them here.

Thanks! and I found some ftrace testcases in LTP,

https://github.com/linux-test-project/ltp/tree/master/testcases/kernel/tracing/ftrace_stress_test/ftrace_stress

Most of them are for stress test, but basic ideas are good for
unit test. And IMHO, ftrace unit test should be within the kernel
tree.

> 
> 
> On Tue, 05 Aug 2014 02:45:44 +0000
> Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
> 
>> Hi,
>>
>> I'd like to introduce a collection of testcases for ftrace to
>> avoid regressions.
>>
>> For a long time, we've tried to stabilize and extend ftrace
>> tracing infrastructure. This small test framework is a kind of
>> stabilizing work for ftrace. For the first step, this series
>> just introduces a few basic testcases. However, it is easy to
>> add additional tests. I'd like to ask you, ftrace developers,
>> to add tests for your features to ensure it will not be broken
>> by future works.
>>
>> ftracetest is a tiny bash script so that anyone can easily
>> understand what it does. I think it is better to share and
>> discuss this tests before growing it.
>>
>>  - Is it enough to support bash script? (of course you can
>>    invoke other commands from the script)
>>  - What's the good naming method of testcases?
>>  - Is any dependency check required?
>>
>> BTW, I decided to put this under tools/testing/ftrace instead
>> of tools/testing/selftests/, because all tests requires root
>> privilege. It will be one of discussion points. Anyway,
>> it is easy to integrate this to the selftests.
> 
> I agree. I think having its own directory is a good idea. Lets see what
> other people think.

Actually, current sefltests provides just a space, not minimal
functions, like log management, test statistics, etc. which ftracetest
has. And I doubt that Make-based test framework is good for providing
such functions.
I think it is also another option to generalize the ftracetest script
for selftests :)

> When I get time, I'll see if I can start a branch
> that pulls this in and start adding my own tests on top of it.

I look forward to see your tests :)

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [RFC PATCH 0/3] ftrace: Add a ftrace test collection

[Namhyung Kim]

Hi Steve and Masami,

On Tue, 5 Aug 2014 17:37:52 -0400, Steven Rostedt wrote:
> Hi Masami,
>
> This looks great. I'm a bit busy at the moment (just came back from
> vacation, and digging myself out of the hole that left me). But I
> definitely want this in. I have a bunch of tests too, that I can put on
> top of this. My tests are rather hacky, and hard code a lot of stuff in
> them, but they do test a bunch of features of ftrace. It shouldn't be
> too hard to include them here.
>
>
> On Tue, 05 Aug 2014 02:45:44 +0000
> Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>
>> Hi,
>> 
>> I'd like to introduce a collection of testcases for ftrace to
>> avoid regressions.
>> 
>> For a long time, we've tried to stabilize and extend ftrace
>> tracing infrastructure. This small test framework is a kind of
>> stabilizing work for ftrace. For the first step, this series
>> just introduces a few basic testcases. However, it is easy to
>> add additional tests. I'd like to ask you, ftrace developers,
>> to add tests for your features to ensure it will not be broken
>> by future works.
>> 
>> ftracetest is a tiny bash script so that anyone can easily
>> understand what it does. I think it is better to share and
>> discuss this tests before growing it.
>> 
>>  - Is it enough to support bash script? (of course you can
>>    invoke other commands from the script)

Btw, does it use any bash-specific feature?


>>  - What's the good naming method of testcases?

I'm okay with the ftracetest, but tracing-test may be an option. :)


>>  - Is any dependency check required?

I think we need to start from no/minimum external dependency.


>> 
>> BTW, I decided to put this under tools/testing/ftrace instead
>> of tools/testing/selftests/, because all tests requires root
>> privilege. It will be one of discussion points. Anyway,
>> it is easy to integrate this to the selftests.
>
> I agree. I think having its own directory is a good idea. Lets see what
> other people think. When I get time, I'll see if I can start a branch
> that pulls this in and start adding my own tests on top of it.

I also agree to have a separate directory and it's not a selftest :)

Steve, I think you already have a lot of testcases that I want to add,
I'll take a look if you setup the branch and try to add my own if
needed.

Thanks,
Namhyung

Re: [RFC PATCH 0/3] ftrace: Add a ftrace test collection

[Namhyung Kim]

On Wed, 13 Aug 2014 15:59:04 +0900, Namhyung Kim wrote:
> On Tue, 5 Aug 2014 17:37:52 -0400, Steven Rostedt wrote:
>> Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>>>  - What's the good naming method of testcases?
>
> I'm okay with the ftracetest, but tracing-test may be an option. :)

Ouch, did you say about *.tc file name?  I have no problem with that too
anyway :)

Thanks,
Namhyung

Re: [RFC PATCH 0/3] ftrace: Add a ftrace test collection

[Masami Hiramatsu]

(2014/08/13 15:59), Namhyung Kim wrote:
> Hi Steve and Masami,
> 
> On Tue, 5 Aug 2014 17:37:52 -0400, Steven Rostedt wrote:
>> Hi Masami,
>>
>> This looks great. I'm a bit busy at the moment (just came back from
>> vacation, and digging myself out of the hole that left me). But I
>> definitely want this in. I have a bunch of tests too, that I can put on
>> top of this. My tests are rather hacky, and hard code a lot of stuff in
>> them, but they do test a bunch of features of ftrace. It shouldn't be
>> too hard to include them here.
>>
>>
>> On Tue, 05 Aug 2014 02:45:44 +0000
>> Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com> wrote:
>>
>>> Hi,
>>>
>>> I'd like to introduce a collection of testcases for ftrace to
>>> avoid regressions.
>>>
>>> For a long time, we've tried to stabilize and extend ftrace
>>> tracing infrastructure. This small test framework is a kind of
>>> stabilizing work for ftrace. For the first step, this series
>>> just introduces a few basic testcases. However, it is easy to
>>> add additional tests. I'd like to ask you, ftrace developers,
>>> to add tests for your features to ensure it will not be broken
>>> by future works.
>>>
>>> ftracetest is a tiny bash script so that anyone can easily
>>> understand what it does. I think it is better to share and
>>> discuss this tests before growing it.
>>>
>>>  - Is it enough to support bash script? (of course you can
>>>    invoke other commands from the script)
> 
> Btw, does it use any bash-specific feature?

Not much ("function" keyword is possible bashism). I usually
use bash and sometimes unintentionally use bash-specific features :)
Anyway, I guess bash is enough common now and sometimes its extensions
are good for short scripting.

>>>  - What's the good naming method of testcases?
> 
> I'm okay with the ftracetest, but tracing-test may be an option. :)

Ah, as you said, I meant its extensions *.tc.  :)

>>>  - Is any dependency check required?
> 
> I think we need to start from no/minimum external dependency.
> 
> 
>>>
>>> BTW, I decided to put this under tools/testing/ftrace instead
>>> of tools/testing/selftests/, because all tests requires root
>>> privilege. It will be one of discussion points. Anyway,
>>> it is easy to integrate this to the selftests.
>>
>> I agree. I think having its own directory is a good idea. Lets see what
>> other people think. When I get time, I'll see if I can start a branch
>> that pulls this in and start adding my own tests on top of it.
> 
> I also agree to have a separate directory and it's not a selftest :)
> 
> Steve, I think you already have a lot of testcases that I want to add,
> I'll take a look if you setup the branch and try to add my own if
> needed.

Great! That's so helpful for us :)

Thank you,

-- 
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hiramatsu.pt@hitachi.com