[LTP] [PATCH 1/2] docs: Add running the tests section

[LTP] [PATCH 1/2] docs: Add running the tests section

[Richard Palethorpe]

The simplest/shortest possible guide to installing from source and running the tests.

Signed-off-by: Richard Palethorpe <rpalethorpe@suse.com>
---

In the common case, LTP is now easy to install and use. The documentation is
also thorough. However I have had feedback from a few developers that they
find installing the LTP and running a single test very difficult.

I think the reason for this is that the documentation is too spread out and
difficult to navigate. Even while writing this I discovered new things which I
have not seen before.

This guide is aimed at the casual LTP user, who just wants to fix some failing
test case (usually from syscalls), so their work flow will be:

* Patch Kernel
* Install it in VM
* Install LTP in VM
* Run test to see if it still fails

Users who are just curious will probably just want to run syscalls as well.

I think it is best to have a short end-to-end guide as the first thing people
see when coming to the project. Then link off into more detailed documentation
for people who fall off the happy path.

 README | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

diff --git a/README b/README
index bf0061c41..5c3236783 100644
--- a/README
+++ b/README
@@ -30,6 +30,62 @@ Don't run them on production systems. Growfiles, doio, and iogen in particular
 stress the I/O capabilities of systems and while they should not cause problems
 on properly functioning systems, they are intended to find (or cause) problems.
 
+Quick guide to running the tests
+--------------------------------
+
+If you have Git, autoconf, automake, m4, the linux headers and the common
+developer packages installed, the chances are the following will
+work.
+
+$ git clone https://github.com/the-linux-test-project/ltp.git
+$ cd ltp
+$ make autotools
+$ ./configure
+$ make
+$ make install
+
+This will install LTP to /opt/ltp.
+- If you have a problem see doc/mini-howto-building-ltp-from-git.txt.
+- If you still have a problem see INSTALL and './configure --help'.
+- Failing that, ask for help on the mailing list or Github.
+
+Some tests will be disabled if the configure script can not find their build
+dependencies.
+- If a test fails due to a missing component, check the ./configure output.
+- If a tests fails due to a missing user or group, see the Quick Start section
+  of INSTALL.
+
+To run all the test suites
+
+$ cd /opt/ltp
+$ ./runltp
+
+Note that many test cases have to be executed as root.
+
+To run a particular test suite
+
+$ ./runltp -f syscalls
+
+To run all tests with 'madvise' in the name
+
+$ ./runltp -f syscalls -s madvise
+
+Also see
+
+$ ./runltp --help
+
+Test suites (e.g. syscalls) are defined in the runtest directory. Each file
+contains a list of test cases in a simple format, see doc/ltp-run-files.txt.
+
+Each test case has its own executable or script, these can be executed
+directly
+
+$ testcases/bin/abort01
+
+Some have arguments
+
+$ testcases/bin/fork13 -i 37
+
 Developers corner
 -----------------
 
-- 
2.12.2

[LTP] [PATCH 2/2] docs: Add description of runtest file format

[Richard Palethorpe]

An informal description of the runtest file contents.

Signed-off-by: Richard Palethorpe <rpalethorpe@suse.com>
---
 doc/ltp-run-files.txt | 41 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 40 insertions(+), 1 deletion(-)

diff --git a/doc/ltp-run-files.txt b/doc/ltp-run-files.txt
index 36180b97b..f8cc36b9e 100644
--- a/doc/ltp-run-files.txt
+++ b/doc/ltp-run-files.txt
@@ -1,4 +1,43 @@
-List of scripts or executables that need to be run for complete LTP execution.
+The runtest files contain a list of test cases to be executed.
+
+File Format
+-----------
+
+Lines starting with a '#' are comments and blank lines are ignored.
+
+Otherwise, lines start with a test name followed by white space, then some
+shell script to be executed. For example
+
+Test Name
+|       Delimiter               Test case argument
+|       |                       |
+v       v                       v
+splice02 seq 1 20000 | splice02 splice02-temp
+         ^             ^                    ^
+         |             |                    |
+         |             Test case executable |
+         -----------Shell script-------------
+
+So the splice02 runtest entry pipes the output of seq into the splice02 test
+executable. Most runtest entries are simpler than this, for example
+
+splice03 splice03
+
+Here the test name and executable have the same name and no arguments have
+been supplied.
+
+Run test files should start with a comment describing the tests they contain,
+e.g.
+
+#DESCRIPTION:Kernel system calls
+
+Note that the LTP has absorbed a number of other projects. Some of these have
+been fully converted to the LTP format, others have runtest files generated
+for them during installation, while some use a shell script to integrate them
+with the other tests.
+
+Test suites
+-----------
 
  - syscalls  (except epoll, see below)
  - fs
-- 
2.12.2

[LTP] [PATCH 1/2] docs: Add running the tests section

[Cyril Hrubis]

Hi!
> The simplest/shortest possible guide to installing from source and running the tests.

This is a great addition, thanks for writing it, a few comments below.

> Signed-off-by: Richard Palethorpe <rpalethorpe@suse.com>
> ---
> 
> In the common case, LTP is now easy to install and use. The documentation is
> also thorough. However I have had feedback from a few developers that they
> find installing the LTP and running a single test very difficult.
> 
> I think the reason for this is that the documentation is too spread out and
> difficult to navigate. Even while writing this I discovered new things which I
> have not seen before.
> 
> This guide is aimed at the casual LTP user, who just wants to fix some failing
> test case (usually from syscalls), so their work flow will be:
> 
> * Patch Kernel
> * Install it in VM
> * Install LTP in VM
> * Run test to see if it still fails
> 
> Users who are just curious will probably just want to run syscalls as well.
> 
> I think it is best to have a short end-to-end guide as the first thing people
> see when coming to the project. Then link off into more detailed documentation
> for people who fall off the happy path.
> 
>  README | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 56 insertions(+)
> 
> diff --git a/README b/README
> index bf0061c41..5c3236783 100644
> --- a/README
> +++ b/README
> @@ -30,6 +30,62 @@ Don't run them on production systems. Growfiles, doio, and iogen in particular
>  stress the I/O capabilities of systems and while they should not cause problems
>  on properly functioning systems, they are intended to find (or cause) problems.
>  
> +Quick guide to running the tests
> +--------------------------------
> +
> +If you have Git, autoconf, automake, m4, the linux headers and the common
> +developer packages installed, the chances are the following will
> +work.
> +
> +$ git clone https://github.com/the-linux-test-project/ltp.git
> +$ cd ltp
> +$ make autotools
> +$ ./configure
> +$ make
> +$ make install
> +
> +This will install LTP to /opt/ltp.
> +- If you have a problem see doc/mini-howto-building-ltp-from-git.txt.
> +- If you still have a problem see INSTALL and './configure --help'.
> +- Failing that, ask for help on the mailing list or Github.
> +
> +Some tests will be disabled if the configure script can not find their build
> +dependencies.
> +- If a test fails due to a missing component, check the ./configure output.
               ^
	  Common case here is that test is skipped these days, i.e.
	  reports TCONF.

> +- If a tests fails due to a missing user or group, see the Quick Start section
> +  of INSTALL.
> +
> +To run all the test suites
> +
> +$ cd /opt/ltp
> +$ ./runltp
> +
> +Note that many test cases have to be executed as root.
> +
> +To run a particular test suite
> +
> +$ ./runltp -f syscalls
> +
> +To run all tests with 'madvise' in the name
> +
> +$ ./runltp -f syscalls -s madvise
> +
> +Also see
> +
> +$ ./runltp --help
> +
> +Test suites (e.g. syscalls) are defined in the runtest directory. Each file
> +contains a list of test cases in a simple format, see doc/ltp-run-files.txt.
> +
> +Each test case has its own executable or script, these can be executed
> +directly
> +
> +$ testcases/bin/abort01
> +
> +Some have arguments
> +
> +$ testcases/bin/fork13 -i 37

Quite a lot of the C tests and all shell tests needs at least
/opt/ltp/testcases/bin in the $PATH in order to source scripts and/or
execute helper binaries. Some tests needs LTPROOT (in order to locate
datafiles) as well.

So safest example here is:

LTPROOT=/opt/ltp/ PATH="$PATH:$LTPROOT/testcases/bin" testcases/bin/test_name

Also 99% of the test has help that is printed when -h is passed in
arguments.

>  Developers corner
>  -----------------
>  
> -- 
> 2.12.2
> 
> 
> -- 
> Mailing list info: https://lists.linux.it/listinfo/ltp

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [PATCH 1/2] docs: Add running the tests section

[Richard Palethorpe]

Cyril Hrubis writes:

> Hi!
>> The simplest/shortest possible guide to installing from source and running the tests.
>
> This is a great addition, thanks for writing it, a few comments below.

Thanks for cleaning up the LTP ;-)

I also think it would be a good idea to do a step-by-step tutorial on
writing a simple test case. Then do another one which extends it to fork
a child.

The current guidelines are quite abstract, which is nice as a reference,
but I think most people learn quicker with examples as well and maybe it
would reduce the amount of review time.

>> +
>> +To run a particular test suite
>> +
>> +$ ./runltp -f syscalls
>> +
>> +To run all tests with 'madvise' in the name
>> +
>> +$ ./runltp -f syscalls -s madvise
>> +
>> +Also see
>> +
>> +$ ./runltp --help
>> +
>> +Test suites (e.g. syscalls) are defined in the runtest directory. Each file
>> +contains a list of test cases in a simple format, see doc/ltp-run-files.txt.
>> +
>> +Each test case has its own executable or script, these can be executed
>> +directly
>> +
>> +$ testcases/bin/abort01
>> +
>> +Some have arguments
>> +
>> +$ testcases/bin/fork13 -i 37
>
> Quite a lot of the C tests and all shell tests needs at least
> /opt/ltp/testcases/bin in the $PATH in order to source scripts and/or
> execute helper binaries. Some tests needs LTPROOT (in order to locate
> datafiles) as well.
>
> So safest example here is:
>
> LTPROOT=/opt/ltp/ PATH="$PATH:$LTPROOT/testcases/bin" testcases/bin/test_name
>
> Also 99% of the test has help that is printed when -h is passed in
> arguments.

As it happens though abort01 and fork13 don't require anything. So I
propose making it a bit longer by adding this:

"The vast majority of test cases accept the -h (help) switch

$ testcases/bin/ioctl01 -h

Many require certain environment variables to be set

$ LTPROOT=/opt/ltp PATH="$PATH:$LTPROOT/testcases/bin" testcases/bin/wc01.sh

Most commonly the path variable needs to be set and also LTPROOT, but there are a number
of other variables. runltp usually sets these for you.

Note that, in the example above, a shell script is given and indeed all
shell scripts need the PATH to be set. However this is not limited to shell
scripts, many C based tests need these variables as well."

--
Thank you,
Richard.

[LTP] [PATCH 1/2] docs: Add running the tests section

[Cyril Hrubis]

Hi!
> > This is a great addition, thanks for writing it, a few comments below.
> 
> Thanks for cleaning up the LTP ;-)
> 
> I also think it would be a good idea to do a step-by-step tutorial on
> writing a simple test case. Then do another one which extends it to fork
> a child.

Feel free to write these.

> The current guidelines are quite abstract, which is nice as a reference,
> but I think most people learn quicker with examples as well and maybe it
> would reduce the amount of review time.

Fair enough.

> >> +
> >> +To run a particular test suite
> >> +
> >> +$ ./runltp -f syscalls
> >> +
> >> +To run all tests with 'madvise' in the name
> >> +
> >> +$ ./runltp -f syscalls -s madvise
> >> +
> >> +Also see
> >> +
> >> +$ ./runltp --help
> >> +
> >> +Test suites (e.g. syscalls) are defined in the runtest directory. Each file
> >> +contains a list of test cases in a simple format, see doc/ltp-run-files.txt.
> >> +
> >> +Each test case has its own executable or script, these can be executed
> >> +directly
> >> +
> >> +$ testcases/bin/abort01
> >> +
> >> +Some have arguments
> >> +
> >> +$ testcases/bin/fork13 -i 37
> >
> > Quite a lot of the C tests and all shell tests needs at least
> > /opt/ltp/testcases/bin in the $PATH in order to source scripts and/or
> > execute helper binaries. Some tests needs LTPROOT (in order to locate
> > datafiles) as well.
> >
> > So safest example here is:
> >
> > LTPROOT=/opt/ltp/ PATH="$PATH:$LTPROOT/testcases/bin" testcases/bin/test_name
> >
> > Also 99% of the test has help that is printed when -h is passed in
> > arguments.
> 
> As it happens though abort01 and fork13 don't require anything. So I
> propose making it a bit longer by adding this:
> 
> "The vast majority of test cases accept the -h (help) switch
> 
> $ testcases/bin/ioctl01 -h
> 
> Many require certain environment variables to be set
> 
> $ LTPROOT=/opt/ltp PATH="$PATH:$LTPROOT/testcases/bin" testcases/bin/wc01.sh
> 
> Most commonly the path variable needs to be set and also LTPROOT, but there are a number
> of other variables. runltp usually sets these for you.
> 
> Note that, in the example above, a shell script is given and indeed all
> shell scripts need the PATH to be set. However this is not limited to shell
> scripts, many C based tests need these variables as well."

I find the last paragraph hard to parse. Maybe dropping the "and indeed"
and starting new sentence with "All shell scripts" would help a bit.

Otherwise this looks good.

-- 
Cyril Hrubis
chrubis@suse.cz