[PATCH] selftests: Add install, generate tar, and run_kselftest tools

[PATCH] selftests: Add install, generate tar, and run_kselftest tools

[Shuah Khan]

kselftest_install.sh tool adds support for installing selftests
at user specified location/kselftest. By default this tool
will install selftests in the selftests/kselftest directory.
For example, kselftest_install /tmp will install tests under
/tmp/kselftest

gen_kselftest_tar.sh tool will generate user specified type of
tar archive. Default is .gz

run_kselftest.sh runs all installed selftests.

Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
---
 tools/testing/selftests/gen_kselftest_tar.sh | 55 ++++++++++++++++++++++++
 tools/testing/selftests/kselftest_install.sh | 64 ++++++++++++++++++++++++++++
 tools/testing/selftests/run_kselftest.sh     | 46 ++++++++++++++++++++
 3 files changed, 165 insertions(+)
 create mode 100755 tools/testing/selftests/gen_kselftest_tar.sh
 create mode 100755 tools/testing/selftests/kselftest_install.sh
 create mode 100755 tools/testing/selftests/run_kselftest.sh

diff --git a/tools/testing/selftests/gen_kselftest_tar.sh b/tools/testing/selftests/gen_kselftest_tar.sh
new file mode 100755
index 0000000..17d5bd0
--- /dev/null
+++ b/tools/testing/selftests/gen_kselftest_tar.sh
@@ -0,0 +1,55 @@
+#!/bin/bash
+#
+# gen_kselftest_tar
+# Generate kselftest tarball
+# Author: Shuah Khan <shuahkh@osg.samsung.com>
+# Copyright (C) 2015 Samsung Electronics Co., Ltd.
+
+# This software may be freely redistributed under the terms of the GNU
+# General Public License (GPLv2).
+
+# main
+main()
+{
+	if [ "$#" -eq 0 ]; then
+		echo "$0: Generating default compression gzip"
+		copts="cvzf"
+		ext=".tar.gz"
+	else
+		case "$1" in
+			tar)
+				copts="cvf"
+				ext=".tar"
+				;;
+			targz)
+				copts="cvzf"
+				ext=".tar.gz"
+				;;
+			tarbz2)
+				copts="cvjf"
+				ext=".tar.bz2"
+				;;
+			tarxz)
+				copts="cvJf"
+				ext=".tar.xz"
+				;;
+			*)
+			echo "Unknown tarball format $1"
+			exit 1
+			;;
+	esac
+	fi
+
+	install_dir=./kselftest
+
+# Run install using INSTALL_KSFT_PATH override to generate install
+# directory
+./kselftest_install.sh
+tar $copts kselftest${ext} $install_dir
+echo "Kselftest archive kselftest${ext} created!"
+
+# clean up install directory
+rm -rf kselftest
+}
+
+main "$@"
diff --git a/tools/testing/selftests/kselftest_install.sh b/tools/testing/selftests/kselftest_install.sh
new file mode 100755
index 0000000..8af1c3c
--- /dev/null
+++ b/tools/testing/selftests/kselftest_install.sh
@@ -0,0 +1,64 @@
+#!/bin/bash
+#
+# Kselftest Install
+# Install kselftest tests
+# Author: Shuah Khan <shuahkh@osg.samsung.com>
+# Copyright (C) 2015 Samsung Electronics Co., Ltd.
+
+# This software may be freely redistributed under the terms of the GNU
+# General Public License (GPLv2).
+
+install_loc=`pwd`
+
+main()
+{
+	if [ $(basename $install_loc) !=  "selftests" ]; then
+		echo "$0: Please run it in selftests directory ..."
+		exit 1;
+	fi
+	if [ "$#" -eq 0 ]; then
+		echo "$0: Installing in default location - $install_loc ..."
+	elif [ ! -d "$1" ]; then
+		echo "$0: $1 doesn't exist!!"
+		exit 1;
+	else
+		install_loc=$1
+		echo "$0: Installing in specified location - $install_loc ..."
+	fi
+
+	install_path=$install_loc/kselftest
+	install_dir=$install_loc/kselftest
+
+# Create install directory
+	mkdir -p $install_dir
+# Build tests
+	make all
+# Installs normal tests and skips special tests and kselftest tools
+# gen_kselftesr_tar.sh and kselftes_install.sh
+# Also skips problematic xxxx* file that gets created when execveat
+# test is built. The file name is too long and resulting in error
+# messages.
+	find `pwd`/* -type d -name rcutorture -prune -o -type f \
+		-executable -print | grep -v 'tar\|install\|xxxx' | \
+		xargs install -t $install_dir
+# Install shell scripts that aren't executables
+	find `pwd`/* -type d -name rcutorture -prune -o -name "*.sh" -print | \
+		grep -v 'tar\|install\|run\|on-off' | \
+		xargs install -t $install_dir
+# Special handling for cpu-hotplug and memory-hotplug .sh with the same name
+	install `pwd`/cpu-hotplug/on-off-test.sh \
+		$install_dir/cpu-on-off-test.sh
+	install `pwd`/memory-hotplug/on-off-test.sh \
+		$install_dir/mem-on-off-test.sh
+# Special handling for scripts without .sh extension
+	install `pwd`/vm/run_vmtests $install_dir
+	install `pwd`/net/run_netsocktests $install_dir
+	install `pwd`/net/run_afpackettests $install_dir
+	install `pwd`/sysctl/common_tests $install_dir
+# Install dependent directories for ftrace and exec tests
+	cp -r `pwd`/ftrace/test.d $install_dir
+	install `pwd`/exec/execveat.denatured $install_dir
+	cp -r `pwd`/exec/subdir $install_dir
+}
+
+main "$@"
diff --git a/tools/testing/selftests/run_kselftest.sh b/tools/testing/selftests/run_kselftest.sh
new file mode 100755
index 0000000..a6d3429
--- /dev/null
+++ b/tools/testing/selftests/run_kselftest.sh
@@ -0,0 +1,46 @@
+#!/bin/bash
+#
+# Run Kselftests
+# Install kselftest tests
+# Author: Shuah Khan <shuahkh@osg.samsung.com>
+# Copyright (C) 2015 Samsung Electronics Co., Ltd.
+
+# This software may be freely redistributed under the terms of the GNU
+
+# Exclude self (run_ksefltest) from test run.
+
+# Exclude dependecies that get run from master scripts
+#    common_tests - called from run_numerictests and run_stringtests
+#    huge* scripts - run from run_vmtests
+#    socket, psock_fanout, and psock_tpacket - run from net tests
+#    create-read and open-unlink - run from efivarfs.sh
+#    execveat.denatured - run from execveat
+# Exclude tests that need special setup or arguments for short runs
+# These are run after the normal tests are run
+#    unprivileged-remount-test
+#    memory-hotplug tests
+
+# Create symlink for execveat
+ln -s -f execveat execveat.symlink
+
+echo "Start Kselftest Run"
+find . -type f -executable -print |
+	grep -v 'common\|denatured\|huge\|mem-on\|run_kselftest\|socket\|psock\|create-read\|open-unlink\|script\|unprivileged' |
+while read -r prog
+do
+	echo "Running $prog ---"
+	./"$prog"
+done
+# Run memory-hotplug test in limited scope
+echo "Running memory hotplug test"
+./mem-on-off-test.sh -r 2
+# Run vm and mount tests
+echo "Running vm test ...."
+./run_vmtests
+echo "Running mount test ...."
+if [ -f /proc/self/uid_map ] ; then ./unprivileged-remount-test ; fi
+
+# Cleanup - execveat test creates temp files which will cause problems
+# when test is re-run
+rm -r -f xxxxx*
+echo "End Kselftest Run"
-- 
2.1.0

Re: [PATCH] selftests: Add install, generate tar, and run_kselftest tools

[Dave Jones]

On Mon, Mar 02, 2015 at 09:48:08PM -0700, Shuah Khan wrote:
 > kselftest_install.sh tool adds support for installing selftests
 > at user specified location/kselftest. By default this tool
 > will install selftests in the selftests/kselftest directory.
 > For example, kselftest_install /tmp will install tests under
 > /tmp/kselftest
 
How is this an improvement over having each test install method isolated
to its own Makefile ?

 > +# Installs normal tests and skips special tests and kselftest tools
 > +# gen_kselftesr_tar.sh and kselftes_install.sh
 > +# Also skips problematic xxxx* file that gets created when execveat
 > +# test is built. The file name is too long and resulting in error
 > +# messages.
 > +	find `pwd`/* -type d -name rcutorture -prune -o -type f \
 > +		-executable -print | grep -v 'tar\|install\|xxxx' | \
 > +		xargs install -t $install_dir
 > +# Install shell scripts that aren't executables
 > +	find `pwd`/* -type d -name rcutorture -prune -o -name "*.sh" -print | \
 > +		grep -v 'tar\|install\|run\|on-off' | \
 > +		xargs install -t $install_dir
 > +# Special handling for cpu-hotplug and memory-hotplug .sh with the same name
 > +	install `pwd`/cpu-hotplug/on-off-test.sh \
 > +		$install_dir/cpu-on-off-test.sh
 > +	install `pwd`/memory-hotplug/on-off-test.sh \
 > +		$install_dir/mem-on-off-test.sh
 > +# Special handling for scripts without .sh extension
 > +	install `pwd`/vm/run_vmtests $install_dir
 > +	install `pwd`/net/run_netsocktests $install_dir
 > +	install `pwd`/net/run_afpackettests $install_dir
 > +	install `pwd`/sysctl/common_tests $install_dir
 > +# Install dependent directories for ftrace and exec tests
 > +	cp -r `pwd`/ftrace/test.d $install_dir
 > +	install `pwd`/exec/execveat.denatured $install_dir
 > +	cp -r `pwd`/exec/subdir $install_dir

This already makes my eyes hurt, and is only going to grow as more tests
get added. Additionally when two people are trying to add tests at the
same time, you're guaranteeing future merge conflicts here.

	Dave

Re: [PATCH] selftests: Add install, generate tar, and run_kselftest tools

[Shuah Khan]

On 03/03/2015 07:49 AM, Dave Jones wrote:
> On Mon, Mar 02, 2015 at 09:48:08PM -0700, Shuah Khan wrote:
>  > kselftest_install.sh tool adds support for installing selftests
>  > at user specified location/kselftest. By default this tool
>  > will install selftests in the selftests/kselftest directory.
>  > For example, kselftest_install /tmp will install tests under
>  > /tmp/kselftest
>  
> How is this an improvement over having each test install method isolated
> to its own Makefile ?

Dave/Michael,

Makefile approach requires changes to all the existing test Makefiles.
After looking at the churn to individual Makefiles, I have the following
concerns.

I am concerned about maintenance and potential for mistakes in install
logic in individual Makefiles when new tests get added. I keep seeing
run_tests target breaking when new tests get added and also when
existing tests get modified.

That said, I looked at Michael's patches and Michael's work does address
several of my concerns. Hence, the following plan:

I will take the following patches from Michael after requested
changes are made:

- [PATCH 1/9] selftests: Introduce minimal shared logic for
  running tests
  This improves current run_tests logic. Will need changes to
  account for duplicate cpu and memory hot-plug scripts. Both are
  named on-off-test.sh - won't make a difference for this patch,
  but will for install logic

Note: I am seeing failures when I run sudo make kselftest after
applying this patch

/bin/sh: 1: ./run_netsocktests: Permission denied
selftests: run_netsocktests [FAIL]
/bin/sh: 1: ./run_afpackettests: Permission denied
selftests: run_afpackettests [FAIL]

/bin/sh: 1: ./run_numerictests: Permission denied
selftests: run_numerictests [FAIL]
/bin/sh: 1: ./run_stringtests: Permission denied
selftests: run_stringtests [FAIL]

/bin/sh: 1: ./run_vmtests: Permission denied

  Please make sure make kselftest doesn't regress when run
  as root as well as user. In addition, the following don't
  regress:

  make -C tools/testing/selftests TARGETS=test1 run_tests
  make -C tools/testing/selftests TARGETS="test1 test2" run_tests
  make -C tools/testing/selftests run_hotplug

  Please see Documentation/kselftest.txt - don't want to regress
  the current use-cases.

- [PATCH 2/9] selftests: Add install target
  Looks like lib.mk logic is addressing some of my concerns about the
  individual Makefile install logic.
  I would like to see 1. the all script name changed to run_kselftest.sh

- [PATCH 3/9] selftests: Add install support for the powerpc tests
  This is good as is.

- [PATCH 5/9] kbuild: Don't pass -rR to selftest makefiles
  Drop kselftest_install from this patch. There is no need.
  More on this below.

- PATCH 6/9] selftests: Set CC using CROSS_COMPILE once in lib.mk

- [PATCH 7/9] selftests/timers: Use implicit rules
  Please check John Stultz's timers test work to make sure there is no
  conflict. Please see: [PATCH 01/19] selftests/timers: Cleanup
  Makefile to make it easier to add future tests
  https://lkml.org/lkml/2015/2/5/56

- [PATCH 8/9] selftests/mqueue: Use implicit rules
  This is good as is.

- [PATCH 9/9] selftests/mount: Use implicit rules
  This is good as is.

Drop these patches:
- [PATCH 4/9] kbuild: add a new kselftest_install make target to install
selftests
  I am not seeing any value in adding install target to the main
  Makefile. Invoking test run makes sense as it allows user
  to run them from the git without install step which makes sense.
  Being able to invoke install from main Makefile really doesn't
  add any value. We can isolate install logic under selftests and
  also avoid adding one more target to the main Makefile.

I still want a wrapper script for install, so:

- I will change kselftest_install.sh to leverage the above work. It can
  just invoke make install at the selftests directory after figuring
  base directory logic etc. This will be based on the above patches.

- Keep  gen_kselftest_tar.sh as is - no changes need.

- I can drop run_kselftest tool completely, since emit logic in
  Michael's work takes care of it.

Comments and questions? I am cc'ing Michal Marek to keep him in
the loop.

Michael! Please let me know if you want to see responses to your
individual patches, in addition to this email.

thanks,
-- Shuah

-- 
Shuah Khan
Sr. Linux Kernel Developer
Open Source Innovation Group
Samsung Research America (Silicon Valley)
shuahkh@osg.samsung.com | (970) 217-8978

Re: [PATCH] selftests: Add install, generate tar, and run_kselftest tools

[Michael Ellerman]

On Tue, 2015-03-03 at 10:07 -0700, Shuah Khan wrote:
> On 03/03/2015 07:49 AM, Dave Jones wrote:
> > On Mon, Mar 02, 2015 at 09:48:08PM -0700, Shuah Khan wrote:
> >  > kselftest_install.sh tool adds support for installing selftests
> >  > at user specified location/kselftest. By default this tool
> >  > will install selftests in the selftests/kselftest directory.
> >  > For example, kselftest_install /tmp will install tests under
> >  > /tmp/kselftest
> >  
> > How is this an improvement over having each test install method isolated
> > to its own Makefile ?
> 
> Dave/Michael,
> 
> Makefile approach requires changes to all the existing test Makefiles.
> After looking at the churn to individual Makefiles, I have the following
> concerns.
> 
> I am concerned about maintenance and potential for mistakes in install
> logic in individual Makefiles when new tests get added. I keep seeing
> run_tests target breaking when new tests get added and also when
> existing tests get modified.

With most of the logic isolated in lib.mk I don't think this is a concern.

As an example here is the minimal Makefile required for a simple new test:

	$ cat tools/testing/selftests/hello-world/Makefile
	TEST_PROGS := hello-world
	
	all: $(TEST_PROGS)
	
	include ../lib.mk
	
	clean:
		rm -f $(TEST_PROGS)


That will build, work with run_tests and work with install.

In fact we can merge a template directory like the above with an example
Makefile to make it super easy for people to add new tests with the right
logic.


> That said, I looked at Michael's patches and Michael's work does address
> several of my concerns. Hence, the following plan:
> 
> I will take the following patches from Michael after requested
> changes are made:
> 
> - [PATCH 1/9] selftests: Introduce minimal shared logic for
>   running tests
>   This improves current run_tests logic. Will need changes to
>   account for duplicate cpu and memory hot-plug scripts. Both are
>   named on-off-test.sh - won't make a difference for this patch,
>   but will for install logic

The duplicate names don't matter because I put the installed tests in a sub
directory:

$ cd tools/testing/selftests
$ find install/ -name on-off-test.sh
install/memory-hotplug/on-off-test.sh
install/cpu-hotplug/on-off-test.sh


> Note: I am seeing failures when I run sudo make kselftest after
> applying this patch
> 
> /bin/sh: 1: ./run_netsocktests: Permission denied
> selftests: run_netsocktests [FAIL]
> /bin/sh: 1: ./run_afpackettests: Permission denied
> selftests: run_afpackettests [FAIL]
> 
> /bin/sh: 1: ./run_numerictests: Permission denied
> selftests: run_numerictests [FAIL]
> /bin/sh: 1: ./run_stringtests: Permission denied
> selftests: run_stringtests [FAIL]
> 
> /bin/sh: 1: ./run_vmtests: Permission denied

Ah sorry, I lost the change to the permission bits somewhere along the line.
Thanks for testing. Fixed in v3.


>   Please make sure make kselftest doesn't regress when run
>   as root as well as user. In addition, the following don't
>   regress:
> 
>   make -C tools/testing/selftests TARGETS=test1 run_tests
>   make -C tools/testing/selftests TARGETS="test1 test2" run_tests
>   make -C tools/testing/selftests run_hotplug
> 
>   Please see Documentation/kselftest.txt - don't want to regress
>   the current use-cases.

None of those have regressed as far as I can see.


> - [PATCH 2/9] selftests: Add install target
>   Looks like lib.mk logic is addressing some of my concerns about the
>   individual Makefile install logic.
>   I would like to see 1. the all script name changed to run_kselftest.sh

I don't see why it needs "kselftest" in the name, it's *inside* the install
directory, but I'll rename it because I don't care that much.


> - [PATCH 5/9] kbuild: Don't pass -rR to selftest makefiles
>   Drop kselftest_install from this patch. There is no need.
>   More on this below.

OK. That makes this patch orthogonal to the rest, and it's actually a bug fix,
so I'll send it separately and ideally you can merge it for 4.0.


> - [PATCH 7/9] selftests/timers: Use implicit rules
>   Please check John Stultz's timers test work to make sure there is no
>   conflict. Please see: [PATCH 01/19] selftests/timers: Cleanup
>   Makefile to make it easier to add future tests
>   https://lkml.org/lkml/2015/2/5/56

There will definitely be a conflict. As the maintainer you should either fixup
the conflict, or if you can't, merge one series and then ask the author of the
other series to base their work on top of the other.

In this case the two patches are very similar, so it probably doesn't matter
which you merge first. You'll just have to incorporate the CFLAGS changes from
John's patch into the final result.

> Drop these patches:
> - [PATCH 4/9] kbuild: add a new kselftest_install make target to install selftests
>   I am not seeing any value in adding install target to the main
>   Makefile.

Fine by me.


> I still want a wrapper script for install, so:
> 
> - I will change kselftest_install.sh to leverage the above work. It can
>   just invoke make install at the selftests directory after figuring
>   base directory logic etc. This will be based on the above patches.
> 
> - Keep  gen_kselftest_tar.sh as is - no changes need.
> 
> - I can drop run_kselftest tool completely, since emit logic in
>   Michael's work takes care of it.
> 
> Comments and questions? I am cc'ing Michal Marek to keep him in
> the loop.
> 
> Michael! Please let me know if you want to see responses to your
> individual patches, in addition to this email.

No that's fine.

I'll repost the series shortly.

cheers