[Buildroot] Test infrastructure

[Buildroot] Test infrastructure

[Arnout Vandecappelle]

 This patch series adds test infrastructure to buildroot.  On the
buildroot developer day we had a brief discussion about testing.  The
main conclusion was that we'd set up a site to log test results (an
action point for Peter that is still open).  To make it easier for
people to contribute to those test results, these patches create some
test infrastructure.

 For the time being, it contains just the defconfigs.  The idea is to
add more test configurations.  For instance, Peter and ThomasP could
add the test configurations they already use.

 Regards,
 Arnout

[Buildroot] [PATCH 1/2] Add target to print buildroot version

[Arnout Vandecappelle]

From: "Arnout Vandecappelle (Essensium/Mind)" <arnout@mind.be>

It's convenient to have a target to print the buildroot version, for use
in external scripts calling buildroot.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 Makefile |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/Makefile b/Makefile
index d508888..30099d9 100644
--- a/Makefile
+++ b/Makefile
@@ -48,7 +48,7 @@ export BR2_VERSION_FULL:=$(BR2_VERSION)$(shell $(TOPDIR)/support/scripts/setloca
 noconfig_targets:=menuconfig nconfig gconfig xconfig config oldconfig randconfig \
 	defconfig %_defconfig savedefconfig allyesconfig allnoconfig silentoldconfig release \
 	randpackageconfig allyespackageconfig allnopackageconfig \
-	source-check
+	source-check print-version
 
 # Strip quotes and then whitespaces
 qstrip=$(strip $(subst ",,$(1)))
@@ -721,6 +721,9 @@ release:
 	bzip2 -9 -c < $(OUT).tar > $(OUT).tar.bz2
 	rm -rf $(OUT) $(OUT).tar
 
+print-version:
+	@echo $(BR2_VERSION_FULL)
+
 ################################################################################
 # GENDOC -- generates the make targets needed to build a specific type of
 #           asciidoc documentation.
-- 
1.7.9.1

[Buildroot] [PATCH 2/2] Add test infrastructure

[Arnout Vandecappelle]

From: "Arnout Vandecappelle (Essensium/Mind)" <arnout@mind.be>

The test infrastructure is a script and a list of tests to be run.
Additional test configurations can be added later to the test directory.

Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 test/runtests.sh |  106 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 test/testspecs   |   24 ++++++++++++
 2 files changed, 130 insertions(+), 0 deletions(-)
 create mode 100755 test/runtests.sh
 create mode 100644 test/testspecs

diff --git a/test/runtests.sh b/test/runtests.sh
new file mode 100755
index 0000000..2f547eb
--- /dev/null
+++ b/test/runtests.sh
@@ -0,0 +1,106 @@
+#! /bin/sh
+# Run test builds of buildroot
+# runtest.sh <testspec>...
+#
+# testspec:  <baseconfig>[=<type>,...]
+#   A testspec is a config file name (relative to the buildroot top dir),
+#   optionally followed by a comma-separated list of config types.
+#   Config types are: def, rand, allyes, allno, randpackage, allyespackage,
+#                     allnopackage
+#   If config types are not given, it is taken from a line containing
+#   CONFIGTYPES= in the baseconfig.  If that is also not present, def
+#   and randpackage are tried.
+#
+# If no arguments are given, the testspecs are taken from the testspecs
+# file in the current directory.  If that file doesn't exist, the testspecs
+# file in the test directory is taken.
+#
+# The output goes into the current directory.
+
+testdir=$(dirname $0)
+topdir=$(cd $testdir/..; pwd)
+
+configdir=$PWD/gen-config
+brversion=$(make -C $topdir --no-print-directory print-version)
+
+echo "Running tests for $brversion"
+
+# Run build for config $1 with type $2
+function runbuild() {
+	baseconfig=$1
+	configtype=$2
+
+	echo "Building $baseconfig ${configtype}config"
+
+	# Interprete baseconfig relative to topdir
+	if [ "${baseconfig:0:1}" = / ]; then
+		baseconfigfile=$baseconfig
+	else
+		baseconfigfile=${topdir}/${baseconfig}
+	fi
+
+	if [ ! -r $baseconfigfile ]; then
+		echo "  *** $baseconfig not found!"
+		return 1
+	fi
+
+	# Create the configtype from baseconfig
+	configname=${brversion}-$(basename $baseconfig)
+	hostdir=${PWD}/${configname}.hostdir
+	mkdir -p $configdir
+	cp $baseconfigfile $configdir/.config
+	echo "BR2_HOST_DIR=\"${hostdir}\"" >> $configdir/.config
+	make -C $topdir --no-print-directory O=$configdir ${configtype}config > /dev/null || {
+		echo "  *** make ${configtype}config failed for $baseconfig"
+		return 1
+	}
+
+	# Add the sha of the config file to its name, so we can uniquely
+	# identify it.  This allows us to remove duplicates, e.g. defconfig
+	# and allnoconfig.
+	fullconfigname=${configname}-$(sha1sum $configdir/.config | cut -c -40)
+	if [ -e ${fullconfigname}.success ]; then
+		echo "  ${fullconfigname} already built successfully; skipping"
+		return 0
+	fi
+
+	# Save the defconfig of this config for future reference.
+	make -C $topdir --no-print-directory O=$configdir savedefconfig > /dev/null || {
+		echo "  *** Failed savedefconfig for $fullconfigname"
+		return 1
+	}
+	cp $configdir/defconfig $fullconfigname.defconfig
+
+	outdir=${PWD}/${fullconfigname}.output
+	logfile=${fullconfigname}.log
+	mkdir -p $outdir
+	cp $configdir/.config $outdir
+	make -C $topdir O=$outdir > $logfile 2>&1 || {
+		echo "  *** Failed $fullconfigname - see log file for details"
+		touch $fullconfigname.failed
+		#rm -rf $outdir
+		return 1
+	}
+
+	echo "  Successfully built $baseconfig ${configtype}config"
+	touch $fullconfigname.success
+	rm -rf $outdir
+	bzip2 $logfile
+}
+
+
+if [ "$1" ]; then
+	testspecs="$*"
+elif [ -r testspecs ]; then
+	testspecs="$(cat testspecs)"
+else
+	testspecs="$(cat $testdir/testspecs)"
+fi
+
+for testspec in $testspecs; do
+	baseconfig=${testspec%=*}
+	configtypes=${testspec#*=}
+	for configtype in ${configtypes//,/ }; do
+		runbuild $baseconfig $configtype
+	done
+done
diff --git a/test/testspecs b/test/testspecs
new file mode 100644
index 0000000..2e3be3d
--- /dev/null
+++ b/test/testspecs
@@ -0,0 +1,24 @@
+configs/armadeus_apf9328_defconfig=def,randpackage
+configs/at91rm9200df_defconfig=def,randpackage
+configs/at91sam9260dfc_defconfig=def,randpackage
+configs/at91sam9261ek_defconfig=def,randpackage
+configs/at91sam9263ek_defconfig=def,randpackage
+configs/at91sam9g20dfc_defconfig=def,randpackage
+configs/atngw100_defconfig=def,randpackage
+configs/atstk100x_defconfig=def,randpackage
+configs/ea3250_defconfig=def,randpackage
+configs/fdi3250_defconfig=def,randpackage
+configs/integrator926_defconfig=def,randpackage
+configs/kb9202_defconfig=def,randpackage
+configs/mini2440_defconfig=def,randpackage
+configs/mx53loco_defconfig=def,randpackage
+configs/phy3250_defconfig=def,randpackage
+configs/qemu_arm_versatile_defconfig=def,randpackage
+configs/qemu_mips_malta_defconfig=def,randpackage
+configs/qemu_mipsel_malta_defconfig=def,randpackage
+configs/qemu_ppc_g3beige_defconfig=def,randpackage
+configs/qemu_sh4_r2d_defconfig=def,randpackage
+configs/qemu_sparc_ss10_defconfig=def,randpackage
+configs/qemu_x86_64_defconfig=def,randpackage
+configs/qemu_x86_defconfig=def,randpackage
+configs/sheevaplug_defconfig=def,randpackage
-- 
1.7.9.1

[Buildroot] [PATCH 2/2] Add test infrastructure

[Yann E. MORIN]

Arnout, All,

On Sunday 04 March 2012 00:45:07 Arnout Vandecappelle (Essensium/Mind) wrote:
> From: "Arnout Vandecappelle (Essensium/Mind)" <arnout@mind.be>
> 
> The test infrastructure is a script and a list of tests to be run.
> Additional test configurations can be added later to the test directory.
> 
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---
>  test/runtests.sh |  106 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  test/testspecs   |   24 ++++++++++++
>  2 files changed, 130 insertions(+), 0 deletions(-)
>  create mode 100755 test/runtests.sh
>  create mode 100644 test/testspecs
> 
> diff --git a/test/runtests.sh b/test/runtests.sh
> new file mode 100755
> index 0000000..2f547eb
> --- /dev/null
> +++ b/test/runtests.sh
> @@ -0,0 +1,106 @@
> +#! /bin/sh

/bin/sh is expected to be a POSIX shell, so it has an impact on the
syntax you can use. See below for a bashism I spotted.

> +# Run test builds of buildroot
> +# runtest.sh <testspec>...
> +#
> +# testspec:  <baseconfig>[=<type>,...]
> +#   A testspec is a config file name (relative to the buildroot top dir),
> +#   optionally followed by a comma-separated list of config types.
> +#   Config types are: def, rand, allyes, allno, randpackage, allyespackage,
> +#                     allnopackage
> +#   If config types are not given, it is taken from a line containing
> +#   CONFIGTYPES= in the baseconfig.  If that is also not present, def
> +#   and randpackage are tried.
> +#
> +# If no arguments are given, the testspecs are taken from the testspecs
> +# file in the current directory.  If that file doesn't exist, the testspecs
> +# file in the test directory is taken.
> +#
> +# The output goes into the current directory.
> +
> +testdir=$(dirname $0)
> +topdir=$(cd $testdir/..; pwd)

I would highly suggest using ${foo} instead of $foo because it makes
it explicit what the variable name is. Also, enclose variable between
double-quotes, in case a vicious user uses a dir with a space in it.

This comment is valid throughout the script.

> +configdir=$PWD/gen-config
> +brversion=$(make -C $topdir --no-print-directory print-version)
> +
> +echo "Running tests for $brversion"

Nowadays, I favour 'printf' over 'echo'.
'echo' is deprecated, see section "Application Usage":
    http://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html

But that's personal opinion... ;-)

> +# Run build for config $1 with type $2
> +function runbuild() {
> +	baseconfig=$1
> +	configtype=$2
> +
> +	echo "Building $baseconfig ${configtype}config"
> +
> +	# Interprete baseconfig relative to topdir
> +	if [ "${baseconfig:0:1}" = / ]; then

You're using /bin/sh, so you can not use bashisms: the syntax ${foo:i:j}
is a bashism.

See the "The Open Group Base Specifications Issue 7", utilities chapter 2,
dealing about "Shell Command Language":
    http://pubs.opengroup.org/onlinepubs/9699919799/toc.htm
    http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html

[--SNIP--]

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 2/2] Add test infrastructure

[Arnout Vandecappelle]

On Sunday 04 March 2012 01:13:27 Yann E. MORIN wrote:

[snip]
> > +testdir=$(dirname $0)
> > +topdir=$(cd $testdir/..; pwd)
> 
> I would highly suggest using ${foo} instead of $foo because it makes
> it explicit what the variable name is. Also, enclose variable between
> double-quotes, in case a vicious user uses a dir with a space in it.

 Spaces will make 'make' barf, so aren't supported anyway.  But * and
other ugly stuff would indeed be possible, so I've added quotes
everywhere.

[snip]
> See the "The Open Group Base Specifications Issue 7", utilities chapter 2,
> dealing about "Shell Command Language":
>     http://pubs.opengroup.org/onlinepubs/9699919799/toc.htm
>     http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html

 That's a useful reference!  I'll look at that more often, thanks!

 Regards,
 Arnout

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 2/2] Add test infrastructure

[Jean-Christophe PLAGNIOL-VILLARD]

Hi, 

	I like the idea of test but what is the target here?

	if it's build multiple config take a look on the MAKEALL of barebox

	or we could set a buildbot

Best Regards,
J.
On 00:45 Sun 04 Mar     , Arnout Vandecappelle (Essensium/Mind) wrote:
> From: "Arnout Vandecappelle (Essensium/Mind)" <arnout@mind.be>
> 
> The test infrastructure is a script and a list of tests to be run.
> Additional test configurations can be added later to the test directory.
> 
> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---
>  test/runtests.sh |  106 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  test/testspecs   |   24 ++++++++++++
>  2 files changed, 130 insertions(+), 0 deletions(-)
>  create mode 100755 test/runtests.sh
>  create mode 100644 test/testspecs
> 
> diff --git a/test/runtests.sh b/test/runtests.sh
> new file mode 100755
> index 0000000..2f547eb
> --- /dev/null
> +++ b/test/runtests.sh
> @@ -0,0 +1,106 @@
> +#! /bin/sh
> +# Run test builds of buildroot
> +# runtest.sh <testspec>...
> +#
> +# testspec:  <baseconfig>[=<type>,...]
> +#   A testspec is a config file name (relative to the buildroot top dir),
> +#   optionally followed by a comma-separated list of config types.
> +#   Config types are: def, rand, allyes, allno, randpackage, allyespackage,
> +#                     allnopackage
> +#   If config types are not given, it is taken from a line containing
> +#   CONFIGTYPES= in the baseconfig.  If that is also not present, def
> +#   and randpackage are tried.
> +#
> +# If no arguments are given, the testspecs are taken from the testspecs
> +# file in the current directory.  If that file doesn't exist, the testspecs
> +# file in the test directory is taken.
> +#
> +# The output goes into the current directory.
> +
> +testdir=$(dirname $0)
> +topdir=$(cd $testdir/..; pwd)
> +
> +configdir=$PWD/gen-config
> +brversion=$(make -C $topdir --no-print-directory print-version)
> +
> +echo "Running tests for $brversion"
> +
> +# Run build for config $1 with type $2
> +function runbuild() {
> +	baseconfig=$1
> +	configtype=$2
> +
> +	echo "Building $baseconfig ${configtype}config"
> +
> +	# Interprete baseconfig relative to topdir
> +	if [ "${baseconfig:0:1}" = / ]; then
> +		baseconfigfile=$baseconfig
> +	else
> +		baseconfigfile=${topdir}/${baseconfig}
> +	fi
> +
> +	if [ ! -r $baseconfigfile ]; then
> +		echo "  *** $baseconfig not found!"
> +		return 1
> +	fi
> +
> +	# Create the configtype from baseconfig
> +	configname=${brversion}-$(basename $baseconfig)
> +	hostdir=${PWD}/${configname}.hostdir
> +	mkdir -p $configdir
> +	cp $baseconfigfile $configdir/.config
> +	echo "BR2_HOST_DIR=\"${hostdir}\"" >> $configdir/.config
> +	make -C $topdir --no-print-directory O=$configdir ${configtype}config > /dev/null || {
> +		echo "  *** make ${configtype}config failed for $baseconfig"
> +		return 1
> +	}
> +
> +	# Add the sha of the config file to its name, so we can uniquely
> +	# identify it.  This allows us to remove duplicates, e.g. defconfig
> +	# and allnoconfig.
> +	fullconfigname=${configname}-$(sha1sum $configdir/.config | cut -c -40)
> +	if [ -e ${fullconfigname}.success ]; then
> +		echo "  ${fullconfigname} already built successfully; skipping"
> +		return 0
> +	fi
> +
> +	# Save the defconfig of this config for future reference.
> +	make -C $topdir --no-print-directory O=$configdir savedefconfig > /dev/null || {
> +		echo "  *** Failed savedefconfig for $fullconfigname"
> +		return 1
> +	}
> +	cp $configdir/defconfig $fullconfigname.defconfig
> +
> +	outdir=${PWD}/${fullconfigname}.output
> +	logfile=${fullconfigname}.log
> +	mkdir -p $outdir
> +	cp $configdir/.config $outdir
> +	make -C $topdir O=$outdir > $logfile 2>&1 || {
> +		echo "  *** Failed $fullconfigname - see log file for details"
> +		touch $fullconfigname.failed
> +		#rm -rf $outdir
> +		return 1
> +	}
> +
> +	echo "  Successfully built $baseconfig ${configtype}config"
> +	touch $fullconfigname.success
> +	rm -rf $outdir
> +	bzip2 $logfile
> +}
> +
> +
> +if [ "$1" ]; then
> +	testspecs="$*"
> +elif [ -r testspecs ]; then
> +	testspecs="$(cat testspecs)"
> +else
> +	testspecs="$(cat $testdir/testspecs)"
> +fi
> +
> +for testspec in $testspecs; do
> +	baseconfig=${testspec%=*}
> +	configtypes=${testspec#*=}
> +	for configtype in ${configtypes//,/ }; do
> +		runbuild $baseconfig $configtype
> +	done
> +done
> diff --git a/test/testspecs b/test/testspecs
> new file mode 100644
> index 0000000..2e3be3d
> --- /dev/null
> +++ b/test/testspecs
> @@ -0,0 +1,24 @@
> +configs/armadeus_apf9328_defconfig=def,randpackage
> +configs/at91rm9200df_defconfig=def,randpackage
> +configs/at91sam9260dfc_defconfig=def,randpackage
> +configs/at91sam9261ek_defconfig=def,randpackage
> +configs/at91sam9263ek_defconfig=def,randpackage
> +configs/at91sam9g20dfc_defconfig=def,randpackage
> +configs/atngw100_defconfig=def,randpackage
> +configs/atstk100x_defconfig=def,randpackage
> +configs/ea3250_defconfig=def,randpackage
> +configs/fdi3250_defconfig=def,randpackage
> +configs/integrator926_defconfig=def,randpackage
> +configs/kb9202_defconfig=def,randpackage
> +configs/mini2440_defconfig=def,randpackage
> +configs/mx53loco_defconfig=def,randpackage
> +configs/phy3250_defconfig=def,randpackage
> +configs/qemu_arm_versatile_defconfig=def,randpackage
> +configs/qemu_mips_malta_defconfig=def,randpackage
> +configs/qemu_mipsel_malta_defconfig=def,randpackage
> +configs/qemu_ppc_g3beige_defconfig=def,randpackage
> +configs/qemu_sh4_r2d_defconfig=def,randpackage
> +configs/qemu_sparc_ss10_defconfig=def,randpackage
> +configs/qemu_x86_64_defconfig=def,randpackage
> +configs/qemu_x86_defconfig=def,randpackage
> +configs/sheevaplug_defconfig=def,randpackage
> -- 
> 1.7.9.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/2] Add test infrastructure

[Arnout Vandecappelle]

On Monday 05 March 2012 15:56:25 Jean-Christophe PLAGNIOL-VILLARD wrote:
>         I like the idea of test but what is the target here?

 Good point, I should have explained that.

 Currently, ThomasP and Peter are both independently running tests on the 
buildroot tree.  We would like to have more testing done.  I myself have a
server available that can do continuous testing.  However, a single 
buildbot isn't going to cut it, because it takes quite long to build a
single configuration.  Therefore, I would like to put test infrastructure
in the buildroot repository so others can easily set up their own buildbot.

 Another part of this evolution will be to set up a webpage where test
results can be posted and retrieved.  Peter was going to do that (but 
hasn't had the time yet).

 Regards,
 Arnout

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH 2/2] Add test infrastructure

[Thomas Petazzoni]

Le Tue, 6 Mar 2012 08:06:03 +0100,
Arnout Vandecappelle <arnout@mind.be> a ?crit :

>  Currently, ThomasP and Peter are both independently running tests on
> the buildroot tree.  We would like to have more testing done.  I
> myself have a server available that can do continuous testing.
> However, a single buildbot isn't going to cut it, because it takes
> quite long to build a single configuration.  Therefore, I would like
> to put test infrastructure in the buildroot repository so others can
> easily set up their own buildbot.

I am repeating myself here, but anyway. Buildbot is nice if you always
rebuild the same configuration over and over again, to verify that
there are no regressions.

However, the tests that we do with Peter are *random* configuration
builds. And this doesn't really fit within the Buildbot model: you
can't compare the result of a build (success/failure) to another,
because each build uses a different configuration.

Buildbot works fine if you test the defconfigs that we have, or some
other well-known configurations, but I don't really see how it would
apply to randpackageconfig builds, and this is exactly what Peter and I
are doing.

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com