All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/3] RFC: Package testing
@ 2012-08-23 14:19 Björn Stenberg
  2012-08-23 14:20 ` [PATCH 1/3] Add -ptest package group Björn Stenberg
                   ` (3 more replies)
  0 siblings, 4 replies; 15+ messages in thread
From: Björn Stenberg @ 2012-08-23 14:19 UTC (permalink / raw)
  To: yocto

Hi.

This is a first set of patches implementing the package testing concept
I proposed in 
https://lists.yoctoproject.org/pipermail/yocto/2012-June/009443.html

In summary, the proposal creates a new package group called -ptest that
contains the test suites included with many source packages, compiled to
run on target.

Packaging the test suites sometimes requires patching makefiles to allow
test tools to be built on host but ran on target. And it sometimes
requires patching and/or translation of the test output to produce a
generic output format that can be automatically parsed. I have included
an example patch for bash that does just that.

I have also included "ptest-runnner", a simple script to find and run
all installed ptests.


Björn Stenberg (3):
  Add -ptest package group
  New recipe: ptest-runner
  Enabled bash-ptest

 meta/classes/distutils-common-base.bbclass         |    5 +++-
 meta/classes/image.bbclass                         |    6 +++-
 meta/classes/package.bbclass                       |    8 ++++-
 meta/classes/task.bbclass                          |    2 +-
 meta/conf/bitbake.conf                             |   14 ++++++++-
 .../bash/bash-4.2/build-tests.patch                |   29 ++++++++++++++++++++
 meta/recipes-extended/bash/bash-4.2/run-ptest      |    2 +
 .../bash/bash-4.2/test-output.patch                |   19 +++++++++++++
 meta/recipes-extended/bash/bash.inc                |   18 ++++++++++++
 meta/recipes-extended/bash/bash_4.2.bb             |    5 +++-
 .../ptest-runner/files/ptest-runner                |   16 +++++++++++
 .../ptest-runner/ptest-runner_1.0.bb               |   21 ++++++++++++++
 12 files changed, 136 insertions(+), 9 deletions(-)
 create mode 100644 meta/recipes-extended/bash/bash-4.2/build-tests.patch
 create mode 100644 meta/recipes-extended/bash/bash-4.2/run-ptest
 create mode 100644 meta/recipes-extended/bash/bash-4.2/test-output.patch
 create mode 100644 meta/recipes-support/ptest-runner/files/ptest-runner
 create mode 100644 meta/recipes-support/ptest-runner/ptest-runner_1.0.bb

-- 
1.7.5.4



^ permalink raw reply	[flat|nested] 15+ messages in thread
* [PATCH 1/3] Add -ptest package group
  2012-08-23 14:19 [PATCH 0/3] RFC: Package testing Björn Stenberg
@ 2012-08-23 14:20 ` Björn Stenberg
  2012-08-23 14:42   ` Paul Eggleton
  2012-08-24 15:37   ` Richard Purdie
  2012-08-23 14:20 ` [PATCH 2/3] New recipe: ptest-runner Björn Stenberg
                   ` (2 subsequent siblings)
  3 siblings, 2 replies; 15+ messages in thread
From: Björn Stenberg @ 2012-08-23 14:20 UTC (permalink / raw)
  To: yocto

This patch creates a new package group -ptest to contain the tests for
each package.

---
 meta/classes/distutils-common-base.bbclass |    5 ++++-
 meta/classes/image.bbclass                 |    6 ++++--
 meta/classes/package.bbclass               |    8 ++++++--
 meta/classes/task.bbclass                  |    2 +-
 meta/conf/bitbake.conf                     |   14 ++++++++++++--
 5 files changed, 27 insertions(+), 8 deletions(-)

diff --git a/meta/classes/distutils-common-base.bbclass b/meta/classes/distutils-common-base.bbclass
index f66a5cd..7db015b 100644
--- a/meta/classes/distutils-common-base.bbclass
+++ b/meta/classes/distutils-common-base.bbclass
@@ -5,7 +5,7 @@ EXTRA_OEMAKE = ""
 export STAGING_INCDIR
 export STAGING_LIBDIR
 
-PACKAGES = "${PN}-dev ${PN}-dbg ${PN}-doc ${PN}"
+PACKAGES = "${PN}-dev ${PN}-dbg ${PN}-ptest ${PN}-doc ${PN}"
 
 FILES_${PN} = "${bindir}/* ${libdir}/* ${libdir}/${PYTHON_DIR}/*"
 
@@ -19,3 +19,6 @@ FILES_${PN}-dbg += "\
   ${PYTHON_SITEPACKAGES_DIR}/*/.debug \
   ${PYTHON_SITEPACKAGES_DIR}/*/*/.debug \
 "
+FILES_${PN}-ptest += "\
+  ${PYTHON_SITEPACKAGES_DIR}/${PTEST_NAME} \
+"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 72720f1..c154bc2 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -29,13 +29,13 @@ ROOTFS_BOOTSTRAP_INSTALL = "${@base_contains("IMAGE_FEATURES", "package-manageme
 FEATURE_INSTALL = "${@' '.join(oe.packagegroup.required_packages(oe.data.typed_value('IMAGE_FEATURES', d), d))}"
 FEATURE_INSTALL_OPTIONAL = "${@' '.join(oe.packagegroup.optional_packages(oe.data.typed_value('IMAGE_FEATURES', d), d))}"
 
-# packages to install from features, excluding dev/dbg/doc
+# packages to install from features, excluding dev/dbg/doc/ptest
 NORMAL_FEATURE_INSTALL = "${@' '.join(oe.packagegroup.required_packages(normal_groups(d), d))}"
 NORMAL_FEATURE_INSTALL_OPTIONAL = "${@' '.join(oe.packagegroup.optional_packages(normal_groups(d), d))}"
 
 def normal_groups(d):
     """Return all the IMAGE_FEATURES, with the exception of our special package groups"""
-    extras = set(['dev-pkgs', 'staticdev-pkgs', 'doc-pkgs', 'dbg-pkgs'])
+    extras = set(['dev-pkgs', 'staticdev-pkgs', 'doc-pkgs', 'dbg-pkgs', 'ptest-pkgs'])
     features = set(oe.data.typed_value('IMAGE_FEATURES', d))
     return features.difference(extras)
 
@@ -53,6 +53,8 @@ def complementary_globs(featurevar, d):
             globs.append('*-doc')
         elif feature == 'dbg-pkgs':
             globs.append('*-dbg')
+        elif feature == 'ptest-pkgs':
+            globs.append('*-ptest')
     return ' '.join(globs)
 
 IMAGE_INSTALL_COMPLEMENTARY = '${@complementary_globs("IMAGE_FEATURES", d)}'
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index d122cd9..6483a1f 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1236,7 +1236,7 @@ python package_do_filedeps() {
 
     # Determine dependencies
     for pkg in packages.split():
-        if pkg.endswith('-dbg') or pkg.endswith('-doc') or pkg.find('-locale-') != -1 or pkg.find('-localedata-') != -1 or pkg.find('-gconv-') != -1 or pkg.find('-charmap-') != -1 or pkg.startswith('kernel-module-'):
+        if pkg.endswith('-dbg') or pkg.endswith('-doc') or pkg.find('-locale-') != -1 or pkg.find('-localedata-') != -1 or pkg.find('-gconv-') != -1 or pkg.find('-charmap-') != -1 or pkg.startswith('kernel-module-') or pkg.endswith('-ptest'):
             continue
 
         provides_files = []
@@ -1330,7 +1330,7 @@ python package_do_shlibs() {
                 combos.append("-".join(options[0:i]))
             return combos
 
-        if (file.endswith('.dylib') or file.endswith('.so')) and not pkg.endswith('-dev') and not pkg.endswith('-dbg'):
+        if (file.endswith('.dylib') or file.endswith('.so')) and not pkg.endswith('-dev') and not pkg.endswith('-dbg') and not pkg.endswith('-ptest'):
             # Drop suffix
             name = file.rsplit(".",1)[0]
             # Find all combinations
@@ -1648,6 +1648,8 @@ python package_depchains() {
                 depend = depend.replace('-dev', '')
             if depend.endswith('-dbg'):
                 depend = depend.replace('-dbg', '')
+            if depend.endswith('-ptest'):
+                depend = depend.replace('-ptest', '')
             pkgname = getname(depend, suffix)
             #bb.note("Adding %s for %s" % (pkgname, depend))
             if pkgname not in rreclist and pkgname != pkg:
@@ -1669,6 +1671,8 @@ python package_depchains() {
                 depend = depend.replace('-dev', '')
             if depend.endswith('-dbg'):
                 depend = depend.replace('-dbg', '')
+            if depend.endswith('-ptest'):
+                depend = depend.replace('-ptest', '')
             pkgname = getname(depend, suffix)
             #bb.note("Adding %s for %s" % (pkgname, depend))
             if pkgname not in rreclist and pkgname != pkg:
diff --git a/meta/classes/task.bbclass b/meta/classes/task.bbclass
index 6ec154a..653f149 100644
--- a/meta/classes/task.bbclass
+++ b/meta/classes/task.bbclass
@@ -20,7 +20,7 @@ python () {
     packages = d.getVar('PACKAGES', True).split()
     genpackages = []
     for pkg in packages:
-        for postfix in ['-dbg', '-dev']:
+        for postfix in ['-dbg', '-ptest', '-dev']:
             genpackages.append(pkg+postfix)
     d.setVar('PACKAGES', ' '.join(packages+genpackages))
 }
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index d5a43e9..07d057f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -220,6 +220,10 @@ SUMMARY_${PN}-dbg ?= "${SUMMARY} - Debugging files"
 DESCRIPTION_${PN}-dbg ?= "${DESCRIPTION}  \
 This package contains ELF symbols and related sources for debugging purposes."
 
+SUMMARY_${PN}-ptest ?= "${SUMMARY} - Package test files"
+DESCRIPTION_${PN}-ptest ?= "${DESCRIPTION}  \
+This package contains test directory with the name ${PTEST_NAME} for package test purposes."
+
 SUMMARY_${PN}-dev ?= "${SUMMARY} - Development files"
 DESCRIPTION_${PN}-dev ?= "${DESCRIPTION}  \
 This package contains symbolic links, header files, and \
@@ -242,7 +246,7 @@ HOMEPAGE = "unknown"
 # Ensure that -dev packages recommend the corresponding -dev packages of their
 # deps, and the same for -dbg.
 DEPCHAIN_PRE  = ""
-DEPCHAIN_POST = "-dev -dbg"
+DEPCHAIN_POST = "-dev -dbg -ptest"
 
 DEPENDS = ""
 RDEPENDS = ""
@@ -263,7 +267,7 @@ SOLIBSDEV_darwin8 = ".dylib"
 SOLIBSDEV_darwin9 = ".dylib"
 
 PACKAGE_BEFORE_PN ?= ""
-PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
+PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PN}-ptest ${PACKAGE_BEFORE_PN} ${PN}"
 PACKAGES_DYNAMIC = "${PN}-locale-*"
 FILES = ""
 
@@ -307,6 +311,12 @@ SECTION_${PN}-dbg = "devel"
 ALLOW_EMPTY_${PN}-dbg = "1"
 RRECOMMENDS_${PN}-dbg = "${PN} (= ${EXTENDPKGV})"
 
+PTEST_PATH ?= "${libdir}/${PN}/ptest"
+FILES_${PN}-ptest = "${PTEST_PATH}/*"
+SECTION_${PN}-ptest = "devel"
+ALLOW_EMPTY_${PN}-ptest = "1"
+RDEPENDS_${PN}-ptest = "${PN} (= ${EXTENDPKGV})"
+
 FILES_${PN}-locale = "${datadir}/locale"
 
 # File manifest
-- 
1.7.5.4



^ permalink raw reply related	[flat|nested] 15+ messages in thread
* [PATCH 2/3] New recipe: ptest-runner
  2012-08-23 14:19 [PATCH 0/3] RFC: Package testing Björn Stenberg
  2012-08-23 14:20 ` [PATCH 1/3] Add -ptest package group Björn Stenberg
@ 2012-08-23 14:20 ` Björn Stenberg
  2012-08-23 14:20 ` [PATCH 3/3] Enable bash-ptest Björn Stenberg
  2012-08-23 21:53 ` [PATCH 0/3] RFC: Package testing Burton, Ross
  3 siblings, 0 replies; 15+ messages in thread
From: Björn Stenberg @ 2012-08-23 14:20 UTC (permalink / raw)
  To: yocto

This package contains the top-level script that seeks out and starts all
package tests.

---
 .../ptest-runner/files/ptest-runner                |   16 +++++++++++++++
 .../ptest-runner/ptest-runner_1.0.bb               |   21 ++++++++++++++++++++
 2 files changed, 37 insertions(+), 0 deletions(-)
 create mode 100644 meta/recipes-support/ptest-runner/files/ptest-runner
 create mode 100644 meta/recipes-support/ptest-runner/ptest-runner_1.0.bb

diff --git a/meta/recipes-support/ptest-runner/files/ptest-runner b/meta/recipes-support/ptest-runner/files/ptest-runner
new file mode 100644
index 0000000..0aa7970
--- /dev/null
+++ b/meta/recipes-support/ptest-runner/files/ptest-runner
@@ -0,0 +1,16 @@
+#!/bin/sh
+
+echo "START: $0"
+cd /usr/lib
+for x in *
+do
+    if [ -d "/usr/lib/$x/ptest" ]; then
+	date "+%Y-%m-%dT%H:%M"
+        echo "BEGIN: $x"
+        cd /usr/lib/$x/ptest
+        ./run-ptest
+        echo "END: $x"
+	date "+%Y-%m-%dT%H:%M"
+    fi
+done
+echo "STOP: $0"
diff --git a/meta/recipes-support/ptest-runner/ptest-runner_1.0.bb b/meta/recipes-support/ptest-runner/ptest-runner_1.0.bb
new file mode 100644
index 0000000..63598b3
--- /dev/null
+++ b/meta/recipes-support/ptest-runner/ptest-runner_1.0.bb
@@ -0,0 +1,21 @@
+SRC_URI += "file://ptest-runner"
+
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=3f40d7994397109285ec7b81fdeb3b58 \
+                    file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+do_install () {
+    mkdir -p ${D}${bindir}
+    install -m 0755 ${WORKDIR}/ptest-runner ${D}${bindir}
+}
+
+do_patch[noexec] = "1"
+do_configure[noexec] = "1"
+do_compile[noexec] = "1"
+do_build[noexec] = "1"
-- 
1.7.5.4



^ permalink raw reply related	[flat|nested] 15+ messages in thread
* [PATCH 3/3] Enable bash-ptest
  2012-08-23 14:19 [PATCH 0/3] RFC: Package testing Björn Stenberg
  2012-08-23 14:20 ` [PATCH 1/3] Add -ptest package group Björn Stenberg
  2012-08-23 14:20 ` [PATCH 2/3] New recipe: ptest-runner Björn Stenberg
@ 2012-08-23 14:20 ` Björn Stenberg
  2012-08-24 15:40   ` Richard Purdie
  2012-08-24 19:10   ` Saul Wold
  2012-08-23 21:53 ` [PATCH 0/3] RFC: Package testing Burton, Ross
  3 siblings, 2 replies; 15+ messages in thread
From: Björn Stenberg @ 2012-08-23 14:20 UTC (permalink / raw)
  To: yocto

Patch Makefile.in to allow test programs be built on host and ran on
target.
Patch tests/run-all to output PASS/FAIL for each testcase.
Patch recipe to build and install test programs and test suite.

---
 .../bash/bash-4.2/build-tests.patch                |   29 ++++++++++++++++++++
 meta/recipes-extended/bash/bash-4.2/run-ptest      |    2 +
 .../bash/bash-4.2/test-output.patch                |   19 +++++++++++++
 meta/recipes-extended/bash/bash.inc                |   18 ++++++++++++
 meta/recipes-extended/bash/bash_4.2.bb             |    5 +++-
 5 files changed, 72 insertions(+), 1 deletions(-)
 create mode 100644 meta/recipes-extended/bash/bash-4.2/build-tests.patch
 create mode 100644 meta/recipes-extended/bash/bash-4.2/run-ptest
 create mode 100644 meta/recipes-extended/bash/bash-4.2/test-output.patch

diff --git a/meta/recipes-extended/bash/bash-4.2/build-tests.patch b/meta/recipes-extended/bash/bash-4.2/build-tests.patch
new file mode 100644
index 0000000..e92fab3
--- /dev/null
+++ b/meta/recipes-extended/bash/bash-4.2/build-tests.patch
@@ -0,0 +1,29 @@
+diff -uNr a/Makefile.in b/Makefile.in
+--- a/Makefile.in	2012-06-14 10:15:15.063465304 +0200
++++ b/Makefile.in	2012-06-14 10:47:02.985115849 +0200
+@@ -827,18 +827,21 @@
+ 	fi
+ 
+ recho$(EXEEXT):		$(SUPPORT_SRC)recho.c
+-	@$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)recho.c ${LIBS_FOR_BUILD}
++	@$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $<
+ 
+ zecho$(EXEEXT):		$(SUPPORT_SRC)zecho.c
+-	@$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)zecho.c ${LIBS_FOR_BUILD}
++	@$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $<
+ 
+ printenv$(EXEEXT):	$(SUPPORT_SRC)printenv.c
+-	@$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)printenv.c ${LIBS_FOR_BUILD}
++	@$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $<
+ 
+ xcase$(EXEEXT):	$(SUPPORT_SRC)xcase.c
+-	@$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)xcase.c ${LIBS_FOR_BUILD}
++	@$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $<
+ 
+ test tests check:	force $(Program) $(TESTS_SUPPORT)
++	$(MAKE) runtest
++
++runtest:
+ 	@-test -d tests || mkdir tests
+ 	@cp $(TESTS_SUPPORT) tests
+ 	@( cd $(srcdir)/tests && \
diff --git a/meta/recipes-extended/bash/bash-4.2/run-ptest b/meta/recipes-extended/bash/bash-4.2/run-ptest
new file mode 100644
index 0000000..8dd3b99
--- /dev/null
+++ b/meta/recipes-extended/bash/bash-4.2/run-ptest
@@ -0,0 +1,2 @@
+#!/bin/sh
+make -k THIS_SH=/bin/bash BUILD_DIR=. runtest
diff --git a/meta/recipes-extended/bash/bash-4.2/test-output.patch b/meta/recipes-extended/bash/bash-4.2/test-output.patch
new file mode 100644
index 0000000..065a85c
--- /dev/null
+++ b/meta/recipes-extended/bash/bash-4.2/test-output.patch
@@ -0,0 +1,19 @@
+diff -uNr a/tests/run-all b/tests/run-all
+--- a/tests/run-all	2012-06-14 10:15:15.007467700 +0200
++++ b/tests/run-all	2012-06-14 10:19:26.464678067 +0200
+@@ -22,6 +22,14 @@
+ 	case $x in
+ 	$0|run-minimal|run-gprof)	;;
+ 	*.orig|*~) ;;
+-	*)	echo $x ; sh $x ;;
++	*)	echo $x
++		output=`sh $x`
++		if [ -n "$output" ]; then
++			echo "$output"
++			echo "FAIL: $x"
++		else
++			echo "PASS: $x"
++		fi
++		;;
+ 	esac
+ done
diff --git a/meta/recipes-extended/bash/bash.inc b/meta/recipes-extended/bash/bash.inc
index 3684191..04f9940 100644
--- a/meta/recipes-extended/bash/bash.inc
+++ b/meta/recipes-extended/bash/bash.inc
@@ -21,18 +21,36 @@ ALTERNATIVE_PRIORITY = "100"
 
 export AUTOHEADER = "true"
 
+RDEPENDS_${PN}-ptest += "make"
+FILES_${PN}-dbg += "${PTEST_PATH}/.debug"
+
 do_configure_prepend () {
 	if [ ! -e acinclude.m4 ]; then
 		cat aclocal.m4 > acinclude.m4
 	fi
 }
 
+do_compile_append () {
+	oe_runmake recho zecho printenv xcase
+}
+
 do_install_append () {
 	# Move /usr/bin/bash to /bin/bash, if need
 	if [ "${base_bindir}" != "${bindir}" ]; then
 		mkdir -p ${D}${base_bindir}
 		mv ${D}${bindir}/bash ${D}${base_bindir}
 	fi
+
+        if [ "${PN}" = "${BPN}" ]; then
+            mkdir -p ${D}${PTEST_PATH}
+            install -m 0755 ${WORKDIR}/run-ptest    ${D}${PTEST_PATH}
+            install -m 0755 ${B}/Makefile           ${D}${PTEST_PATH}
+            install -m 0755 ${B}/recho              ${D}${PTEST_PATH}
+            install -m 0755 ${B}/zecho              ${D}${PTEST_PATH}
+            install -m 0755 ${B}/printenv           ${D}${PTEST_PATH}
+            install -m 0755 ${B}/xcase              ${D}${PTEST_PATH}
+            cp -r ${B}/tests                        ${D}${PTEST_PATH}/
+        fi
 }
 
 pkg_postinst_${PN} () {
diff --git a/meta/recipes-extended/bash/bash_4.2.bb b/meta/recipes-extended/bash/bash_4.2.bb
index 50f2ff6..5e9b50e 100644
--- a/meta/recipes-extended/bash/bash_4.2.bb
+++ b/meta/recipes-extended/bash/bash_4.2.bb
@@ -1,6 +1,6 @@
 require bash.inc
 
-PR = "r4"
+PR = "r5"
 
 SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \
            ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-001;apply=yes;striplevel=0;name=patch001 \
@@ -14,6 +14,9 @@ SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \
            ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-009;apply=yes;striplevel=0;name=patch009 \
            ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-010;apply=yes;striplevel=0;name=patch010 \
            file://execute_cmd.patch;striplevel=0 \
+           file://build-tests.patch \
+           file://test-output.patch \
+           file://run-ptest \
            "
 
 SRC_URI[tarball.md5sum] = "3fb927c7c33022f1c327f14a81c0d4b0"
-- 
1.7.5.4



^ permalink raw reply related	[flat|nested] 15+ messages in thread
* Re: [PATCH 1/3] Add -ptest package group
  2012-08-23 14:20 ` [PATCH 1/3] Add -ptest package group Björn Stenberg
@ 2012-08-23 14:42   ` Paul Eggleton
  2012-08-24  6:51     ` Björn Stenberg
  2012-08-24 15:37   ` Richard Purdie
  1 sibling, 1 reply; 15+ messages in thread
From: Paul Eggleton @ 2012-08-23 14:42 UTC (permalink / raw)
  To: Björn Stenberg; +Cc: yocto

On Thursday 23 August 2012 16:20:00 Björn Stenberg wrote:
> This patch creates a new package group -ptest to contain the tests for
> each package.
>...
> @@ -242,7 +246,7 @@ HOMEPAGE = "unknown"
>  # Ensure that -dev packages recommend the corresponding -dev packages of
> their # deps, and the same for -dbg.
>  DEPCHAIN_PRE  = ""
> -DEPCHAIN_POST = "-dev -dbg"
> +DEPCHAIN_POST = "-dev -dbg -ptest"

Is this really what you want? Surely the tests for a particular piece of 
software are mostly independent of those of its dependencies?

Rest of it looks good.

Cheers,
Paul
 
-- 

Paul Eggleton
Intel Open Source Technology Centre


^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: [PATCH 0/3] RFC: Package testing
  2012-08-23 14:19 [PATCH 0/3] RFC: Package testing Björn Stenberg
                   ` (2 preceding siblings ...)
  2012-08-23 14:20 ` [PATCH 3/3] Enable bash-ptest Björn Stenberg
@ 2012-08-23 21:53 ` Burton, Ross
  2012-08-24  6:49   ` Björn Stenberg
  3 siblings, 1 reply; 15+ messages in thread
From: Burton, Ross @ 2012-08-23 21:53 UTC (permalink / raw)
  To: Björn Stenberg; +Cc: yocto

On 23 August 2012 15:19, Björn Stenberg <bjst@enea.com> wrote:
> And it sometimes
> requires patching and/or translation of the test output to produce a
> generic output format that can be automatically parsed. I have included
> an example patch for bash that does just that.

What is this generic output format?  From your patch to bash I suspect
it's at least inspired by the automake "make check" output.

FWIW, this is what "make check" on librest (the first package I had to
hand) gives:

...
make  check-TESTS
make[2]: Entering directory `/home/ross/Programming/cvs/librest/tests'
PASS: proxy
PASS: proxy-continuous
PASS: threaded
PASS: oauth
PASS: oauth-async
PASS: oauth2
PASS: flickr
PASS: lastfm

** (process:1267): ERROR **: Generated output for parsed XML does not match:
in:  <node0 a00='v00' a01='v01'><node1 a10='v10'></node1><node1
a10='v10'></node1>Cont0</node0>
out: <node0 a01='v01' a00='v00'><node1 a10='v10'></node1><node1
a10='v10'></node1>Cont0</node0>

/bin/sh: line 5:  1267 Trace/breakpoint trap   (core dumped) ${dir}$tst
XFAIL: xml
PASS: custom-serialize
=====================================================
All 10 tests behaved as expected (1 expected failure)
=====================================================
make[2]: Leaving directory `/home/ross/Programming/cvs/librest/tests'
...

Would this test suite just work if packaged as ptests?

Ross


^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: [PATCH 0/3] RFC: Package testing
  2012-08-23 21:53 ` [PATCH 0/3] RFC: Package testing Burton, Ross
@ 2012-08-24  6:49   ` Björn Stenberg
  0 siblings, 0 replies; 15+ messages in thread
From: Björn Stenberg @ 2012-08-24  6:49 UTC (permalink / raw)
  To: Burton, Ross; +Cc: yocto

Burton, Ross wrote:
> What is this generic output format?  From your patch to bash I suspect
> it's at least inspired by the automake "make check" output.

Yes, I used the automake-style "(PASS|FAIL): <testcase>" format.

In addition to that, before/after each package test suite is started, "(BEGIN|END): <package>" is printed. So a typical output looks like this (paraphrased):

BEGIN: bash
PASS: test1
FAIL: test2
PASS: test3
END: bash
BEGIN: busybox
PASS: test1
FAIL: test2
END: busybox

-- 
Björn


^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: [PATCH 1/3] Add -ptest package group
  2012-08-23 14:42   ` Paul Eggleton
@ 2012-08-24  6:51     ` Björn Stenberg
  0 siblings, 0 replies; 15+ messages in thread
From: Björn Stenberg @ 2012-08-24  6:51 UTC (permalink / raw)
  To: Paul Eggleton; +Cc: yocto

Paul Eggleton wrote:
> > +DEPCHAIN_POST = "-dev -dbg -ptest"
> 
> Is this really what you want? Surely the tests for a particular piece of 
> software are mostly independent of those of its dependencies?

You're right, that change is not necessary. I misunderstood the purpose of DEPCHAIN_POST.

-- 
Björn


^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: [PATCH 1/3] Add -ptest package group
  2012-08-23 14:20 ` [PATCH 1/3] Add -ptest package group Björn Stenberg
  2012-08-23 14:42   ` Paul Eggleton
@ 2012-08-24 15:37   ` Richard Purdie
  2012-08-28 11:17     ` Björn Stenberg
  1 sibling, 1 reply; 15+ messages in thread
From: Richard Purdie @ 2012-08-24 15:37 UTC (permalink / raw)
  To: Björn Stenberg; +Cc: yocto

On Thu, 2012-08-23 at 16:20 +0200, Björn Stenberg wrote:
> This patch creates a new package group -ptest to contain the tests for
> each package.
> 
> ---
>  meta/classes/distutils-common-base.bbclass |    5 ++++-
>  meta/classes/image.bbclass                 |    6 ++++--
>  meta/classes/package.bbclass               |    8 ++++++--
>  meta/classes/task.bbclass                  |    2 +-
>  meta/conf/bitbake.conf                     |   14 ++++++++++++--
>  5 files changed, 27 insertions(+), 8 deletions(-)
> 
> diff --git a/meta/classes/distutils-common-base.bbclass b/meta/classes/distutils-common-base.bbclass
> index f66a5cd..7db015b 100644
> --- a/meta/classes/distutils-common-base.bbclass
> +++ b/meta/classes/distutils-common-base.bbclass
> @@ -5,7 +5,7 @@ EXTRA_OEMAKE = ""
>  export STAGING_INCDIR
>  export STAGING_LIBDIR
>  
> -PACKAGES = "${PN}-dev ${PN}-dbg ${PN}-doc ${PN}"
> +PACKAGES = "${PN}-dev ${PN}-dbg ${PN}-ptest ${PN}-doc ${PN}"
>  
>  FILES_${PN} = "${bindir}/* ${libdir}/* ${libdir}/${PYTHON_DIR}/*"
>  
> @@ -19,3 +19,6 @@ FILES_${PN}-dbg += "\
>    ${PYTHON_SITEPACKAGES_DIR}/*/.debug \
>    ${PYTHON_SITEPACKAGES_DIR}/*/*/.debug \
>  "
> +FILES_${PN}-ptest += "\
> +  ${PYTHON_SITEPACKAGES_DIR}/${PTEST_NAME} \
> +"
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 72720f1..c154bc2 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -29,13 +29,13 @@ ROOTFS_BOOTSTRAP_INSTALL = "${@base_contains("IMAGE_FEATURES", "package-manageme
>  FEATURE_INSTALL = "${@' '.join(oe.packagegroup.required_packages(oe.data.typed_value('IMAGE_FEATURES', d), d))}"
>  FEATURE_INSTALL_OPTIONAL = "${@' '.join(oe.packagegroup.optional_packages(oe.data.typed_value('IMAGE_FEATURES', d), d))}"
>  
> -# packages to install from features, excluding dev/dbg/doc
> +# packages to install from features, excluding dev/dbg/doc/ptest
>  NORMAL_FEATURE_INSTALL = "${@' '.join(oe.packagegroup.required_packages(normal_groups(d), d))}"
>  NORMAL_FEATURE_INSTALL_OPTIONAL = "${@' '.join(oe.packagegroup.optional_packages(normal_groups(d), d))}"
>  
>  def normal_groups(d):
>      """Return all the IMAGE_FEATURES, with the exception of our special package groups"""
> -    extras = set(['dev-pkgs', 'staticdev-pkgs', 'doc-pkgs', 'dbg-pkgs'])
> +    extras = set(['dev-pkgs', 'staticdev-pkgs', 'doc-pkgs', 'dbg-pkgs', 'ptest-pkgs'])
>      features = set(oe.data.typed_value('IMAGE_FEATURES', d))
>      return features.difference(extras)
>  
> @@ -53,6 +53,8 @@ def complementary_globs(featurevar, d):
>              globs.append('*-doc')
>          elif feature == 'dbg-pkgs':
>              globs.append('*-dbg')
> +        elif feature == 'ptest-pkgs':
> +            globs.append('*-ptest')
>      return ' '.join(globs)
>  
>  IMAGE_INSTALL_COMPLEMENTARY = '${@complementary_globs("IMAGE_FEATURES", d)}'
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index d122cd9..6483a1f 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1236,7 +1236,7 @@ python package_do_filedeps() {
>  
>      # Determine dependencies
>      for pkg in packages.split():
> -        if pkg.endswith('-dbg') or pkg.endswith('-doc') or pkg.find('-locale-') != -1 or pkg.find('-localedata-') != -1 or pkg.find('-gconv-') != -1 or pkg.find('-charmap-') != -1 or pkg.startswith('kernel-module-'):
> +        if pkg.endswith('-dbg') or pkg.endswith('-doc') or pkg.find('-locale-') != -1 or pkg.find('-localedata-') != -1 or pkg.find('-gconv-') != -1 or pkg.find('-charmap-') != -1 or pkg.startswith('kernel-module-') or pkg.endswith('-ptest'):
>              continue

I think filedeps could be useful in this case, autodetecting any
dependencies scripts in the -ptest package could have for example so I'd
not do the above.

>  
>          provides_files = []
> @@ -1330,7 +1330,7 @@ python package_do_shlibs() {
>                  combos.append("-".join(options[0:i]))
>              return combos
>  
> -        if (file.endswith('.dylib') or file.endswith('.so')) and not pkg.endswith('-dev') and not pkg.endswith('-dbg'):
> +        if (file.endswith('.dylib') or file.endswith('.so')) and not pkg.endswith('-dev') and not pkg.endswith('-dbg') and not pkg.endswith('-ptest'):


ditto, we want to detect shlib dependencies.

>              # Drop suffix
>              name = file.rsplit(".",1)[0]
>              # Find all combinations
> @@ -1648,6 +1648,8 @@ python package_depchains() {
>                  depend = depend.replace('-dev', '')
>              if depend.endswith('-dbg'):
>                  depend = depend.replace('-dbg', '')
> +            if depend.endswith('-ptest'):
> +                depend = depend.replace('-ptest', '')
>              pkgname = getname(depend, suffix)
>              #bb.note("Adding %s for %s" % (pkgname, depend))
>              if pkgname not in rreclist and pkgname != pkg:
> @@ -1669,6 +1671,8 @@ python package_depchains() {
>                  depend = depend.replace('-dev', '')
>              if depend.endswith('-dbg'):
>                  depend = depend.replace('-dbg', '')
> +            if depend.endswith('-ptest'):
> +                depend = depend.replace('-ptest', '')
>              pkgname = getname(depend, suffix)
>              #bb.note("Adding %s for %s" % (pkgname, depend))
>              if pkgname not in rreclist and pkgname != pkg:

I also suspect the above two hunks are not needed since ptest won't use
depchains.


> diff --git a/meta/classes/task.bbclass b/meta/classes/task.bbclass
> index 6ec154a..653f149 100644
> --- a/meta/classes/task.bbclass
> +++ b/meta/classes/task.bbclass
> @@ -20,7 +20,7 @@ python () {
>      packages = d.getVar('PACKAGES', True).split()
>      genpackages = []
>      for pkg in packages:
> -        for postfix in ['-dbg', '-dev']:
> +        for postfix in ['-dbg', '-ptest', '-dev']:
>              genpackages.append(pkg+postfix)
>      d.setVar('PACKAGES', ' '.join(packages+genpackages))
>  }
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index d5a43e9..07d057f 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -220,6 +220,10 @@ SUMMARY_${PN}-dbg ?= "${SUMMARY} - Debugging files"
>  DESCRIPTION_${PN}-dbg ?= "${DESCRIPTION}  \
>  This package contains ELF symbols and related sources for debugging purposes."
>  
> +SUMMARY_${PN}-ptest ?= "${SUMMARY} - Package test files"
> +DESCRIPTION_${PN}-ptest ?= "${DESCRIPTION}  \
> +This package contains test directory with the name ${PTEST_NAME} for package test purposes."
> +
>  SUMMARY_${PN}-dev ?= "${SUMMARY} - Development files"
>  DESCRIPTION_${PN}-dev ?= "${DESCRIPTION}  \
>  This package contains symbolic links, header files, and \
> @@ -242,7 +246,7 @@ HOMEPAGE = "unknown"
>  # Ensure that -dev packages recommend the corresponding -dev packages of their
>  # deps, and the same for -dbg.
>  DEPCHAIN_PRE  = ""
> -DEPCHAIN_POST = "-dev -dbg"
> +DEPCHAIN_POST = "-dev -dbg -ptest"
>  
>  DEPENDS = ""
>  RDEPENDS = ""
> @@ -263,7 +267,7 @@ SOLIBSDEV_darwin8 = ".dylib"
>  SOLIBSDEV_darwin9 = ".dylib"
>  
>  PACKAGE_BEFORE_PN ?= ""
> -PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
> +PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PN}-ptest ${PACKAGE_BEFORE_PN} ${PN}"
>  PACKAGES_DYNAMIC = "${PN}-locale-*"
>  FILES = ""
>  
> @@ -307,6 +311,12 @@ SECTION_${PN}-dbg = "devel"
>  ALLOW_EMPTY_${PN}-dbg = "1"
>  RRECOMMENDS_${PN}-dbg = "${PN} (= ${EXTENDPKGV})"
>  
> +PTEST_PATH ?= "${libdir}/${PN}/ptest"
> +FILES_${PN}-ptest = "${PTEST_PATH}/*"
> +SECTION_${PN}-ptest = "devel"
> +ALLOW_EMPTY_${PN}-ptest = "1"
> +RDEPENDS_${PN}-ptest = "${PN} (= ${EXTENDPKGV})"
> +
>  FILES_${PN}-locale = "${datadir}/locale"
>  
>  # File manifest

Would a ptest.bbclass containing the above make more sense? How
widespread do we think these tests will be?

Also these patches are against OE-Core so need to go to the
openembedded-code mailing list.

I do like the idea of better testing of packages and this is a good
start though. My biggest worry is the overhead of the patches to
software to enable the tests. Are you planning to engage the upstreams
and see if we can get them to accept the patches? I have to be very
cautious of anything which would "strangle" us with patch
overhead/overload and stop us being unable to update recipes.

Cheers,

Richard




^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: [PATCH 3/3] Enable bash-ptest
  2012-08-23 14:20 ` [PATCH 3/3] Enable bash-ptest Björn Stenberg
@ 2012-08-24 15:40   ` Richard Purdie
  2012-08-24 19:10   ` Saul Wold
  1 sibling, 0 replies; 15+ messages in thread
From: Richard Purdie @ 2012-08-24 15:40 UTC (permalink / raw)
  To: Björn Stenberg; +Cc: yocto

On Thu, 2012-08-23 at 16:20 +0200, Björn Stenberg wrote:
> Patch Makefile.in to allow test programs be built on host and ran on
> target.
> Patch tests/run-all to output PASS/FAIL for each testcase.
> Patch recipe to build and install test programs and test suite.
> 
> ---
>  .../bash/bash-4.2/build-tests.patch                |   29 ++++++++++++++++++++
>  meta/recipes-extended/bash/bash-4.2/run-ptest      |    2 +
>  .../bash/bash-4.2/test-output.patch                |   19 +++++++++++++
>  meta/recipes-extended/bash/bash.inc                |   18 ++++++++++++
>  meta/recipes-extended/bash/bash_4.2.bb             |    5 +++-
>  5 files changed, 72 insertions(+), 1 deletions(-)
>  create mode 100644 meta/recipes-extended/bash/bash-4.2/build-tests.patch
>  create mode 100644 meta/recipes-extended/bash/bash-4.2/run-ptest
>  create mode 100644 meta/recipes-extended/bash/bash-4.2/test-output.patch
> 
> diff --git a/meta/recipes-extended/bash/bash-4.2/build-tests.patch b/meta/recipes-extended/bash/bash-4.2/build-tests.patch
> new file mode 100644
> index 0000000..e92fab3
> --- /dev/null
> +++ b/meta/recipes-extended/bash/bash-4.2/build-tests.patch
> @@ -0,0 +1,29 @@
> +diff -uNr a/Makefile.in b/Makefile.in
> +--- a/Makefile.in	2012-06-14 10:15:15.063465304 +0200
> ++++ b/Makefile.in	2012-06-14 10:47:02.985115849 +0200
> +@@ -827,18 +827,21 @@
> + 	fi
> + 
> + recho$(EXEEXT):		$(SUPPORT_SRC)recho.c
> +-	@$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)recho.c ${LIBS_FOR_BUILD}
> ++	@$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $<
> + 
> + zecho$(EXEEXT):		$(SUPPORT_SRC)zecho.c
> +-	@$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)zecho.c ${LIBS_FOR_BUILD}
> ++	@$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $<
> + 
> + printenv$(EXEEXT):	$(SUPPORT_SRC)printenv.c
> +-	@$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)printenv.c ${LIBS_FOR_BUILD}
> ++	@$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $<
> + 
> + xcase$(EXEEXT):	$(SUPPORT_SRC)xcase.c
> +-	@$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)xcase.c ${LIBS_FOR_BUILD}
> ++	@$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $<
> + 
> + test tests check:	force $(Program) $(TESTS_SUPPORT)
> ++	$(MAKE) runtest
> ++
> ++runtest:
> + 	@-test -d tests || mkdir tests
> + 	@cp $(TESTS_SUPPORT) tests
> + 	@( cd $(srcdir)/tests && \


As it stands, I can bet on upstream saying "no" to this for good reason.
A better concept might be to introduce $(CC_FOR_TEST) which could
default to CC_FOR_BUILD. This would let us do what you're thinking here
yet keep upstream happy too.

Just thinking out loud really...

Cheers,

Richard



^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: [PATCH 3/3] Enable bash-ptest
  2012-08-23 14:20 ` [PATCH 3/3] Enable bash-ptest Björn Stenberg
  2012-08-24 15:40   ` Richard Purdie
@ 2012-08-24 19:10   ` Saul Wold
  1 sibling, 0 replies; 15+ messages in thread
From: Saul Wold @ 2012-08-24 19:10 UTC (permalink / raw)
  To: Björn Stenberg; +Cc: yocto

On 08/23/2012 07:20 AM, Björn Stenberg wrote:
> Patch Makefile.in to allow test programs be built on host and ran on
> target.
> Patch tests/run-all to output PASS/FAIL for each testcase.
> Patch recipe to build and install test programs and test suite.
>
> ---
>   .../bash/bash-4.2/build-tests.patch                |   29 ++++++++++++++++++++
>   meta/recipes-extended/bash/bash-4.2/run-ptest      |    2 +
>   .../bash/bash-4.2/test-output.patch                |   19 +++++++++++++
>   meta/recipes-extended/bash/bash.inc                |   18 ++++++++++++
>   meta/recipes-extended/bash/bash_4.2.bb             |    5 +++-
>   5 files changed, 72 insertions(+), 1 deletions(-)
>   create mode 100644 meta/recipes-extended/bash/bash-4.2/build-tests.patch
>   create mode 100644 meta/recipes-extended/bash/bash-4.2/run-ptest
>   create mode 100644 meta/recipes-extended/bash/bash-4.2/test-output.patch
>
> diff --git a/meta/recipes-extended/bash/bash-4.2/build-tests.patch b/meta/recipes-extended/bash/bash-4.2/build-tests.patch
> new file mode 100644
> index 0000000..e92fab3
> --- /dev/null
> +++ b/meta/recipes-extended/bash/bash-4.2/build-tests.patch

Thanks for proposing and getting the initial implementation started, 
this has been something we have been thinking about for a while, great 
to have the things like this become reality!

When introducing new patches, please follow the guidelines for patches 
and commit message:

http://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines

I know this is still and RFC, but your commits themselves also need 
Signed-off-by:


> @@ -0,0 +1,29 @@
> +diff -uNr a/Makefile.in b/Makefile.in
> +--- a/Makefile.in	2012-06-14 10:15:15.063465304 +0200
> ++++ b/Makefile.in	2012-06-14 10:47:02.985115849 +0200
> +@@ -827,18 +827,21 @@
> + 	fi
> +
> + recho$(EXEEXT):		$(SUPPORT_SRC)recho.c
> +-	@$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)recho.c ${LIBS_FOR_BUILD}
> ++	@$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $<
> +
> + zecho$(EXEEXT):		$(SUPPORT_SRC)zecho.c
> +-	@$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)zecho.c ${LIBS_FOR_BUILD}
> ++	@$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $<
> +
> + printenv$(EXEEXT):	$(SUPPORT_SRC)printenv.c
> +-	@$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)printenv.c ${LIBS_FOR_BUILD}
> ++	@$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $<
> +
> + xcase$(EXEEXT):	$(SUPPORT_SRC)xcase.c
> +-	@$(CC_FOR_BUILD) $(CCFLAGS_FOR_BUILD) -o $@ $(SUPPORT_SRC)xcase.c ${LIBS_FOR_BUILD}
> ++	@$(CC) $(CCFLAGS) $(LDFLAGS) -o $@ $<
> +
> + test tests check:	force $(Program) $(TESTS_SUPPORT)
> ++	$(MAKE) runtest
> ++
> ++runtest:
> + 	@-test -d tests || mkdir tests
> + 	@cp $(TESTS_SUPPORT) tests
> + 	@( cd $(srcdir)/tests && \
> diff --git a/meta/recipes-extended/bash/bash-4.2/run-ptest b/meta/recipes-extended/bash/bash-4.2/run-ptest
> new file mode 100644
> index 0000000..8dd3b99
> --- /dev/null
> +++ b/meta/recipes-extended/bash/bash-4.2/run-ptest
> @@ -0,0 +1,2 @@
> +#!/bin/sh
> +make -k THIS_SH=/bin/bash BUILD_DIR=. runtest
> diff --git a/meta/recipes-extended/bash/bash-4.2/test-output.patch b/meta/recipes-extended/bash/bash-4.2/test-output.patch
> new file mode 100644
> index 0000000..065a85c
> --- /dev/null
> +++ b/meta/recipes-extended/bash/bash-4.2/test-output.patch
> @@ -0,0 +1,19 @@
> +diff -uNr a/tests/run-all b/tests/run-all
> +--- a/tests/run-all	2012-06-14 10:15:15.007467700 +0200
> ++++ b/tests/run-all	2012-06-14 10:19:26.464678067 +0200
> +@@ -22,6 +22,14 @@
> + 	case $x in
> + 	$0|run-minimal|run-gprof)	;;
> + 	*.orig|*~) ;;
> +-	*)	echo $x ; sh $x ;;
> ++	*)	echo $x
> ++		output=`sh $x`
> ++		if [ -n "$output" ]; then
> ++			echo "$output"
> ++			echo "FAIL: $x"
> ++		else
> ++			echo "PASS: $x"
> ++		fi
> ++		;;
> + 	esac
> + done
> diff --git a/meta/recipes-extended/bash/bash.inc b/meta/recipes-extended/bash/bash.inc
> index 3684191..04f9940 100644
> --- a/meta/recipes-extended/bash/bash.inc
> +++ b/meta/recipes-extended/bash/bash.inc
> @@ -21,18 +21,36 @@ ALTERNATIVE_PRIORITY = "100"
>
>   export AUTOHEADER = "true"
>
> +RDEPENDS_${PN}-ptest += "make"
> +FILES_${PN}-dbg += "${PTEST_PATH}/.debug"
> +
>   do_configure_prepend () {
>   	if [ ! -e acinclude.m4 ]; then
>   		cat aclocal.m4 > acinclude.m4
>   	fi
>   }
>
> +do_compile_append () {
> +	oe_runmake recho zecho printenv xcase
> +}
> +

One thing that we have discussed a couple of times, but no real outcome 
yet is to make the building of tests conditional so as more recipes are 
enabled with this kind of change, we don't extend the default build time 
too long.

What's been discussed is using DISTRO_FEATURE to test first.

Thoughts?


>   do_install_append () {
>   	# Move /usr/bin/bash to /bin/bash, if need
>   	if [ "${base_bindir}" != "${bindir}" ]; then
>   		mkdir -p ${D}${base_bindir}
>   		mv ${D}${bindir}/bash ${D}${base_bindir}
>   	fi
> +
> +        if [ "${PN}" = "${BPN}" ]; then
> +            mkdir -p ${D}${PTEST_PATH}
> +            install -m 0755 ${WORKDIR}/run-ptest    ${D}${PTEST_PATH}
> +            install -m 0755 ${B}/Makefile           ${D}${PTEST_PATH}
> +            install -m 0755 ${B}/recho              ${D}${PTEST_PATH}
> +            install -m 0755 ${B}/zecho              ${D}${PTEST_PATH}
> +            install -m 0755 ${B}/printenv           ${D}${PTEST_PATH}
> +            install -m 0755 ${B}/xcase              ${D}${PTEST_PATH}
> +            cp -r ${B}/tests                        ${D}${PTEST_PATH}/
> +        fi
>   }
>
>   pkg_postinst_${PN} () {
> diff --git a/meta/recipes-extended/bash/bash_4.2.bb b/meta/recipes-extended/bash/bash_4.2.bb
> index 50f2ff6..5e9b50e 100644
> --- a/meta/recipes-extended/bash/bash_4.2.bb
> +++ b/meta/recipes-extended/bash/bash_4.2.bb
> @@ -1,6 +1,6 @@
>   require bash.inc
>
> -PR = "r4"
> +PR = "r5"
>
>   SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \
>              ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-001;apply=yes;striplevel=0;name=patch001 \
> @@ -14,6 +14,9 @@ SRC_URI = "${GNU_MIRROR}/bash/${BPN}-${PV}.tar.gz;name=tarball \
>              ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-009;apply=yes;striplevel=0;name=patch009 \
>              ${GNU_MIRROR}/bash/bash-4.2-patches/bash42-010;apply=yes;striplevel=0;name=patch010 \
>              file://execute_cmd.patch;striplevel=0 \
> +           file://build-tests.patch \
> +           file://test-output.patch \
> +           file://run-ptest \
>              "
>
>   SRC_URI[tarball.md5sum] = "3fb927c7c33022f1c327f14a81c0d4b0"
>

Thanks
	Sau!



^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: [PATCH 1/3] Add -ptest package group
  2012-08-24 15:37   ` Richard Purdie
@ 2012-08-28 11:17     ` Björn Stenberg
  0 siblings, 0 replies; 15+ messages in thread
From: Björn Stenberg @ 2012-08-28 11:17 UTC (permalink / raw)
  To: Richard Purdie; +Cc: yocto

Richard Purdie wrote:
> Would a ptest.bbclass containing the above make more sense? How
> widespread do we think these tests will be?

Most source packages include some form of test suite that we want to make into a -ptest package. I would guesstimate the number to be somewhere between the number of -dbg and the number of -dev packages. This is why I felt it made sense to add the handling of -ptest side-by-side with the handling of -dev and -dbg.

> My biggest worry is the overhead of the patches to
> software to enable the tests. Are you planning to engage the upstreams
> and see if we can get them to accept the patches?

Upstreaming is absolutely the idea, both for the benefit of the entire ecosystem and of course to reduce the number of patches.

However I won't downplay the effort required to get hundreds of upstream projects to agree on a common concept for test cross compilation and result formatting.

I see two main options: Implement first or discuss first.

a) Implement first: We design and implement a good system for yocto/oe and then, once it has proven itself, engage the larger open source community with our already tested concept.

Pro: It's easier to discuss an already working system
Con: We'll have to manage patches for all packages until agreement

b) Discuss first: We invite the whole open source community to discuss and try to agree on how to handle cross-compiled test cases and test result formatting before we implement anything.

Pro: No increased patch load until agreement
Con: No result until agreement, which could potentially take a long time

My suggestion is a). I'm open for any and all suggestions that would make the patch load easier to manage.

> Also these patches are against OE-Core so need to go to the
> openembedded-code mailing list.

I'll update the patches with the feedback I've got and post them to oe-core for further discussion.

Thanks.

-- 
Björn


^ permalink raw reply	[flat|nested] 15+ messages in thread
* [PATCH 1/3] Add -ptest package group
  2012-08-31 14:10 Björn Stenberg
@ 2012-08-31 14:10 ` Björn Stenberg
  2012-09-01 18:21   ` Koen Kooi
  0 siblings, 1 reply; 15+ messages in thread
From: Björn Stenberg @ 2012-08-31 14:10 UTC (permalink / raw)
  To: openembedded-core

This patch creates a new package group -ptest to contain the tests from
each package.

Signed-off-by: Björn Stenberg <bjst@enea.com>
---
 meta/classes/distutils-common-base.bbclass |    5 ++++-
 meta/classes/image.bbclass                 |    6 ++++--
 meta/classes/task.bbclass                  |    2 +-
 meta/conf/bitbake.conf                     |   12 +++++++++++-
 4 files changed, 20 insertions(+), 5 deletions(-)

diff --git a/meta/classes/distutils-common-base.bbclass b/meta/classes/distutils-common-base.bbclass
index f66a5cd..7db015b 100644
--- a/meta/classes/distutils-common-base.bbclass
+++ b/meta/classes/distutils-common-base.bbclass
@@ -5,7 +5,7 @@ EXTRA_OEMAKE = ""
 export STAGING_INCDIR
 export STAGING_LIBDIR
 
-PACKAGES = "${PN}-dev ${PN}-dbg ${PN}-doc ${PN}"
+PACKAGES = "${PN}-dev ${PN}-dbg ${PN}-ptest ${PN}-doc ${PN}"
 
 FILES_${PN} = "${bindir}/* ${libdir}/* ${libdir}/${PYTHON_DIR}/*"
 
@@ -19,3 +19,6 @@ FILES_${PN}-dbg += "\
   ${PYTHON_SITEPACKAGES_DIR}/*/.debug \
   ${PYTHON_SITEPACKAGES_DIR}/*/*/.debug \
 "
+FILES_${PN}-ptest += "\
+  ${PYTHON_SITEPACKAGES_DIR}/${PTEST_NAME} \
+"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 72720f1..c154bc2 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -29,13 +29,13 @@ ROOTFS_BOOTSTRAP_INSTALL = "${@base_contains("IMAGE_FEATURES", "package-manageme
 FEATURE_INSTALL = "${@' '.join(oe.packagegroup.required_packages(oe.data.typed_value('IMAGE_FEATURES', d), d))}"
 FEATURE_INSTALL_OPTIONAL = "${@' '.join(oe.packagegroup.optional_packages(oe.data.typed_value('IMAGE_FEATURES', d), d))}"
 
-# packages to install from features, excluding dev/dbg/doc
+# packages to install from features, excluding dev/dbg/doc/ptest
 NORMAL_FEATURE_INSTALL = "${@' '.join(oe.packagegroup.required_packages(normal_groups(d), d))}"
 NORMAL_FEATURE_INSTALL_OPTIONAL = "${@' '.join(oe.packagegroup.optional_packages(normal_groups(d), d))}"
 
 def normal_groups(d):
     """Return all the IMAGE_FEATURES, with the exception of our special package groups"""
-    extras = set(['dev-pkgs', 'staticdev-pkgs', 'doc-pkgs', 'dbg-pkgs'])
+    extras = set(['dev-pkgs', 'staticdev-pkgs', 'doc-pkgs', 'dbg-pkgs', 'ptest-pkgs'])
     features = set(oe.data.typed_value('IMAGE_FEATURES', d))
     return features.difference(extras)
 
@@ -53,6 +53,8 @@ def complementary_globs(featurevar, d):
             globs.append('*-doc')
         elif feature == 'dbg-pkgs':
             globs.append('*-dbg')
+        elif feature == 'ptest-pkgs':
+            globs.append('*-ptest')
     return ' '.join(globs)
 
 IMAGE_INSTALL_COMPLEMENTARY = '${@complementary_globs("IMAGE_FEATURES", d)}'
diff --git a/meta/classes/task.bbclass b/meta/classes/task.bbclass
index 6ec154a..653f149 100644
--- a/meta/classes/task.bbclass
+++ b/meta/classes/task.bbclass
@@ -20,7 +20,7 @@ python () {
     packages = d.getVar('PACKAGES', True).split()
     genpackages = []
     for pkg in packages:
-        for postfix in ['-dbg', '-dev']:
+        for postfix in ['-dbg', '-ptest', '-dev']:
             genpackages.append(pkg+postfix)
     d.setVar('PACKAGES', ' '.join(packages+genpackages))
 }
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 7dfeda3..d10e933 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -220,6 +220,10 @@ SUMMARY_${PN}-dbg ?= "${SUMMARY} - Debugging files"
 DESCRIPTION_${PN}-dbg ?= "${DESCRIPTION}  \
 This package contains ELF symbols and related sources for debugging purposes."
 
+SUMMARY_${PN}-ptest ?= "${SUMMARY} - Package test files"
+DESCRIPTION_${PN}-ptest ?= "${DESCRIPTION}  \
+This package contains test directory with the name ${PTEST_NAME} for package test purposes."
+
 SUMMARY_${PN}-dev ?= "${SUMMARY} - Development files"
 DESCRIPTION_${PN}-dev ?= "${DESCRIPTION}  \
 This package contains symbolic links, header files, and \
@@ -263,7 +267,7 @@ SOLIBSDEV_darwin8 = ".dylib"
 SOLIBSDEV_darwin9 = ".dylib"
 
 PACKAGE_BEFORE_PN ?= ""
-PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
+PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PN}-ptest ${PACKAGE_BEFORE_PN} ${PN}"
 PACKAGES_DYNAMIC = "${PN}-locale-*"
 FILES = ""
 
@@ -307,6 +311,12 @@ SECTION_${PN}-dbg = "devel"
 ALLOW_EMPTY_${PN}-dbg = "1"
 RRECOMMENDS_${PN}-dbg = "${PN} (= ${EXTENDPKGV})"
 
+PTEST_PATH ?= "${libdir}/${PN}/ptest"
+FILES_${PN}-ptest = "${PTEST_PATH}/*"
+SECTION_${PN}-ptest = "devel"
+ALLOW_EMPTY_${PN}-ptest = "1"
+RDEPENDS_${PN}-ptest = "${PN} (= ${EXTENDPKGV})"
+
 FILES_${PN}-locale = "${datadir}/locale"
 
 # File manifest
-- 
1.7.5.4




^ permalink raw reply related	[flat|nested] 15+ messages in thread
* Re: [PATCH 1/3] Add -ptest package group
  2012-08-31 14:10 ` [PATCH 1/3] Add -ptest package group Björn Stenberg
@ 2012-09-01 18:21   ` Koen Kooi
  2012-09-03  7:08     ` Björn Stenberg
  0 siblings, 1 reply; 15+ messages in thread
From: Koen Kooi @ 2012-09-01 18:21 UTC (permalink / raw)
  To: Björn Stenberg; +Cc: openembedded-core


Op 31 aug. 2012, om 16:10 heeft Björn Stenberg <bjst@enea.com> het volgende geschreven:

> This patch creates a new package group -ptest to contain the tests from
> each package.

These have always been called ${PN}-tests, why the change?





^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: [PATCH 1/3] Add -ptest package group
  2012-09-01 18:21   ` Koen Kooi
@ 2012-09-03  7:08     ` Björn Stenberg
  0 siblings, 0 replies; 15+ messages in thread
From: Björn Stenberg @ 2012-09-03  7:08 UTC (permalink / raw)
  To: Koen Kooi; +Cc: openembedded-core

Koen Kooi wrote:
> These have always been called ${PN}-tests, why the change?

When I discussed package testing informally before submitting patches, I found that some people tend to lump all types of testing into one big "Test" bucket and get very confused.

By giving this concept a slightly less generic name, we can discuss it specifically without confusing it with other types of tests.

The existing -tests recipes are, as far as I can see, standalone packages that have little in common with each other. In contrast, -ptest packages are designed to be more like -dev or -dbg packages in that they don't have separate recipes and they share a common interface.

-- 
Björn



^ permalink raw reply	[flat|nested] 15+ messages in thread
end of thread, other threads:[~2012-09-03  7:20 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-08-23 14:19 [PATCH 0/3] RFC: Package testing Björn Stenberg
2012-08-23 14:20 ` [PATCH 1/3] Add -ptest package group Björn Stenberg
2012-08-23 14:42   ` Paul Eggleton
2012-08-24  6:51     ` Björn Stenberg
2012-08-24 15:37   ` Richard Purdie
2012-08-28 11:17     ` Björn Stenberg
2012-08-23 14:20 ` [PATCH 2/3] New recipe: ptest-runner Björn Stenberg
2012-08-23 14:20 ` [PATCH 3/3] Enable bash-ptest Björn Stenberg
2012-08-24 15:40   ` Richard Purdie
2012-08-24 19:10   ` Saul Wold
2012-08-23 21:53 ` [PATCH 0/3] RFC: Package testing Burton, Ross
2012-08-24  6:49   ` Björn Stenberg
  -- strict thread matches above, loose matches on Subject: below --
2012-08-31 14:10 Björn Stenberg
2012-08-31 14:10 ` [PATCH 1/3] Add -ptest package group Björn Stenberg
2012-09-01 18:21   ` Koen Kooi
2012-09-03  7:08     ` Björn Stenberg

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.