* [Buildroot] [PATCH 1/6] python-robotframework: New package
2015-08-31 9:59 [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages Denis THULIN
@ 2015-08-31 9:59 ` Denis THULIN
2015-08-31 9:59 ` [Buildroot] [PATCH 2/6] Adds package test infrastructure Denis THULIN
` (6 subsequent siblings)
7 siblings, 0 replies; 12+ messages in thread
From: Denis THULIN @ 2015-08-31 9:59 UTC (permalink / raw)
To: buildroot
A generic test automation framework written in python.
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
v0:
- Generated using python-package-generator.py
- Dependencies on python-zlib and python-pyexpat were added manually
- Modification of packages/Config.in done manually.
v1:
- fixed indentation error on dependencies (was 4 spaces instead of tab)
- Changed hash from md5 to sha256 (modification of scanpypi)
- Fixed typos (modification of scanpypi)
v2:
- Added the md5 hash from pypi
- Fixed typos
v3:
- Changed the hash file
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
package/Config.in | 1 +
package/Config.in.host | 1 +
package/python-robotframework/Config.in | 9 +++++++++
package/python-robotframework/Config.in.host | 8 ++++++++
package/python-robotframework/python-robotframework.hash | 4 ++++
package/python-robotframework/python-robotframework.mk | 16 ++++++++++++++++
6 files changed, 39 insertions(+)
create mode 100644 package/python-robotframework/Config.in
create mode 100644 package/python-robotframework/Config.in.host
create mode 100644 package/python-robotframework/python-robotframework.hash
create mode 100644 package/python-robotframework/python-robotframework.mk
diff --git a/package/Config.in b/package/Config.in
index 783fbb4..c7fadd6 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -656,6 +656,7 @@ menu "external python modules"
source "package/python-pyyaml/Config.in"
source "package/python-pyzmq/Config.in"
source "package/python-requests/Config.in"
+ source "package/python-robotframework/Config.in"
source "package/python-rtslib-fb/Config.in"
source "package/python-serial/Config.in"
source "package/python-setuptools/Config.in"
diff --git a/package/Config.in.host b/package/Config.in.host
index 1e047aa..4526892 100644
--- a/package/Config.in.host
+++ b/package/Config.in.host
@@ -22,6 +22,7 @@ menu "Host utilities"
source "package/patchelf/Config.in.host"
source "package/pwgen/Config.in.host"
source "package/qemu/Config.in.host"
+ source "package/python-robotframework/Config.in.host"
source "package/sam-ba/Config.in.host"
source "package/squashfs/Config.in.host"
source "package/sunxi-tools/Config.in.host"
diff --git a/package/python-robotframework/Config.in b/package/python-robotframework/Config.in
new file mode 100644
index 0000000..9523893
--- /dev/null
+++ b/package/python-robotframework/Config.in
@@ -0,0 +1,9 @@
+config BR2_PACKAGE_PYTHON_ROBOTFRAMEWORK
+ bool "python-robotframework"
+ depends on BR2_PACKAGE_PYTHON
+ select BR2_PACKAGE_PYTHON_ZLIB
+ select BR2_PACKAGE_PYTHON_PYEXPAT
+ help
+ A generic test automation framework
+
+ http://robotframework.org
diff --git a/package/python-robotframework/Config.in.host b/package/python-robotframework/Config.in.host
new file mode 100644
index 0000000..c1f5883
--- /dev/null
+++ b/package/python-robotframework/Config.in.host
@@ -0,0 +1,8 @@
+config BR2_PACKAGE_HOST_PYTHON_ROBOTFRAMEWORK
+ bool "host python-robotframework"
+ depends on BR2_PACKAGE_HOST_PYTHON
+ help
+ A generic test automation framework used for testing buildroot
+ packages
+
+ http://robotframework.org
diff --git a/package/python-robotframework/python-robotframework.hash b/package/python-robotframework/python-robotframework.hash
new file mode 100644
index 0000000..58ff12d
--- /dev/null
+++ b/package/python-robotframework/python-robotframework.hash
@@ -0,0 +1,4 @@
+# md5 from https://pypi.python.org/pypi/robotframework/json
+md5 db4a3330dfaa761c1599ca1064396eee robotframework-2.9.tar.gz
+# sha256 calculated by scanpypi
+sha256 13a0340864e8c7735302ce39f8d3f9d01bed7db7c055e55043396242a615d8f1 robotframework-2.9.tar.gz
diff --git a/package/python-robotframework/python-robotframework.mk b/package/python-robotframework/python-robotframework.mk
new file mode 100644
index 0000000..afef695
--- /dev/null
+++ b/package/python-robotframework/python-robotframework.mk
@@ -0,0 +1,16 @@
+################################################################################
+#
+# python-robotframework
+#
+################################################################################
+
+PYTHON_ROBOTFRAMEWORK_VERSION = 2.9
+PYTHON_ROBOTFRAMEWORK_SOURCE = robotframework-$(PYTHON_ROBOTFRAMEWORK_VERSION).tar.gz
+PYTHON_ROBOTFRAMEWORK_SITE = https://pypi.python.org/packages/source/r/robotframework
+PYTHON_ROBOTFRAMEWORK_SETUP_TYPE = setuptools
+PYTHON_ROBOTFRAMEWORK_LICENSE = Apache-2.0
+PYTHON_ROBOTFRAMEWORK_LICENSE_FILES = LICENSE.txt
+
+
+$(eval $(python-package))
+$(eval $(host-python-package))
--
2.5.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [Buildroot] [PATCH 2/6] Adds package test infrastructure
2015-08-31 9:59 [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages Denis THULIN
2015-08-31 9:59 ` [Buildroot] [PATCH 1/6] python-robotframework: New package Denis THULIN
@ 2015-08-31 9:59 ` Denis THULIN
2015-08-31 9:59 ` [Buildroot] [PATCH 3/6] tests: create variable files through kconfig Denis THULIN
` (5 subsequent siblings)
7 siblings, 0 replies; 12+ messages in thread
From: Denis THULIN @ 2015-08-31 9:59 UTC (permalink / raw)
To: buildroot
This patch adds a test generation step in pkg-generic.mk.
The tests are generated used various information found in the package's
.mk file. Tests can be written either directly inside the .mk file or
in a separate file.
The test are written for robot-framework.
The create-test step is an order only dependency of
TARGET_INSTALL_TARGET. It requires configure to be done.
There are two kinds of tests:
- host tests: Tests that are ran directly on the output/target
directory. Those test are mostly meant to make sure that some files
- target tests: Tests that are meant to be ran from a running
buildroot on itself. Those need to be copied manually inside the
the rootfs and require package python-robotframework (And python 2
as robotframework does not work with python 3).
For any kind of test, you can put global test resources (Keywords,
Variables, Libraries) inside the support/test/<test-kind>/ folder.
Those can be used from any test of this kind.
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
Makefile | 9 ++++-
package/Makefile.in | 1 +
package/pkg-generic.mk | 20 +++++++++-
package/pkg-test.mk | 94 +++++++++++++++++++++++++++++++++++++++++++++
tests/host/resource.robot | 7 ++++
tests/target/resource.robot | 1 +
tests/tests.mk | 17 ++++++++
7 files changed, 146 insertions(+), 3 deletions(-)
create mode 100644 package/pkg-test.mk
create mode 100644 tests/host/resource.robot
create mode 100644 tests/target/resource.robot
create mode 100644 tests/tests.mk
diff --git a/Makefile b/Makefile
index 07fd69b..63c27c1 100644
--- a/Makefile
+++ b/Makefile
@@ -196,6 +196,7 @@ TARGET_DIR := $(BASE_DIR)/target
# .config. HOST_DIR will be overwritten later when .config is included.
HOST_DIR := $(BASE_DIR)/host
GRAPHS_DIR := $(BASE_DIR)/graphs
+TEST_DIR := $(BASE_DIR)/tests
LEGAL_INFO_DIR = $(BASE_DIR)/legal-info
REDIST_SOURCES_DIR_TARGET = $(LEGAL_INFO_DIR)/sources
@@ -422,6 +423,8 @@ include fs/common.mk
include $(BR2_EXTERNAL)/external.mk
+include tests/tests.mk
+
dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
$(HOST_DIR) $(BINARIES_DIR)
@@ -556,7 +559,7 @@ endif
$(TARGETS_ROOTFS): target-finalize
-target-finalize: $(PACKAGES)
+target-finalize: $(PACKAGES) tests-common-resources
@$(call MESSAGE,"Finalizing target directory")
$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
@@ -623,6 +626,7 @@ endif
$(call MESSAGE,"Executing post-build script $(s)"); \
$(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
+
target-post-image: $(TARGETS_ROOTFS) target-finalize
@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
$(call MESSAGE,"Executing post-image script $(s)"); \
@@ -825,7 +829,8 @@ printvars:
clean:
rm -rf $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \
$(BUILD_DIR) $(BASE_DIR)/staging \
- $(LEGAL_INFO_DIR) $(GRAPHS_DIR)
+ $(LEGAL_INFO_DIR) $(GRAPHS_DIR) \
+ $(TEST_DIR)
distclean: clean
ifeq ($(DL_DIR),$(TOPDIR)/dl)
diff --git a/package/Makefile.in b/package/Makefile.in
index 545694f..68a970a 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -400,3 +400,4 @@ include package/pkg-generic.mk
include package/pkg-kconfig.mk
include package/pkg-rebar.mk
include package/pkg-kernel-module.mk
+include package/pkg-test.mk
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 6a7d97e..f688219 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -222,6 +222,17 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
$(Q)touch $@
@$(call step_end,install-staging)
+# Create corresponding tests
+$(BUILD_DIR)/%/.stamp_tests_created:
+ $(call step_start,create-tests)
+ $(foreach hook,$($(PKG)_PRE_CREATE_TESTS_HOOKS),$(call $(hook))$(sep))
+ @$(call MESSAGE,"Creating corresponding tests")
+ $(call TEST_BUILD_ALL)
+ $(foreach hook,$($(PKG)_POST_CREATE_TESTS_HOOKS),$(call $(hook))$(sep))
+ $(Q)touch $@
+ @$(call step_end,create-tests)
+
+
# Install to images dir
$(BUILD_DIR)/%/.stamp_images_installed:
@$(call step_start,install-image)
@@ -472,7 +483,7 @@ $(2)_TARGET_PATCH = $$($(2)_DIR)/.stamp_patched
$(2)_TARGET_EXTRACT = $$($(2)_DIR)/.stamp_extracted
$(2)_TARGET_SOURCE = $$($(2)_DIR)/.stamp_downloaded
$(2)_TARGET_DIRCLEAN = $$($(2)_DIR)/.stamp_dircleaned
-
+$(2)_CREATE_TESTS = $$($(2)_DIR)/.stamp_tests_created
# default extract command
$(2)_EXTRACT_CMDS ?= \
$$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $$(DL_DIR)/$$($(2)_SOURCE) | \
@@ -563,6 +574,9 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
# configure
$$($(2)_TARGET_CONFIGURE): $$($(2)_TARGET_PATCH)
+$(1)-create-tests: $$($(2)_CREATE_TESTS)
+$$($(2)_TARGET_INSTALL_TARGET): | $$($(2)_CREATE_TESTS)
+
$(1)-patch: $$($(2)_TARGET_PATCH)
$$($(2)_TARGET_PATCH): $$($(2)_TARGET_EXTRACT)
# Order-only dependency
@@ -663,6 +677,9 @@ $$($(2)_TARGET_INSTALL_STAGING): PKG=$(2)
$$($(2)_TARGET_INSTALL_IMAGES): PKG=$(2)
$$($(2)_TARGET_INSTALL_HOST): PKG=$(2)
$$($(2)_TARGET_BUILD): PKG=$(2)
+$$($(2)_CREATE_TESTS): PKG=$(2)
+$$($(2)_CREATE_TESTS): pkg=$(1)
+$$($(2)_CREATE_TESTS): ORIGINAL_DIR=$(pkgdir)
$$($(2)_TARGET_CONFIGURE): PKG=$(2)
$$($(2)_TARGET_RSYNC): SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
$$($(2)_TARGET_RSYNC): PKG=$(2)
@@ -833,6 +850,7 @@ endif
$(1)-graph-depends \
$(1)-install \
$(1)-install-host \
+ $(1)-create-tests \
$(1)-install-images \
$(1)-install-staging \
$(1)-install-target \
diff --git a/package/pkg-test.mk b/package/pkg-test.mk
new file mode 100644
index 0000000..34c5494
--- /dev/null
+++ b/package/pkg-test.mk
@@ -0,0 +1,94 @@
+################################################################################
+#
+# This file contains everything that is necessary in order to create
+# Robotframework tests for any buildroot package. Tests are written either
+# directly inside the package's .mk file or inside specified directories and
+# files. Common ressource files and libraries can be found in support/tests.
+#
+################################################################################
+
+# TEST_BUILD_ALL is called by pkg-generic. It Creates two kinds of tests:
+# Tests that are meant to be ran on the output/target directory, called host
+# tests.
+# And tests that are ran directly on the running buildroot, called target
+# tests.
+# Those need to be copied to the rootfs and require package
+# python-robotframework to run (And python 2 as robot framework does not
+# support python 3 yet).
+define TEST_BUILD_ALL
+ $(eval HOST_TEST := )
+ $(eval TEST_KIND := target)
+ $(call TEST_BUILD)
+ $(eval HOST_TEST := _HOST)
+ $(eval TEST_KIND := host)
+ $(call TEST_BUILD)
+endef
+
+# To create tests, we need a lot of information:
+# - $(PKG)_TEST_DIR is the test output directory
+# - $(PKG)_TEST_SUITE is the tests suite being created
+# - $(ORIGINAL_DIR)/$(TEST_KIND)_test_material is a set of directory from
+# which tests/resource/libraries are to be copied.
+# - $(TEST_DIR)/$(TEST_KIND)/resources/resource.robot is a global resource
+# file.
+# A tiny wrapper around the telnet library is also provided
+# - $(PKG)_TEST_ORIGINAL is a robotframework test file that is merged with
+# the test suite being built. Test that take more than two lines should be
+# written there rather than in the .mk file (and use variables as much as
+# possible)
+# - $(PKG)_TEST_VARIABLES is a list of variables used by test suites.
+# Those should look like myvar="Itsvalue". Those variables will be formated
+# to work inside the test suite
+# - $(PKG)_TEST_INSTALL_LOCATION should contain a file or a folder that makes
+# sure the package is installed inside the target directory. A test is
+# automatically created if it is present.
+# - $(PKG)_TEST_CASES contains all test cases that are written inside the .mk
+# file. Use a define to write your tests this way. Please use TEST_ORIGINAL
+# rather than this if your test is more than two lines long.
+# - $(PKG)_TEST_KEYWORDS contains keywords from the .mk file.
+define TEST_BUILD
+ $(eval $(PKG)_TEST_DIR := $(TEST_DIR)/$(TEST_KIND)/$(pkg))
+ $(eval $(PKG)_TEST_SUITE := $($(PKG)_TEST_DIR)/$(pkg).robot)
+
+ @mkdir -p $($(PKG)_TEST_DIR)
+ if [ ! -z $($(PKG)$(HOST_TEST)_TEST_MATERIAL) ]; then \
+ cp -r $(ORIGINAL_DIR)$($(PKG)$(HOST_TEST)_TEST_MATERIAL) $($(PKG)_TEST_DIR); \
+ fi
+
+ @echo '*** Settings ***' > $($(PKG)_TEST_SUITE)
+ @echo "Resource ../resources/resource.robot" >> $($(PKG)_TEST_SUITE)
+
+ @if [ ! -z $($(PKG)$(HOST_TEST)_TEST_ORIGINAL) ]; then \
+ sed -n -e '/\*\*\* Settings \*\*\*/,/\*\*\*/ {/\*\*\*/b; /^$$/b; p}' \
+ $(ORIGINAL_DIR)$($(PKG)$(HOST_TEST)_TEST_ORIGINAL) >> $($(PKG)_TEST_SUITE); \
+ fi
+
+ @echo '*** Variables ***' >> $($(PKG)_TEST_SUITE)
+ @for var in $($(PKG)$(HOST_TEST)_TEST_VARIABLES); do \
+ echo $$var | sed 's/^\(.*\)=\(.*\)$$/$${\1}= \2/' >> $($(PKG)_TEST_SUITE); \
+ done
+ @if [ ! -z $($(PKG)$(HOST_TEST)_TEST_ORIGINAL) ]; then \
+ sed -n -e '/\*\*\* Variables \*\*\*/,/\*\*\*/ {/\*\*\*/b; /^$$/b; p}' \
+ $(ORIGINAL_DIR)$($(PKG)$(HOST_TEST)_TEST_ORIGINAL) >> $($(PKG)_TEST_SUITE); \
+ fi
+
+ @echo '*** Test Cases ***' >> $($(PKG)_TEST_SUITE)
+ @if [ $(TEST_KIND) = host ]; then \
+ if [ ! -z $($(PKG)_TEST_INSTALL_LOCATION) ] ; then \
+ echo '$(pkg) Is Installed' >> $($(PKG)_TEST_SUITE); \
+ echo ' [Template] Package Installed On Target' >> $($(PKG)_TEST_SUITE); \
+ echo ' $($(PKG)_TEST_INSTALL_LOCATION)' >> $($(PKG)_TEST_SUITE); \
+ fi; \
+ fi
+ @echo -e $($(PKG)$(HOST_TEST)_TEST_CASES) >> $($(PKG)_TEST_SUITE)
+ @if [ ! -z $($(PKG)$(HOST_TEST)_TEST_ORIGINAL) ]; then \
+ sed -n -e '/\*\*\* Test Cases \*\*\*/,/\*\*\*/ {/\*\*\*/b; /^$$/b; p}' \
+ $(ORIGINAL_DIR)$($(PKG)$(HOST_TEST)_TEST_ORIGINAL) >> $($(PKG)_TEST_SUITE); \
+ fi
+
+ @echo '*** Keywords ***' >> $($(PKG)_TEST_SUITE)
+ @if [ ! -z $($(PKG)$(HOST_TEST)_TEST_ORIGINAL) ]; then \
+ sed -n -e '/\*\*\* Keywords \*\*\*/,/\*\*\*/ {/\*\*\*/b; /^$$/b; p}' \
+ $(ORIGINAL_DIR)$($(PKG)$(HOST_TEST)_TEST_ORIGINAL) >> $($(PKG)_TEST_SUITE); \
+ fi
+endef
diff --git a/tests/host/resource.robot b/tests/host/resource.robot
new file mode 100644
index 0000000..650d4c8
--- /dev/null
+++ b/tests/host/resource.robot
@@ -0,0 +1,7 @@
+*** Settings ***
+Library OperatingSystem
+Variables common_variables.py
+*** Keywords ***
+Package Installed On Target
+ [arguments] ${File}
+ OperatingSystem.Should Exist ${target_dir}/${File}
diff --git a/tests/target/resource.robot b/tests/target/resource.robot
new file mode 100644
index 0000000..c742ffc
--- /dev/null
+++ b/tests/target/resource.robot
@@ -0,0 +1 @@
+*** Variables ***
diff --git a/tests/tests.mk b/tests/tests.mk
new file mode 100644
index 0000000..a246c6e
--- /dev/null
+++ b/tests/tests.mk
@@ -0,0 +1,17 @@
+tests-common-resources: host-python-robotframework
+ rm -rf $(TEST_DIR)/host/resources
+ rm -rf $(TEST_DIR)/target/resources
+ mkdir -p $(TEST_DIR)/host
+ cp -r tests/host $(TEST_DIR)/host/resources
+ echo target_dir = \"$(TARGET_DIR)\" >> $(TEST_DIR)/host/resources/common_variables.py
+ mkdir -p $(TEST_DIR)/target
+ cp -r tests/target $(TEST_DIR)/target/resources
+
+
+PYBOT = $(HOST_DIR)/usr/bin/pybot
+PYBOT_OPTS ?= -d $(TEST_DIR)/logs -x xunit.xml
+
+tests: tests-common-resources
+ @$(PYBOT) $(PYBOT_OPTS) $(TEST_DIR)/host
+
+.PHONY: test-common-resources tests
--
2.5.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [Buildroot] [PATCH 3/6] tests: create variable files through kconfig
2015-08-31 9:59 [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages Denis THULIN
2015-08-31 9:59 ` [Buildroot] [PATCH 1/6] python-robotframework: New package Denis THULIN
2015-08-31 9:59 ` [Buildroot] [PATCH 2/6] Adds package test infrastructure Denis THULIN
@ 2015-08-31 9:59 ` Denis THULIN
2015-08-31 9:59 ` [Buildroot] [PATCH 4/6] tests: Adds user manual entry Denis THULIN
` (4 subsequent siblings)
7 siblings, 0 replies; 12+ messages in thread
From: Denis THULIN @ 2015-08-31 9:59 UTC (permalink / raw)
To: buildroot
Install of target tests can now be selected from menuconfig
This patch adds a new menu entry in Config.in in order to configure
the test environnment.
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
Config.in | 6 ++++++
tests/Config.in | 8 ++++++++
tests/tests.mk | 3 +++
3 files changed, 17 insertions(+)
create mode 100644 tests/Config.in
diff --git a/Config.in b/Config.in
index 8b1bd8d..51e0ea2 100644
--- a/Config.in
+++ b/Config.in
@@ -624,3 +624,9 @@ menu "User-provided options"
source "$BR2_EXTERNAL/Config.in"
endmenu
+
+menu "Test Configuration Settings"
+
+source "tests/Config.in"
+
+endmenu
diff --git a/tests/Config.in b/tests/Config.in
new file mode 100644
index 0000000..937753c
--- /dev/null
+++ b/tests/Config.in
@@ -0,0 +1,8 @@
+comment "Target tests require robotframework"
+ depends on !BR2_PACKAGE_PYTHON_ROBOTFRAMEWORK
+
+config BR2_TEST_TARGET
+ bool "Install tests to target"
+ depends on BR2_PACKAGE_PYTHON_ROBOTFRAMEWORK
+ help
+ Enable this if you want to install tests on your target
diff --git a/tests/tests.mk b/tests/tests.mk
index a246c6e..ad05f7c 100644
--- a/tests/tests.mk
+++ b/tests/tests.mk
@@ -6,6 +6,9 @@ tests-common-resources: host-python-robotframework
echo target_dir = \"$(TARGET_DIR)\" >> $(TEST_DIR)/host/resources/common_variables.py
mkdir -p $(TEST_DIR)/target
cp -r tests/target $(TEST_DIR)/target/resources
+ifeq ($(BR2_TEST_TARGET),y)
+ cp -r $(TEST_DIR)/target $(TARGET_DIR)/usr/share/buildroot_tests
+endif
PYBOT = $(HOST_DIR)/usr/bin/pybot
--
2.5.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [Buildroot] [PATCH 4/6] tests: Adds user manual entry
2015-08-31 9:59 [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages Denis THULIN
` (2 preceding siblings ...)
2015-08-31 9:59 ` [Buildroot] [PATCH 3/6] tests: create variable files through kconfig Denis THULIN
@ 2015-08-31 9:59 ` Denis THULIN
2015-08-31 9:59 ` [Buildroot] [PATCH 5/6] flask: Adds robotframework tests Denis THULIN
` (3 subsequent siblings)
7 siblings, 0 replies; 12+ messages in thread
From: Denis THULIN @ 2015-08-31 9:59 UTC (permalink / raw)
To: buildroot
Adds user manual section about how to use and write tests
The documentation covers everything from writing your own tests to setting up
your configuration to test throught telnet and qemu.
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
docs/manual/common-usage.txt | 2 +
docs/manual/test-infrastructure.txt | 239 ++++++++++++++++++++++++++++++++++++
2 files changed, 241 insertions(+)
create mode 100644 docs/manual/test-infrastructure.txt
diff --git a/docs/manual/common-usage.txt b/docs/manual/common-usage.txt
index 5b27b1f..69c90b6 100644
--- a/docs/manual/common-usage.txt
+++ b/docs/manual/common-usage.txt
@@ -276,3 +276,5 @@ BR2_GRAPH_OUT=png make graph-build
include::eclipse-integration.txt[]
include::advanced.txt[]
+
+include::test-infrastructure.txt[]
diff --git a/docs/manual/test-infrastructure.txt b/docs/manual/test-infrastructure.txt
new file mode 100644
index 0000000..2a74660
--- /dev/null
+++ b/docs/manual/test-infrastructure.txt
@@ -0,0 +1,239 @@
+// -*- mode:doc; -*-
+// vim: set syntax=asciidoc:
+
+=== Test Infrastructure
+
+You may want to test your packages in Buildroot. Buildroot provides a test
+generation infrastructure. Those tests are Robot Framework tests,
+see https://github.com/Robotframework/Robotframework/blob/master/INSTALL.rst[here]
+for installation instructions.
+
+There can be two kinds of tests: host tests and target tests.
+
+If you want to run host tests, run:
+
+----------------------------
+make tests-host
+----------------------------
+
+or
+
+----------------------------
+make tests
+----------------------------
+
+==== Host tests
+
+Host tests are NOT tests for host packages. Host tests are tests that are ran
+from your build machine, on your build machine.
+
+Those tests are mostly meant to make sure that some files are correctly
+installed in the output/target directory.
+
+You should use Keywords from the http://Robotframework.org/Robotframework/latest/libraries/OperatingSystem.html[OperatingSystem Library]
+to make sure that the installation happened correctly (+File Should Exist+,
++File Should Not Exist+, +Directory Should Exist+,
++Directory Should Not Be Empty+, etc...).
+
+==== Target Tests
+
+Target tests are ran from a running target on itself, and thus require
+Robot Framework to be installed on the target. You also need to manually copy
+the target tests to your board.
+
+Target tests are useful when you already have tests written for packages.
+
+==== Writing tests
+
+Tests can either be written in full flegged Robot Framework tests files or
+directly in the <package>.mk file.
+
+===== Configuration dependent tests
+
+You should avoid writing tests directly inside the <package>.mk file unless a
+test depends on your configuration.
+
+Let's see what the <package>.mk looks like.
+
+------------------------------
+01: ################################################################################
+02: #
+03: # libfoo
+04: #
+05: ################################################################################
+06:
+07: LIBFOO_VERSION = 1.0
+08: LIBFOO_SOURCE = libfoo-$(LIBFOO_VERSION).tar.gz
+09: LIBFOO_SITE = http://www.foosoftware.org/download
+10: LIBFOO_LICENSE = GPLv3+
+11: LIBFOO_LICENSE_FILES = COPYING
+12: LIBFOO_INSTALL_STAGING = YES
+13: LIBFOO_CONFIG_SCRIPTS = libfoo-config
+14: LIBFOO_DEPENDENCIES = host-libaaa libbbb
+15:
+16: ifeq ($(BR2_INT_BUSYBOX)$(BR2_INIT_SYSV),y)
+17: LIBFOO_TEST_INSTALL_LOCATION = /etc/init.d/S90libfoo
+18: else ifeq ($(BR2_INIT_SYSTEMD),y)
+19: LIBFOO_TEST_INSTALL_LOCATION = /usr/lib/systemd/system/libfoo.service
+20: endif
+21:
+22: LIBFOO_HOST_TEST_VARIABLES = bar=/path/to/bar
+23: define LIBFOO_HOST_TEST_CASES =
+24: 'File Bar Exists\n'\
+25: ' File Should Exist ${bar}\n\n'
+26: endef
+27:
+28: LIBFOO_TEST_ORIGINAL = libfoo.robot
+29: ifeq ($(BR2_INT_BUSYBOX)$(BR2_INIT_SYSV),y)
+30: LIBFOO_TEST_VARIABLES = expected_result=1
+31: else ifeq ($(BR2_INIT_SYSTEMD),y)
+32: LIBFOO_TEST_VARIABLES = expected_result=0
+33: endif
+34:
+35: $(eval $(generic-package))
+--------------------------------
+
+From Line 16 to 20, we specify an installation file for libfoo that depends on
+the init system. When writing tests, make will automatically create a host test
+called +Libfoo Is Installed+ if variable +LIBFOO_TEST_INSTALL_LOCATION+ is
+defined.
+
+On line 22, we specify that host tests will use a variable +bar+ whose value is
+'/path/to/bar' using LIBFOO_HOST_TEST_VARIABLES. In the test suite, it will look
+like
+
+--------------------------------
+${bar}= /path/to/bar
+--------------------------------
+
+From line 23 to 26, we define a host test case which checks that the file bar
+exists. Test are stored into a multiline variable called
++LIBFOO_HOST_TEST_CASES+.
+
+On line 28, we define the original file for target tests. This file will be
+meged with information from the mk file.
+From line 28 to 31, we define a variable that will be used by target tests.
+See the '<libfoo>package/libfoo.robot' file:
+
+------------------------------
+01: *** Settings ***
+02: Documentation Target tests of buildroot package libfoo
+03: Library Process
+04: *** Test Cases ***
+05: Example
+06: ${out}= Run Process libfoo
+07: Should Be Equal As Integers ${expected_result} ${out.rc}
+--------------------------------
+
+===== Independent tests
+
+Most tests for packages will depend on your configuration. You should specify
+those files inside variables +$(TARGET_TEST_MATERIAL)+ and
++$(HOST_TEST_MATERIAL)+.
+The content of those folder is always copied to the
+'output/tests/<test_kind>/libfoo/' folder.
+
+Those will be ran as different test suites from the one created by the
+makefile.
+
+==== Test syntax reference
+
+Variables for host tests are specified with +LIBFOO_HOST_TEST+ and those for
+target tests are just +LIBFOO_TEST+.
+
+The list of test related variables that can be set in a .mk file without test
+kind (assuming thepackage name is +libfoo+)
+
+* +LIBFOO_TEST_ORIGINAL+ is a Robot Framework test file that is merged with
+ the test suite being built. Test should preferably be written there rather
+ than in the .mk file (and use variables as much as possible).
+
+* +LIBFOO_TEST_VARIABLES+ is a list of variables used by test suites.
+ Those should look like myvar=Itsvalue. Those variables will be formated
+ to work inside the test suite.
+
+* +LIBFOO_TEST_CASES+ contains all test cases that are written inside the .mk
+ file. Use a define to write your tests this way. Please use
+ +LIBFOO_TEST_ORIGINAL+ rather than this if you can (for readability).
+
+The list of variables that do not depend on the test kind:
+
+* +LIBFOO_TEST_INSTALL_LOCATION+ should contain a file or a folder that exists
+ in the target directory if the package was installed correctly. A host test
+ is automatically created if it is present.
+
+==== General advices about writing tests
+
+Even though you can use both the original file and the .mk file to write tests,
+it is not a very good idea to use both a lot if you don't need to.
+
+Variables are easier to write in the original file, especially lists and
+dictionnaries. Hence, in the +.mk+ file, you should create Robot Framework
+variables directly from existing variables in your configuration and in the
+original file, you should use those variables.
+Example: If you need libfoo's version to create Robot Framework variables
+version_minor and version major. +libfoo.mk+ will be :
+
+------------------------------
+...
+LIBFOO_VERSION = 1.0
+...
+LIBFOO_TEST_ORIGINAL = target_test_original.robot
+LIBFOO_TEST_VARIABLES += version=$(LIBFOO_VERSION)
+...
+------------------------------
+
+And +target_test_original.robot+ will look like
+
+------------------------------
+...
+*** Variables ***
+${version_major}= ${version.split('.')[0]}
+${version_minor}= ${version.split('.')[1]}
+*** Test cases ***
+...
+------------------------------
+
+Where to write test cases depends on your situation, if your tests do not
+change when your configuration change, you should write those either in
+external test suites (inside <test-kind>_test_material) or in the original
+file.
+
+If your whole test suite drastically change because of a single variable,
+you should probably have multiple original files and chose the correct one
+according to the variable in the +.mk+ file.
+Example: When you want to test d?mons, you test will difer wether you use
+systemd or System V.
+
+If most of your tests individually depends on different variables having
+different values. You should have those tests in the mkfile.
+
+If a lot of your test suites require keywords or variables, you should write
+those inside resource files/variable files/librairies located in
+'<buildroot>/tests/<test-kind>/<yourfile>' and in your test suite, there should
+be:
+
+------------------------------
+*** Settings ***
+...
+Librairy ../resources/<your-librairy>.py
+Variables ../resources/<your-variable-file>.py
+Resource ../resources/<your-resource-file>.robot
+...
+------------------------------
+
+==== Useful links for writing tests:
+
+The Robot Framework documentation page contain a lot of usefull links:
+see http://Robotframework.org/#documentation[here]
+
+You should start with the https://github.com/Robotframework/QuickStartGuide/blob/master/QuickStart.rst[Quick Start Guide]
+
+Documentation for librairies provided with Robot Framework can be found inside
+the http://Robotframework.org/Robotframework/#user-guide[User Guide]
+
+Also very useful:
+
+* http://www.slideshare.net/pekkaklarck/robot-framework-dos-and-donts[dos and don't]
+
+* https://code.google.com/p/Robotframework/wiki/HowToWriteGoodTestCases[How to write good test cases]
--
2.5.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [Buildroot] [PATCH 5/6] flask: Adds robotframework tests
2015-08-31 9:59 [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages Denis THULIN
` (3 preceding siblings ...)
2015-08-31 9:59 ` [Buildroot] [PATCH 4/6] tests: Adds user manual entry Denis THULIN
@ 2015-08-31 9:59 ` Denis THULIN
2015-08-31 9:59 ` [Buildroot] [PATCH 6/6] python: Adds tests Denis THULIN
` (2 subsequent siblings)
7 siblings, 0 replies; 12+ messages in thread
From: Denis THULIN @ 2015-08-31 9:59 UTC (permalink / raw)
To: buildroot
Adds a 'host' test to check that the flask folder is created,
a 'target' test that checks that flask runs correctly
The 'host' test is ran on the target folder, it merely checks that
the flask folder exists
The 'target' test starts a flask process on a specified port and localhost
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
package/python-flask/python-flask.mk | 5 +++++
.../target_test_material/target_test.robot | 25 ++++++++++++++++++++++
2 files changed, 30 insertions(+)
create mode 100644 package/python-flask/target_test_material/target_test.robot
diff --git a/package/python-flask/python-flask.mk b/package/python-flask/python-flask.mk
index 8aa675e..18dc846 100644
--- a/package/python-flask/python-flask.mk
+++ b/package/python-flask/python-flask.mk
@@ -12,4 +12,9 @@ PYTHON_FLASK_LICENSE = BSD-3c
PYTHON_FLASK_LICENSE_FILES = LICENSE
PYTHON_FLASK_DEPENDENCIES = python-werkzeug python-jinja2 python-itsdangerous
+
+PYTHON_FLASK_TEST_INSTALL_LOCATION = /usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages/flask
+
+PYTHON_FLASK_TEST_MATERIAL = target_test_material
+
$(eval $(python-package))
diff --git a/package/python-flask/target_test_material/target_test.robot b/package/python-flask/target_test_material/target_test.robot
new file mode 100644
index 0000000..78f4d38
--- /dev/null
+++ b/package/python-flask/target_test_material/target_test.robot
@@ -0,0 +1,25 @@
+*** Settings ***
+Documentation This is a simple test suite meant to make sure that flask works
+... properly. Please ensure that no other program is using tcp port
+... 5000 before running it.
+Library Process
+*** Variables ***
+${flask_range}= "127.0.0.1"
+${flask_port}= 5000
+${flask_address}= "127.0.0.1"
+*** Test Cases ***
+Test Port
+ [setup] Start Flask With Hello World
+ [Teardown] Kill Flask
+ Wait Until Keyword Succeeds 2 sec 10 sec Wget Returns OK
+*** Keywords ***
+Start Flask With Hello World
+ ${handle}= Start Process python2 -c from flask import Flask; app \= Flask(__name__); app.route(\"/\")(lambda: \"hello world\"); app.run(host\=${flask_range},port\=${flask_port})
+ Set Suite Variable ${HANDLE} ${handle}
+
+Kill Flask
+ Terminate Process ${HANDLE}
+
+Wget Returns Ok
+ ${result}= Run Process wget http://127.0.0.1:5000
+ Should Be Equal As Integers 0 ${result.rc}
--
2.5.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [Buildroot] [PATCH 6/6] python: Adds tests
2015-08-31 9:59 [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages Denis THULIN
` (4 preceding siblings ...)
2015-08-31 9:59 ` [Buildroot] [PATCH 5/6] flask: Adds robotframework tests Denis THULIN
@ 2015-08-31 9:59 ` Denis THULIN
2015-08-31 12:48 ` [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages Thomas Petazzoni
2015-09-06 22:46 ` Arnout Vandecappelle
7 siblings, 0 replies; 12+ messages in thread
From: Denis THULIN @ 2015-08-31 9:59 UTC (permalink / raw)
To: buildroot
Adds a 'host' test (usr/bin/python exists) and a few target tests.
Current target tests:
Check Python Version: Should be 2
Check Python Math: import math module and check that it works
Check Python Libc: import ctypes and load libc
Check Python Zlib: import zlib
Check Python SSL: import ssl
Tests of zlib and ssl make sure that the result is correct according
to the buildroot configuration.
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
Signed-off-by: Denis THULIN <denis.thulin@openwide.fr>
---
package/python/python.mk | 17 +++++++++++++++++
package/python/python.robot | 32 ++++++++++++++++++++++++++++++++
2 files changed, 49 insertions(+)
create mode 100644 package/python/python.robot
diff --git a/package/python/python.mk b/package/python/python.mk
index 80ffbde..047567a 100644
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -214,6 +214,23 @@ endif
# Provided to other packages
PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/:$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages/
+# Host Tests
+PYTHON_TEST_INSTALL_LOCATION := /usr/bin/python
+
+# Target tests
+PYTHON_TEST_ORIGINAL = python.robot
+ifeq ($(BR2_PACKAGE_PYTHON_ZLIB),y)
+ PYTHON_TEST_VARIABLES += zlib_return_code=0
+else
+ PYTHON_TEST_VARIABLES += zlib_return_code=1
+endif
+
+ifeq ($(BR2_PACKAGE_PYTHON_SSL),y)
+ PYTHON_TEST_VARIABLES += ssl_return_code=0
+else
+ PYTHON_TEST_VARIABLES += ssl_return_code=1
+endif
+
$(eval $(autotools-package))
$(eval $(host-autotools-package))
diff --git a/package/python/python.robot b/package/python/python.robot
new file mode 100644
index 0000000..9ea3203
--- /dev/null
+++ b/package/python/python.robot
@@ -0,0 +1,32 @@
+*** Settings ***
+Documentation Test suite for package python of buildroot
+Library Process
+*** Variables ***
+*** Test Cases ***
+Python Version
+ ${result}= Run Process python --version stderr=STDOUT
+ Should Start With ${result.stdout} Python 2
+
+Python Math
+ [Template] Check Python Return Code of Python Command
+ 0 import math; math.floor(12.3)
+
+Python Libc
+ [Template] Check Python Return Code of Python Command
+ 0 import ctypes; libc = ctypes.dll.LoadLibrary(\\"libc.so\\"); print libc.time(None)
+
+Python Zlib
+ [Template] Check Python Return Code of Python Command
+ ${zlib_return_code} import zlib
+
+Python SSL
+ [Template] Check Python Return Code of Python Command
+ ${ssl_return_code} import ssl
+
+*** Keywords ***
+Check Python Return Code of Python Command
+ [arguments] ${return_code} ${command}
+ ${result}= Run Process python -c "${command}"
+ Should Be Equal As Integers ${return_code} ${result.rc}
+
+
--
2.5.0
^ permalink raw reply related [flat|nested] 12+ messages in thread* [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages
2015-08-31 9:59 [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages Denis THULIN
` (5 preceding siblings ...)
2015-08-31 9:59 ` [Buildroot] [PATCH 6/6] python: Adds tests Denis THULIN
@ 2015-08-31 12:48 ` Thomas Petazzoni
2015-09-01 9:52 ` Denis Thulin
2015-09-06 22:46 ` Arnout Vandecappelle
7 siblings, 1 reply; 12+ messages in thread
From: Thomas Petazzoni @ 2015-08-31 12:48 UTC (permalink / raw)
To: buildroot
Denis,
On Mon, 31 Aug 2015 11:59:05 +0200, Denis THULIN wrote:
> This patch series adds a test generation infrastructure for packages to
> Buildroot.
> The generated tests are robotframework tests.
>
> While it is always necessary to test that packages are installed
> correctly and work in an intended way, Buildroot does not currently
> provide any tests of package nor any way to automate the process of
> creating tests depending on your configuration or even share tests you
> have written for packages. You have to write tests manually almost
> everytime you start a new project with buildroot. This can cost a lot
> of time.
>
> This patch series is intented as a solution for that problem.
Thanks a lot for working on the topic of runtime testing, which is an
important topic that needs to be addressed in the Buildroot community.
However, I have to say I personally dislike quite a bit some of the
principles of your solutions:
* I don't like the Robot Framework. It requires you to learn another
new, specialized (and weird) language to write tests, instead of
simply writing them in Python. The Robot Framework is probably good
when the people who will write the tests are not developers. But
that's not the case of Buildroot users and developers. Many of them
probably already know Python (or at least the basics of it), while
nobody knows this specialized language. It will be a barrier to
getting tests written by Buildroot contributors.
* I don't think tying tests to packages is the correct idea. Some
tests need to be done for filesystem formats, some tests for core
mechanisms (like checking BR2_EXTERNAL functionality, checking core
package infrastructure mechanisms, etc.). I don't think the test
mechanism should modify Buildroot itself.
Also, this may be me not understanding properly, but I fail to see
where are your test cases. My impression is that you add tests to
packages, and then your idea is that whenever a package is enabled in a
given configuration, you will run the tests of this package. But who
will generate the configurations to be tested? They surely cannot be
randomly generated, because many combinations of packages do not make
sense (for example a random configuration that has Dropbear and OpenSSH
enabled will build fine, but will completely fail at runtime because
both will try to bind on port 22).
I certainly do not claim that my proposal at
https://github.com/tpetazzoni/buildroot-runtime-test was ideal and
perfect, but it is IMO much more in-line with what we need for
Buildroot: fully written in a language that is generally well-known,
and a simple list of test-cases that associate a configuration to be
built/executed with a set of actions to be verified.
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 12+ messages in thread* [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages
2015-08-31 12:48 ` [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages Thomas Petazzoni
@ 2015-09-01 9:52 ` Denis Thulin
0 siblings, 0 replies; 12+ messages in thread
From: Denis Thulin @ 2015-09-01 9:52 UTC (permalink / raw)
To: buildroot
Hi Thomas,
Thanks for answering that fast.
It seems I the use case for this patch series is not clear.
I'm trying to create tests for Buildroot users, those tests
are not intended for autobuilds (They could be used, but that
was not what I intented when I wrote this patch series).
----- Le 31 Ao? 15, ? 14:48, Thomas Petazzoni thomas.petazzoni at free-electrons.com a ?crit :
> Denis,
>
> On Mon, 31 Aug 2015 11:59:05 +0200, Denis THULIN wrote:
>> This patch series adds a test generation infrastructure for packages to
>> Buildroot.
>> The generated tests are robotframework tests.
>>
>> While it is always necessary to test that packages are installed
>> correctly and work in an intended way, Buildroot does not currently
>> provide any tests of package nor any way to automate the process of
>> creating tests depending on your configuration or even share tests you
>> have written for packages. You have to write tests manually almost
>> everytime you start a new project with buildroot. This can cost a lot
>> of time.
>>
>> This patch series is intented as a solution for that problem.
>
> Thanks a lot for working on the topic of runtime testing, which is an
> important topic that needs to be addressed in the Buildroot community.
>
> However, I have to say I personally dislike quite a bit some of the
> principles of your solutions:
>
> * I don't like the Robot Framework. It requires you to learn another
> new, specialized (and weird) language to write tests, instead of
> simply writing them in Python. The Robot Framework is probably good
> when the people who will write the tests are not developers. But
> that's not the case of Buildroot users and developers. Many of them
> probably already know Python (or at least the basics of it), while
> nobody knows this specialized language. It will be a barrier to
> getting tests written by Buildroot contributors.
There seems to be a big misunderstanding here. This patch series is
not meant for the same use case as
https://github.com/tpetazzoni/buildroot-runtime-test or
what I wrote at https://github.com/etkaDT/Rfw-buildroot-tests
The test infrastructure is meant to be used by Buildroot users rather
than Buildroot devs. The goal here is for users to test if their
packages work properply with their Buildroot configuration, not
to test if the packages work in general.
In that context, I feel that it is important that anybody can understand
the tests results.
>
> * I don't think tying tests to packages is the correct idea. Some
> tests need to be done for filesystem formats, some tests for core
> mechanisms (like checking BR2_EXTERNAL functionality, checking core
> package infrastructure mechanisms, etc.). I don't think the test
> mechanism should modify Buildroot itself.
>
Well those tests are not included in the use case I'm working on and I
don't think they should be.
> Also, this may be me not understanding properly, but I fail to see
> where are your test cases. My impression is that you add tests to
> packages, and then your idea is that whenever a package is enabled in a
> given configuration, you will run the tests of this package. But who
> will generate the configurations to be tested? They surely cannot be
> randomly generated, because many combinations of packages do not make
> sense (for example a random configuration that has Dropbear and OpenSSH
> enabled will build fine, but will completely fail at runtime because
> both will try to bind on port 22).
You understand how these tests works.
Again, those tests are not meant to work with random configurations
but rather with a user's configuration.
In that case, the user will notice that his configuration is not working
and will change it so that his packages work correctly.
>
> I certainly do not claim that my proposal at
> https://github.com/tpetazzoni/buildroot-runtime-test was ideal and
> perfect, but it is IMO much more in-line with what we need for
> Buildroot: fully written in a language that is generally well-known,
> and a simple list of test-cases that associate a configuration to be
> built/executed with a set of actions to be verified.
I agree on that point,
This patch series is just meant for a different use case.
Best regards,
Denis
>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages
2015-08-31 9:59 [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages Denis THULIN
` (6 preceding siblings ...)
2015-08-31 12:48 ` [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages Thomas Petazzoni
@ 2015-09-06 22:46 ` Arnout Vandecappelle
2015-09-07 8:00 ` Denis Thulin
7 siblings, 1 reply; 12+ messages in thread
From: Arnout Vandecappelle @ 2015-09-06 22:46 UTC (permalink / raw)
To: buildroot
Hi Denis,
On 31-08-15 11:59, Denis THULIN wrote:
> This patch series adds a test generation infrastructure for packages to
> Buildroot.
> The generated tests are robotframework tests.
>
> While it is always necessary to test that packages are installed
> correctly and work in an intended way, Buildroot does not currently
> provide any tests of package nor any way to automate the process of
> creating tests depending on your configuration or even share tests you
> have written for packages. You have to write tests manually almost
> everytime you start a new project with buildroot. This can cost a lot
> of time.
Just to be clear: the intention of this series is to make it possible to
include in the buildroot package a way to specify some tests to run on the host
and on the target, where the purpose of these tests is to validate that the
package has been built correctly. So it is not exactly testing the buildroot
infrastructure (like what Thomas is trying to do), and it's not really running
the tests included with the package itself (these are not built and installed),
but something in-between.
However, as far as I can see, this patch series only solves a small part of the
problem. It basically creates and installs a bunch of .robot files, and that's
it, right? You still have to make the robot framework run on these files. And
you still have to express in the robot framework or in jenkins the dependencies
to make sure the tests are first built and then executed. Correct?
In addition, it seems to me that the munging done by the TEST_BUILD macro has
little to offer above simply copying the .robot file from the package directory
to the test directory. But possibly I don't sufficiently understand what's going on.
And finally, buildroot tries not to enforce any particular policy or tool on
the user. So, much as I like robot, requiring the tests to use this specific
framework is definitely a no-go (at least, if it's meant to be used by end
users). Especially since it means that a rootfs with python3 can't be tested...
I'm not entirely sure what the way to go is. It definitely is nice to have some
kind of test description as part of the packages - I like that more than the way
Thomas did it in buildroot-runtime-test, where the buildroot tree is basically
replicated externally for the tests. But the exact location of the test files is
just a minor detail I guess.
Bottom line: I'm not in favour of this series either. Of course, the first
patch adding python-robotframework is still relevant independently.
Regards,
Arnout
>
> This patch series is intented as a solution for that problem.
>
> Why Robotframework Tests:
> Robotframework is a widely used test automation framework that allow
> writing tests in various languages (Python, Java and even C), provides
> useful test librairies and allow writing tests in various syntaxes.
> Robotframework tests are meant to be easy to read by people whom did
> not write the tests, and that can definitly be useful.
>
> List of patches from the series :
> * Patch 1/6 adds a test generation step before install target step.
> * Patch 2/6 adds useful keywords for testing through Telnet and Qemu
> * Patch 3/6 adds menuconfig options to ease the configuration required
> for testing through telnet and Qemu
> * Patch 4/6 adds documentation for patches 1,2 and 3
> * Patch 5/6 adds exemple tests for package python-flask
> * patch 6/6 adds exemple tests for package python (inspired by tests
> from https://github.com/tpetazzoni/buildroot-runtime-test)
>
> Denis THULIN (6):
> python-robotframework: New package
> Adds package test infrastructure
> tests: create variable files through kconfig
> tests: Adds user manual entry
> flask: Adds robotframework tests
> python: Adds tests
>
> Config.in | 6 +
> Makefile | 9 +-
> docs/manual/common-usage.txt | 2 +
> docs/manual/test-infrastructure.txt | 239 +++++++++++++++++++++
> package/Config.in | 1 +
> package/Config.in.host | 1 +
> package/Makefile.in | 1 +
> package/pkg-generic.mk | 20 +-
> package/pkg-test.mk | 94 ++++++++
> package/python-flask/python-flask.mk | 5 +
> .../target_test_material/target_test.robot | 25 +++
> package/python-robotframework/Config.in | 9 +
> package/python-robotframework/Config.in.host | 8 +
> .../python-robotframework.hash | 4 +
> .../python-robotframework/python-robotframework.mk | 16 ++
> package/python/python.mk | 17 ++
> package/python/python.robot | 32 +++
> tests/Config.in | 8 +
> tests/host/resource.robot | 7 +
> tests/target/resource.robot | 1 +
> tests/tests.mk | 20 ++
> 21 files changed, 522 insertions(+), 3 deletions(-)
> create mode 100644 docs/manual/test-infrastructure.txt
> create mode 100644 package/pkg-test.mk
> create mode 100644 package/python-flask/target_test_material/target_test.robot
> create mode 100644 package/python-robotframework/Config.in
> create mode 100644 package/python-robotframework/Config.in.host
> create mode 100644 package/python-robotframework/python-robotframework.hash
> create mode 100644 package/python-robotframework/python-robotframework.mk
> create mode 100644 package/python/python.robot
> create mode 100644 tests/Config.in
> create mode 100644 tests/host/resource.robot
> create mode 100644 tests/target/resource.robot
> create mode 100644 tests/tests.mk
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 12+ messages in thread* [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages
2015-09-06 22:46 ` Arnout Vandecappelle
@ 2015-09-07 8:00 ` Denis Thulin
2015-10-04 18:35 ` Arnout Vandecappelle
0 siblings, 1 reply; 12+ messages in thread
From: Denis Thulin @ 2015-09-07 8:00 UTC (permalink / raw)
To: buildroot
Hi Arnout,
----- Le 7 Sep 15, ? 0:46, Arnout Vandecappelle arnout at mind.be a ?crit :
> Hi Denis,
>
> On 31-08-15 11:59, Denis THULIN wrote:
>> This patch series adds a test generation infrastructure for packages to
>> Buildroot.
>> The generated tests are robotframework tests.
>>
>> While it is always necessary to test that packages are installed
>> correctly and work in an intended way, Buildroot does not currently
>> provide any tests of package nor any way to automate the process of
>> creating tests depending on your configuration or even share tests you
>> have written for packages. You have to write tests manually almost
>> everytime you start a new project with buildroot. This can cost a lot
>> of time.
>
> Just to be clear: the intention of this series is to make it possible to
> include in the buildroot package a way to specify some tests to run on the host
> and on the target, where the purpose of these tests is to validate that the
> package has been built correctly. So it is not exactly testing the buildroot
> infrastructure (like what Thomas is trying to do), and it's not really running
> the tests included with the package itself (these are not built and installed),
> but something in-between.
Yes.
>
>
> However, as far as I can see, this patch series only solves a small part of the
> problem. It basically creates and installs a bunch of .robot files, and that's
> it, right? You still have to make the robot framework run on these files. And
> you still have to express in the robot framework or in jenkins the dependencies
> to make sure the tests are first built and then executed. Correct?
>
Yes you do.
>
> In addition, it seems to me that the munging done by the TEST_BUILD macro has
> little to offer above simply copying the .robot file from the package directory
> to the test directory. But possibly I don't sufficiently understand what's going
> on.
It mostly copies the .robot files.
Some tests or robot variables can be written in the .mk file so that they depend
on the Buildroot configuration.
>
>
> And finally, buildroot tries not to enforce any particular policy or tool on
> the user. So, much as I like robot, requiring the tests to use this specific
> framework is definitely a no-go (at least, if it's meant to be used by end
> users). Especially since it means that a rootfs with python3 can't be tested...
>
Ok.
>
> I'm not entirely sure what the way to go is. It definitely is nice to have some
> kind of test description as part of the packages - I like that more than the way
> Thomas did it in buildroot-runtime-test, where the buildroot tree is basically
> replicated externally for the tests. But the exact location of the test files is
> just a minor detail I guess.
Yes it is.
>
>
> Bottom line: I'm not in favour of this series either. Of course, the first
> patch adding python-robotframework is still relevant independently.
Ok, thanks for the answer :)
Regards,
Denis
>
>
> Regards,
> Arnout
>
>>
>> This patch series is intented as a solution for that problem.
>>
>> Why Robotframework Tests:
>> Robotframework is a widely used test automation framework that allow
>> writing tests in various languages (Python, Java and even C), provides
>> useful test librairies and allow writing tests in various syntaxes.
>> Robotframework tests are meant to be easy to read by people whom did
>> not write the tests, and that can definitly be useful.
>>
>> List of patches from the series :
>> * Patch 1/6 adds a test generation step before install target step.
>> * Patch 2/6 adds useful keywords for testing through Telnet and Qemu
>> * Patch 3/6 adds menuconfig options to ease the configuration required
>> for testing through telnet and Qemu
>> * Patch 4/6 adds documentation for patches 1,2 and 3
>> * Patch 5/6 adds exemple tests for package python-flask
>> * patch 6/6 adds exemple tests for package python (inspired by tests
>> from https://github.com/tpetazzoni/buildroot-runtime-test)
>>
>> Denis THULIN (6):
>> python-robotframework: New package
>> Adds package test infrastructure
>> tests: create variable files through kconfig
>> tests: Adds user manual entry
>> flask: Adds robotframework tests
>> python: Adds tests
>>
>> Config.in | 6 +
>> Makefile | 9 +-
>> docs/manual/common-usage.txt | 2 +
>> docs/manual/test-infrastructure.txt | 239 +++++++++++++++++++++
>> package/Config.in | 1 +
>> package/Config.in.host | 1 +
>> package/Makefile.in | 1 +
>> package/pkg-generic.mk | 20 +-
>> package/pkg-test.mk | 94 ++++++++
>> package/python-flask/python-flask.mk | 5 +
>> .../target_test_material/target_test.robot | 25 +++
>> package/python-robotframework/Config.in | 9 +
>> package/python-robotframework/Config.in.host | 8 +
>> .../python-robotframework.hash | 4 +
>> .../python-robotframework/python-robotframework.mk | 16 ++
>> package/python/python.mk | 17 ++
>> package/python/python.robot | 32 +++
>> tests/Config.in | 8 +
>> tests/host/resource.robot | 7 +
>> tests/target/resource.robot | 1 +
>> tests/tests.mk | 20 ++
>> 21 files changed, 522 insertions(+), 3 deletions(-)
>> create mode 100644 docs/manual/test-infrastructure.txt
>> create mode 100644 package/pkg-test.mk
>> create mode 100644 package/python-flask/target_test_material/target_test.robot
>> create mode 100644 package/python-robotframework/Config.in
>> create mode 100644 package/python-robotframework/Config.in.host
>> create mode 100644 package/python-robotframework/python-robotframework.hash
>> create mode 100644 package/python-robotframework/python-robotframework.mk
>> create mode 100644 package/python/python.robot
>> create mode 100644 tests/Config.in
>> create mode 100644 tests/host/resource.robot
>> create mode 100644 tests/target/resource.robot
>> create mode 100644 tests/tests.mk
>>
>
>
> --
> Arnout Vandecappelle arnout at mind be
> Senior Embedded Software Architect +32-16-286500
> Essensium/Mind http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 12+ messages in thread
* [Buildroot] [PATCH 0/6] [RFC] Adds test infrastructure for packages
2015-09-07 8:00 ` Denis Thulin
@ 2015-10-04 18:35 ` Arnout Vandecappelle
0 siblings, 0 replies; 12+ messages in thread
From: Arnout Vandecappelle @ 2015-10-04 18:35 UTC (permalink / raw)
To: buildroot
On 07-09-15 09:00, Denis Thulin wrote:
> Hi Arnout,
>
> ----- Le 7 Sep 15, ? 0:46, Arnout Vandecappelle arnout at mind.be a ?crit :
>
>> Hi Denis,
>>
>> On 31-08-15 11:59, Denis THULIN wrote:
>>> This patch series adds a test generation infrastructure for packages to
>>> Buildroot.
>>> The generated tests are robotframework tests.
[snip]
>> Bottom line: I'm not in favour of this series either. Of course, the first
>> patch adding python-robotframework is still relevant independently.
>
> Ok, thanks for the answer :)
Hi Denis,
We discussed this again at the BR Developer meeting in Dublin, and concluded
that we're going to proceed with the simpler and less-invasive unit-test-based
approach. It is in fact already running on Jenkins at [1]. Therefore, I've
marked most of the series as Rejected in patchwork.
Regarding python-robotframework, as mentioned before, it is still relevant.
However, neither of the two versions you submitted are suitable at the moment:
- the version of [2] is still using 2.9rc1;
- the version of [3] add a host- version which we don't need.
Note that the host-version was not even needed in this series: for something
like this, we expect it to be installed on the system rather than building it.
So these patches are both marked as Changes Requested.
Regards,
Arnout
[1] http://jenkins.free-electrons.com/job/buildroot-runtime-tests/
[2] http://patchwork.ozlabs.org/patch/501203/
[3] http://patchwork.ozlabs.org/patch/512420/
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 12+ messages in thread