[Buildroot] [PATCH 1/7] support/testing: test_syslinux: fix flake8 errors

[Buildroot] [PATCH 1/7] support/testing: test_syslinux: fix flake8 errors

[Romain Naour]

Reported by gitlab:
https://gitlab.com/kubu93/buildroot/-/jobs/499374911

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/testing/tests/boot/test_syslinux.py | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/support/testing/tests/boot/test_syslinux.py b/support/testing/tests/boot/test_syslinux.py
index 6b1a8bc7d7..4105e5edac 100644
--- a/support/testing/tests/boot/test_syslinux.py
+++ b/support/testing/tests/boot/test_syslinux.py
@@ -1,13 +1,14 @@
 import infra.basetest
 
+
 class TestSysLinuxBase(infra.basetest.BRTest):
     x86_toolchain_config = \
         """
-	BR2_x86_i686=y
-	BR2_TOOLCHAIN_EXTERNAL=y
-	BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
-	BR2_TOOLCHAIN_EXTERNAL_URL="http://toolchains.bootlin.com/downloads/releases/toolchains/x86-i686/tarballs/x86-i686--glibc--bleeding-edge-2018.11-1.tar.bz2"
-	BR2_TOOLCHAIN_EXTERNAL_GCC_8=y
+        BR2_x86_i686=y
+        BR2_TOOLCHAIN_EXTERNAL=y
+        BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y
+        BR2_TOOLCHAIN_EXTERNAL_URL="http://toolchains.bootlin.com/downloads/releases/toolchains/x86-i686/tarballs/x86-i686--glibc--bleeding-edge-2018.11-1.tar.bz2"
+        BR2_TOOLCHAIN_EXTERNAL_GCC_8=y
         BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_14=y
         BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
         BR2_TOOLCHAIN_EXTERNAL_CXX=y
@@ -46,6 +47,7 @@ class TestSysLinuxBase(infra.basetest.BRTest):
         BR2_TARGET_SYSLINUX_EFI=y
         """
 
+
 class TestSysLinuxX86LegacyBios(TestSysLinuxBase):
     config = \
         TestSysLinuxBase.x86_toolchain_config + \
@@ -55,6 +57,7 @@ class TestSysLinuxX86LegacyBios(TestSysLinuxBase):
     def test_run(self):
         pass
 
+
 class TestSysLinuxX86EFI(TestSysLinuxBase):
     config = \
         TestSysLinuxBase.x86_toolchain_config + \
@@ -64,6 +67,7 @@ class TestSysLinuxX86EFI(TestSysLinuxBase):
     def test_run(self):
         pass
 
+
 class TestSysLinuxX86_64LegacyBios(TestSysLinuxBase):
     config = \
         TestSysLinuxBase.x86_64_toolchain_config + \
@@ -73,6 +77,7 @@ class TestSysLinuxX86_64LegacyBios(TestSysLinuxBase):
     def test_run(self):
         pass
 
+
 class TestSysLinuxX86_64EFI(TestSysLinuxBase):
     config = \
         TestSysLinuxBase.x86_64_toolchain_config + \
-- 
2.25.1

[Buildroot] [PATCH 2/7] support/testing: test_syslinux: remove BR2_TOOLCHAIN_EXTERNAL_LOCALE

[Romain Naour]

The BR2_TOOLCHAIN_EXTERNAL_LOCALE is only defined for uClibc based
custom external toolchains.

The test_syslinux is using a glibc toolchain, so the
"# BR2_TOOLCHAIN_EXTERNAL_LOCALE is not set" line disapear from
the generated dot config (.config) leading to an error due to
a new check in the testsuite infra.

There is the same problem with:
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG=y
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS=y
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_NPTL=y

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/testing/tests/boot/test_syslinux.py | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/support/testing/tests/boot/test_syslinux.py b/support/testing/tests/boot/test_syslinux.py
index 4105e5edac..f25cddfb6c 100644
--- a/support/testing/tests/boot/test_syslinux.py
+++ b/support/testing/tests/boot/test_syslinux.py
@@ -24,12 +24,8 @@ class TestSysLinuxBase(infra.basetest.BRTest):
         BR2_TOOLCHAIN_EXTERNAL_URL="http://toolchains.bootlin.com/downloads/releases/toolchains/x86-64-core-i7/tarballs/x86-64-core-i7--glibc--stable-2018.11-1.tar.bz2"
         BR2_TOOLCHAIN_EXTERNAL_GCC_7=y
         BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_1=y
-        # BR2_TOOLCHAIN_EXTERNAL_LOCALE is not set
         BR2_TOOLCHAIN_EXTERNAL_CXX=y
         BR2_TOOLCHAIN_EXTERNAL_HAS_SSP=y
-        BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG=y
-        BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS=y
-        BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_NPTL=y
         BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
         """
 
-- 
2.25.1

[Buildroot] [PATCH 3/7] support/testing: test_glxinfo: remove BR2_TOOLCHAIN_EXTERNAL_LOCALE

[Romain Naour]

The BR2_TOOLCHAIN_EXTERNAL_LOCALE is only defined for uClibc based
custom external toolchains.

The test_glxinfo is using a glibc toolchain, so the
"# BR2_TOOLCHAIN_EXTERNAL_LOCALE is not set" line disapear from
the generated dot config (.config) leading to an error due to
a new check in the testsuite infra.

There is the same problem with:
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG=y
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS=y
BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_NPTL=y

Signed-off-by: Romain Naour <romain.naour@gmail.com>
---
 support/testing/tests/package/test_glxinfo.py | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/support/testing/tests/package/test_glxinfo.py b/support/testing/tests/package/test_glxinfo.py
index a9df6ddf6f..f37236d161 100644
--- a/support/testing/tests/package/test_glxinfo.py
+++ b/support/testing/tests/package/test_glxinfo.py
@@ -15,12 +15,8 @@ class TestGlxinfo(infra.basetest.BRTest):
         BR2_TOOLCHAIN_EXTERNAL_URL="http://toolchains.bootlin.com/downloads/releases/toolchains/x86-core2/tarballs/x86-core2--glibc--bleeding-edge-2018.11-1.tar.bz2"
         BR2_TOOLCHAIN_EXTERNAL_GCC_8=y
         BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_14=y
-        # BR2_TOOLCHAIN_EXTERNAL_LOCALE is not set
         BR2_TOOLCHAIN_EXTERNAL_CXX=y
         BR2_TOOLCHAIN_EXTERNAL_HAS_SSP=y
-        BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG=y
-        BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS=y
-        BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_NPTL=y
         BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
         BR2_LINUX_KERNEL=y
         BR2_LINUX_KERNEL_CUSTOM_VERSION=y
-- 
2.25.1

[Buildroot] [PATCH 4/7] support/testing: test_tmux: remove BR2_GENERATE_LOCALE

[Romain Naour]

The BR2_GENERATE_LOCALE can only be used by uClibc based toolchains with the
internal toolchain backend [1].

The test_tmux is using a external uClibc toolchain, so the
"BR2_GENERATE_LOCALE="en_US.UTF-8"" line disapear from
the generated dot config (.config) leading to an error due to
a new check in the testsuite infra.

[1] https://git.buildroot.net/buildroot/commit/?id=bd0ffe2206fbd32baf7f4a1dc5fde81cfad70462

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
---
 support/testing/tests/package/test_tmux.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/support/testing/tests/package/test_tmux.py b/support/testing/tests/package/test_tmux.py
index 7eb23cac97..b31854cab3 100644
--- a/support/testing/tests/package/test_tmux.py
+++ b/support/testing/tests/package/test_tmux.py
@@ -6,7 +6,6 @@ import infra.basetest
 class TestTmux(infra.basetest.BRTest):
     config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
         """
-        BR2_GENERATE_LOCALE="en_US.UTF-8"
         BR2_PACKAGE_TMUX=y
         BR2_TARGET_ROOTFS_CPIO=y
         # BR2_TARGET_ROOTFS_TAR is not set
-- 
2.25.1

[Buildroot] [PATCH 5/7] support/testing: test_lxc: use ARM arm external toolchain

[Romain Naour]

Until now, the lxc test was using the ARM CodeSourcery 2014.05 armv5 toolchain.
But the recent systemd version bump to 245 added a toolchain dependency
on systemd package due to build issues with gcc < 5.0.

Before [1] the lxc test was failing to build with the ARM CodeSourcery 2014.05
toolchain. After [1], the test is faling at runtime since the
"BR2_INIT_SYSTEMD=y" symbol disapear from the dot config (.config) due to
the new toolchain dependency.

Fix this by using the same toolchain as for the systemd tests [2]

[1] 2196ee25ff29c87001a42a382655d5b52d5de3c4
[2] b3d979c0d1b10de501576644e7dcdba708889b0c

Signed-off-by: Romain Naour <romain.naour@gmail.com>
---
 support/testing/tests/package/test_lxc.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/support/testing/tests/package/test_lxc.py b/support/testing/tests/package/test_lxc.py
index 346247da08..b32b7a20c4 100644
--- a/support/testing/tests/package/test_lxc.py
+++ b/support/testing/tests/package/test_lxc.py
@@ -7,6 +7,8 @@ class TestLxc(infra.basetest.BRTest):
     config = \
             """
             BR2_arm=y
+            BR2_cortex_a9=y
+            BR2_ARM_ENABLE_VFP=y
             BR2_TOOLCHAIN_EXTERNAL=y
             BR2_LINUX_KERNEL=y
             BR2_LINUX_KERNEL_CUSTOM_VERSION=y
-- 
2.25.1

[Buildroot] [PATCH 6/7] support/testing: don't use TestPythonPackageBase.config and linaro toolchain

[Romain Naour]

TestPythonPackageBase.config provide already the basic uClibc toolchain.
So by adding the symbols for the linaro toolchain some warning are printed
at while loading the configuration:

.config:16:warning: override: reassigning to symbol BR2_TOOLCHAIN_EXTERNAL
.config:16:warning: override: BR2_TOOLCHAIN_EXTERNAL changes choice state
.config:17:warning: override: reassigning to symbol BR2_TARGET_ROOTFS_CPIO
.config:21:warning: override: BR2_TOOLCHAIN_EXTERNAL_LINARO_ARM changes choice state

So, some symbol disapear from the generated dot config (.config) leading
to an error due to a new check in the testsuite infra.

Since this test should use the Linaro toolchain, remove
TestPythonPackageBase.config add BR2_arm=y and disable the rootfs tar option.

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Adam Duskett <Aduskett@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
---
 support/testing/tests/package/test_gst1_python.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/support/testing/tests/package/test_gst1_python.py b/support/testing/tests/package/test_gst1_python.py
index c6b337dd04..10d7907f8d 100644
--- a/support/testing/tests/package/test_gst1_python.py
+++ b/support/testing/tests/package/test_gst1_python.py
@@ -4,10 +4,12 @@ from tests.package.test_python import TestPythonPackageBase
 
 class TestGst1Python(TestPythonPackageBase):
     __test__ = True
-    config = TestPythonPackageBase.config + \
+    config = \
         """
         BR2_TOOLCHAIN_EXTERNAL=y
         BR2_TARGET_ROOTFS_CPIO=y
+        # BR2_TARGET_ROOTFS_TAR is not set
+        BR2_arm=y
         BR2_cortex_a9=y
         BR2_ARM_ENABLE_VFP=y
         BR2_ARM_EABIHF=y
-- 
2.25.1

[Buildroot] [PATCH 7/7] support/testing: check if the defconfig provided for testing is valid

[Romain Naour]

Currently, the build continue even if some symbols disapear from
the generated dot config file (.config).

This patch add a new check in order to stop the test if one
of the provided symbol is missing. This must be treated as error.

For example, if a symbol disapear due to new dependency constraints.

Inspired by is_toolchain_usable() function from genrandconfig:
https://git.busybox.net/buildroot/tree/utils/genrandconfig?h=2020.02#n164

Signed-off-by: Romain Naour <romain.naour@gmail.com>
---
 support/testing/infra/builder.py | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/support/testing/infra/builder.py b/support/testing/infra/builder.py
index 88f01d15c0..922a707220 100644
--- a/support/testing/infra/builder.py
+++ b/support/testing/infra/builder.py
@@ -12,6 +12,23 @@ class Builder(object):
         self.builddir = builddir
         self.logfile = infra.open_log_file(builddir, "build", logtofile)
 
+    def is_defconfig_valid(self, configfile, defconfig):
+        """Check if the .config is contains all lines present in the defconfig."""
+        with open(configfile) as configf:
+            configlines = configf.readlines()
+
+        defconfiglines = defconfig.split("\n")
+
+        # Check that all the defconfig lines are still present
+        for defconfigline in defconfiglines:
+            if defconfigline + "\n" not in configlines:
+                self.logfile.write("WARN: defconfig can't be used\n")
+                self.logfile.write("      Missing: %s\n" % defconfigline.strip())
+                self.logfile.flush()
+                return False
+
+        return True
+
     def configure(self, make_extra_opts=[], make_extra_env={}):
         """Configure the build.
 
@@ -47,6 +64,9 @@ class Builder(object):
         if ret != 0:
             raise SystemError("Cannot olddefconfig")
 
+        if not self.is_defconfig_valid(config_file, self.config):
+            raise SystemError("The defconfig is not valid")
+
     def build(self, make_extra_opts=[], make_extra_env={}):
         """Perform the build.
 
-- 
2.25.1

[Buildroot] [PATCH 1/7] support/testing: test_syslinux: fix flake8 errors

[Thomas Petazzoni]

On Mon,  6 Apr 2020 01:04:08 +0200
Romain Naour <romain.naour@gmail.com> wrote:

> Reported by gitlab:
> https://gitlab.com/kubu93/buildroot/-/jobs/499374911
> 
> Signed-off-by: Romain Naour <romain.naour@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  support/testing/tests/boot/test_syslinux.py | 15 ++++++++++-----
>  1 file changed, 10 insertions(+), 5 deletions(-)

I've applied the entire series, with some changes on PATCH 6/7, to
which I'll reply specifically.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 6/7] support/testing: don't use TestPythonPackageBase.config and linaro toolchain

[Thomas Petazzoni]

On Mon,  6 Apr 2020 01:04:13 +0200
Romain Naour <romain.naour@gmail.com> wrote:

>  class TestGst1Python(TestPythonPackageBase):
>      __test__ = True
> -    config = TestPythonPackageBase.config + \
> +    config = \
>          """
>          BR2_TOOLCHAIN_EXTERNAL=y
>          BR2_TARGET_ROOTFS_CPIO=y
> +        # BR2_TARGET_ROOTFS_TAR is not set
> +        BR2_arm=y
>          BR2_cortex_a9=y
>          BR2_ARM_ENABLE_VFP=y
>          BR2_ARM_EABIHF=y

I like when options are in the same order as they would appear in a
defconfig, so I took this opportunity to re-order them this way.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com