[Buildroot] [PATCH v2 0/6] Python/IPython unit test changes

[Buildroot] [PATCH v2 0/6] Python/IPython unit test changes

[Andrey Smirnov]

Hi everyone,

This is a second version of the Python/IPython related patchset.

Changes since [v1]:

    - Move timeout from login() to run()

    - Split patch 2/3 in original submission into multiple discreet
      changes

Any feedback is much appreciated!

Thanks,
Andrey Smirnov

[v1]: http://lists.busybox.net/pipermail/buildroot/2017-July/196735.html

Andrey Smirnov (6):
  testing/infra/emulator: allow to specify pexpect timeout
  testing/tests/package/test_python: refactor TestPythonBase
  testing/tests/package/test_python: add TestPython3
  testing/tests/package/test_python: do not hardcode interpreter name
  testing/tests/package/test_python: allow to change timeout
  testing/tests/package: add basic unit test for IPython

 support/testing/infra/emulator.py             |  6 ++--
 support/testing/tests/package/test_ipython.py | 38 ++++++++++++++++++++
 support/testing/tests/package/test_python.py  | 52 +++++++++++++++++++++------
 3 files changed, 82 insertions(+), 14 deletions(-)
 create mode 100644 support/testing/tests/package/test_ipython.py

-- 
2.9.4

[Buildroot] [PATCH v2 1/6] testing/infra/emulator: allow to specify pexpect timeout

[Andrey Smirnov]

Some commands take more than 5 seconds to complete under QEMU, so add
provisions to allow individual unit-test to specify different duration
to avoid false negative test failures.

Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
---
 support/testing/infra/emulator.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
index a39d59b..9b079cb 100644
--- a/support/testing/infra/emulator.py
+++ b/support/testing/infra/emulator.py
@@ -89,11 +89,11 @@ class Emulator(object):
             raise SystemError("Cannot login")
         self.run("dmesg -n 1")
 
-    # Run the given 'cmd' on the target
+    # Run the given 'cmd' with a 'timeout' on the target
     # return a tuple (output, exit_code)
-    def run(self, cmd):
+    def run(self, cmd, timeout=-1):
         self.qemu.sendline(cmd)
-        self.qemu.expect("# ")
+        self.qemu.expect("# ", timeout=timeout)
         # Remove double carriage return from qemu stdout so str.splitlines()
         # works as expected.
         output = self.qemu.before.replace("\r\r", "\r").splitlines()[1:]
-- 
2.9.4

[Buildroot] [PATCH v2 1/6] testing/infra/emulator: allow to specify pexpect timeout

[Ricardo Martincoski]

Hello,

On Tue, Jul 11, 2017 at 11:40 PM, Andrey Smirnov wrote:

> Some commands take more than 5 seconds to complete under QEMU, so add
> provisions to allow individual unit-test to specify different duration
> to avoid false negative test failures.
> 
> Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>

Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[I executed all tests in the current master with this patch]
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>

[snip]
> -    def run(self, cmd):
> +    def run(self, cmd, timeout=-1):

Indeed.
From the docs: "When the keyword argument timeout is -1 (default), then TIMEOUT
will raise after the default value specified by the class timeout attribute."

Regards,
Ricardo

[Buildroot] [PATCH v2 1/6] testing/infra/emulator: allow to specify pexpect timeout

[Thomas Petazzoni]

Hello,

On Tue, 11 Jul 2017 19:40:04 -0700, Andrey Smirnov wrote:
> Some commands take more than 5 seconds to complete under QEMU, so add
> provisions to allow individual unit-test to specify different duration
> to avoid false negative test failures.
> 
> Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
> ---
>  support/testing/infra/emulator.py | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v2 2/6] testing/tests/package/test_python: refactor TestPythonBase

[Andrey Smirnov]

Convert TestPythonBase to a true base class that only provides code
implementing various tests without definig tests themselves in a
"discoverable" form.

To retain correct testing functionality, add TestPython2 derived class
that uses code from TestPythonBase to define actual runnable test.

Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
---
 support/testing/tests/package/test_python.py | 23 +++++++++++++++++++----
 1 file changed, 19 insertions(+), 4 deletions(-)

diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
index 5532fb5..250827e 100644
--- a/support/testing/tests/package/test_python.py
+++ b/support/testing/tests/package/test_python.py
@@ -5,31 +5,46 @@ import infra.basetest
 class TestPythonBase(infra.basetest.BRTest):
     config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
 """
-BR2_PACKAGE_PYTHON=y
 BR2_TARGET_ROOTFS_CPIO=y
 # BR2_TARGET_ROOTFS_TAR is not set
 """
-
-    def test_run(self):
+    def login(self):
         cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio")
         self.emulator.boot(arch="armv5",
                            kernel="builtin",
                            options=["-initrd", cpio_file])
         self.emulator.login()
-        cmd = "python --version 2>&1 | grep '^Python 2'"
+
+    def version_test(self, version):
+        cmd = "python --version 2>&1 | grep '^{}'".format(version)
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 0)
 
+    def math_floor_test(self):
         cmd = "python -c 'import math; math.floor(12.3)'"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 0)
 
+    def libc_time_test(self):
         cmd = "python -c 'import ctypes;"
         cmd += "libc = ctypes.cdll.LoadLibrary(\"libc.so.1\");"
         cmd += "print libc.time(None)'"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 0)
 
+    def zlib_test(self):
         cmd = "python -c 'import zlib'"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 1)
+
+class TestPython2(TestPythonBase):
+    config = TestPythonBase.config + \
+"""
+BR2_PACKAGE_PYTHON=y
+"""
+    def test_run(self):
+        self.login()
+        self.version_test("Python 2")
+        self.math_floor_test()
+        self.libc_time_test()
+        self.zlib_test()
-- 
2.9.4

[Buildroot] [PATCH v2 2/6] testing/tests/package/test_python: refactor TestPythonBase

[Ricardo Martincoski]

Hello,

On Tue, Jul 11, 2017 at 11:40 PM, Andrey Smirnov wrote:

> Convert TestPythonBase to a true base class that only provides code
> implementing various tests without definig tests themselves in a

s/definig/defining/

> "discoverable" form.
> 
> To retain correct testing functionality, add TestPython2 derived class
> that uses code from TestPythonBase to define actual runnable test.
> 
> Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>

with the typo fixed
Reviewed-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[I executed the test and checked the -run.log is equivalent to the old one]
Tested-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>

Below a note to be read when applying...

[snip]
> +class TestPython2(TestPythonBase):

The runnable test got renamed, so the entry needs to be updated in
.gitlab-ci.yml
-tests.package.test_python.TestPythonBase: *runtime_test
+tests.package.test_python.TestPython2: *runtime_test

Regards,
Ricardo

[Buildroot] [PATCH v2 2/6] testing/tests/package/test_python: refactor TestPythonBase

[Thomas Petazzoni]

Hello,

On Tue, 11 Jul 2017 19:40:05 -0700, Andrey Smirnov wrote:
> Convert TestPythonBase to a true base class that only provides code
> implementing various tests without definig tests themselves in a
> "discoverable" form.
> 
> To retain correct testing functionality, add TestPython2 derived class
> that uses code from TestPythonBase to define actual runnable test.
> 
> Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
> ---
>  support/testing/tests/package/test_python.py | 23 +++++++++++++++++++----
>  1 file changed, 19 insertions(+), 4 deletions(-)

Applied to master, after fixing the two issues pointed by Ricardo.

BTW Ricardo: you seem to be following the changes on the test
infrastructure, and contributing to it. Would you be willing to be
listed in DEVELOPERS for this support/testing/ folder ? Your review and
testing of the testing infrastructure patches are definitely very
useful.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v2 3/6] testing/tests/package/test_python: add TestPython3

[Andrey Smirnov]

Add Python3 version of TestPython2 to make sure both versions of
Python get unit-tested.

Modify the code of libc_time_test() to support that change (convert
the code to use Python3-style "print").

Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
---
 support/testing/tests/package/test_python.py | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
index 250827e..218235b 100644
--- a/support/testing/tests/package/test_python.py
+++ b/support/testing/tests/package/test_python.py
@@ -26,9 +26,10 @@ BR2_TARGET_ROOTFS_CPIO=y
         self.assertEqual(exit_code, 0)
 
     def libc_time_test(self):
-        cmd = "python -c 'import ctypes;"
+        cmd = "python -c 'from __future__ import print_function;"
+        cmd += "import ctypes;"
         cmd += "libc = ctypes.cdll.LoadLibrary(\"libc.so.1\");"
-        cmd += "print libc.time(None)'"
+        cmd += "print(libc.time(None))'"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 0)
 
@@ -48,3 +49,15 @@ BR2_PACKAGE_PYTHON=y
         self.math_floor_test()
         self.libc_time_test()
         self.zlib_test()
+
+class TestPython3(TestPythonBase):
+    config = TestPythonBase.config + \
+"""
+BR2_PACKAGE_PYTHON3=y
+"""
+    def test_run(self):
+        self.login()
+        self.version_test("Python 3")
+        self.math_floor_test()
+        self.libc_time_test()
+        self.zlib_test()
-- 
2.9.4

[Buildroot] [PATCH v2 4/6] testing/tests/package/test_python: do not hardcode interpreter name

[Andrey Smirnov]

In order to be able to leverage the same test code for testing
different python interpreters (or wrappers around CPython) allow
classes derivate of TestPythonBase to override the name of the
executable used to run tests.

Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
---
 support/testing/tests/package/test_python.py | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
index 218235b..a3ec31b 100644
--- a/support/testing/tests/package/test_python.py
+++ b/support/testing/tests/package/test_python.py
@@ -8,6 +8,8 @@ class TestPythonBase(infra.basetest.BRTest):
 BR2_TARGET_ROOTFS_CPIO=y
 # BR2_TARGET_ROOTFS_TAR is not set
 """
+    interpreter = "python"
+
     def login(self):
         cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio")
         self.emulator.boot(arch="armv5",
@@ -16,17 +18,17 @@ BR2_TARGET_ROOTFS_CPIO=y
         self.emulator.login()
 
     def version_test(self, version):
-        cmd = "python --version 2>&1 | grep '^{}'".format(version)
+        cmd = self.interpreter + " --version 2>&1 | grep '^{}'".format(version)
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 0)
 
     def math_floor_test(self):
-        cmd = "python -c 'import math; math.floor(12.3)'"
+        cmd = self.interpreter + " -c 'import math; math.floor(12.3)'"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 0)
 
     def libc_time_test(self):
-        cmd = "python -c 'from __future__ import print_function;"
+        cmd = self.interpreter + " -c 'from __future__ import print_function;"
         cmd += "import ctypes;"
         cmd += "libc = ctypes.cdll.LoadLibrary(\"libc.so.1\");"
         cmd += "print(libc.time(None))'"
@@ -34,7 +36,7 @@ BR2_TARGET_ROOTFS_CPIO=y
         self.assertEqual(exit_code, 0)
 
     def zlib_test(self):
-        cmd = "python -c 'import zlib'"
+        cmd = self.interpreter + " -c 'import zlib'"
         _, exit_code = self.emulator.run(cmd)
         self.assertEqual(exit_code, 1)
 
-- 
2.9.4

[Buildroot] [PATCH v2 5/6] testing/tests/package/test_python: allow to change timeout

[Andrey Smirnov]

Depending on Python implementation used for testing, time it takes to
perform a given test can vary pretty significantly. To accout for that
allow individual test functions to specify different timeout value.

Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
---
 support/testing/tests/package/test_python.py | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/support/testing/tests/package/test_python.py b/support/testing/tests/package/test_python.py
index a3ec31b..bddf09d 100644
--- a/support/testing/tests/package/test_python.py
+++ b/support/testing/tests/package/test_python.py
@@ -17,27 +17,27 @@ BR2_TARGET_ROOTFS_CPIO=y
                            options=["-initrd", cpio_file])
         self.emulator.login()
 
-    def version_test(self, version):
+    def version_test(self, version, timeout=-1):
         cmd = self.interpreter + " --version 2>&1 | grep '^{}'".format(version)
-        _, exit_code = self.emulator.run(cmd)
+        _, exit_code = self.emulator.run(cmd, timeout)
         self.assertEqual(exit_code, 0)
 
-    def math_floor_test(self):
+    def math_floor_test(self, timeout=-1):
         cmd = self.interpreter + " -c 'import math; math.floor(12.3)'"
-        _, exit_code = self.emulator.run(cmd)
+        _, exit_code = self.emulator.run(cmd, timeout)
         self.assertEqual(exit_code, 0)
 
-    def libc_time_test(self):
+    def libc_time_test(self, timeout=-1):
         cmd = self.interpreter + " -c 'from __future__ import print_function;"
         cmd += "import ctypes;"
         cmd += "libc = ctypes.cdll.LoadLibrary(\"libc.so.1\");"
         cmd += "print(libc.time(None))'"
-        _, exit_code = self.emulator.run(cmd)
+        _, exit_code = self.emulator.run(cmd, timeout)
         self.assertEqual(exit_code, 0)
 
-    def zlib_test(self):
+    def zlib_test(self, timeout=-1):
         cmd = self.interpreter + " -c 'import zlib'"
-        _, exit_code = self.emulator.run(cmd)
+        _, exit_code = self.emulator.run(cmd, timeout)
         self.assertEqual(exit_code, 1)
 
 class TestPython2(TestPythonBase):
-- 
2.9.4

[Buildroot] [PATCH v2 6/6] testing/tests/package: add basic unit test for IPython

[Andrey Smirnov]

Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
---
 support/testing/tests/package/test_ipython.py | 38 +++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 support/testing/tests/package/test_ipython.py

diff --git a/support/testing/tests/package/test_ipython.py b/support/testing/tests/package/test_ipython.py
new file mode 100644
index 0000000..dd8bf50
--- /dev/null
+++ b/support/testing/tests/package/test_ipython.py
@@ -0,0 +1,38 @@
+import os
+
+from tests.package.test_python import TestPythonBase
+#
+# The following pythong tests are not being used here:
+#
+# - version_test: IPython does not support --version option
+#
+# - zlib_test: IPython does not return a non-zero code the way CPython
+#              does, so this test ends up being a false-negative
+#
+class TestIPythonPy2(TestPythonBase):
+    config = TestPythonBase.config + \
+"""
+BR2_PACKAGE_PYTHON=y
+BR2_PACKAGE_PYTHON_IPYTHON=y
+"""
+    interpreter = "ipython"
+
+    def test_run(self):
+        self.login()
+        self.math_floor_test(40)
+        self.libc_time_test(40)
+
+class TestIPythonPy3(TestPythonBase):
+    config = TestPythonBase.config + \
+"""
+BR2_PACKAGE_PYTHON3=y
+BR2_PACKAGE_PYTHON_IPYTHON=y
+"""
+    interpreter = "ipython"
+
+    def test_run(self):
+        self.login()
+        self.math_floor_test(40)
+        self.libc_time_test(40)
+
+
-- 
2.9.4

[Buildroot] [PATCH v2 0/6] Python/IPython unit test changes

[Thomas Petazzoni]

Hello,

On Tue, 11 Jul 2017 19:40:03 -0700, Andrey Smirnov wrote:

> Andrey Smirnov (6):
>   testing/infra/emulator: allow to specify pexpect timeout
>   testing/tests/package/test_python: refactor TestPythonBase
>   testing/tests/package/test_python: add TestPython3
>   testing/tests/package/test_python: do not hardcode interpreter name
>   testing/tests/package/test_python: allow to change timeout
>   testing/tests/package: add basic unit test for IPython

I've applied the remaining patches 3 to 6. One thing was missing:
updating the .gitlab-ci.yml file, so I've done that.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com