[Buildroot] [PATCH 0/5] test infra using Python 3

[Buildroot] [PATCH 0/5] test infra using Python 3

[Ricardo Martincoski]

Hello,

This small series changes the runtime test infra to use Python 3 only.
The reasoning to not keep Python 2 compatibility is part of the commit log for
the last patch.

Patch 1 is not really related to this migration but it is part of this series
only to avoid unnecessary conflicts (in case it stays on the patch backlog for a
while) because the same file is changed by another patch in the series.

Patch 2 and 3 are related to the migration but could be applied right-away
because they don't need the changes to the docker image. Alternatively they
could be squashed to the last patch.

Patch 4 adds Python 3 to the docker image.

Patch 5 migrates the test infra to use Python 3.

Below a full run of '-runtime-tests' for:
 - master [1];
 - master + 1,2,3 [2];
 - master + local docker image + full series [3].

[1] https://gitlab.com/RicardoMartincoski/buildroot/pipelines/75061226
[2] https://gitlab.com/RicardoMartincoski/buildroot/pipelines/75061478
[3] https://gitlab.com/RicardoMartincoski/buildroot/pipelines/75066229

Regards,
Ricardo

Ricardo Martincoski (5):
  support/testing: check ubi image before booting
  support/testing: do not print non-utf-8 chars on cbor test
  support/testing: factor out run_cmd_on_host
  support/docker: add python3
  support/testing: switch to Python 3 only

 support/docker/Dockerfile                     |  3 +++
 support/testing/infra/__init__.py             | 19 +++++++++++++++----
 support/testing/infra/emulator.py             |  1 +
 support/testing/run-tests                     |  2 +-
 support/testing/tests/core/test_hardening.py  |  6 +-----
 .../testing/tests/core/test_post_scripts.py   |  2 +-
 support/testing/tests/download/gitremote.py   |  3 ++-
 support/testing/tests/download/test_git.py    |  2 +-
 support/testing/tests/fs/test_ext.py          |  6 +-----
 support/testing/tests/fs/test_f2fs.py         |  5 +----
 support/testing/tests/fs/test_jffs2.py        |  6 ++----
 support/testing/tests/fs/test_squashfs.py     |  4 +---
 support/testing/tests/fs/test_ubi.py          |  5 ++---
 .../tests/package/sample_python_cbor_enc.py   |  2 +-
 .../testing/tests/utils/test_check_package.py |  3 ++-
 15 files changed, 35 insertions(+), 34 deletions(-)

-- 
2.17.1

[Buildroot] [PATCH 1/5] support/testing: check ubi image before booting

[Ricardo Martincoski]

Do the same as other fs tests and minimally check the ubi image before
booting.
The call to 'file' was already there, but the output wasn't tested for
some unknown reason. Add the assert for the output of the command.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
 support/testing/tests/fs/test_ubi.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/support/testing/tests/fs/test_ubi.py b/support/testing/tests/fs/test_ubi.py
index 015d82f769..e024d417df 100644
--- a/support/testing/tests/fs/test_ubi.py
+++ b/support/testing/tests/fs/test_ubi.py
@@ -25,6 +25,7 @@ class TestUbi(infra.basetest.BRTest):
                                       cwd=self.builddir,
                                       env={"LANG": "C"})
         out = out.splitlines()
+        self.assertIn("UBI image, version 1", out[0])
 
         subprocess.call(["truncate", "-s 128M", img])
 
-- 
2.17.1

[Buildroot] [PATCH 2/5] support/testing: do not print non-utf-8 chars on cbor test

[Ricardo Martincoski]

The python-cbor sample script to be run on the target dumps a binary
variable to the target stdout. This is done for debug purposes-only, in
the case the test on test infra fails.
This non-utf-8 is currently silently ignored by the Emulator class /
pexpect.spawn from the test infra because the infra uses Python 2, that
in turn do not differentiate between string and byte data.

Make the code Python3-friendly (Python 3 in the host) by doing the right
thing and encoding the data before printing it.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
 support/testing/tests/package/sample_python_cbor_enc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/support/testing/tests/package/sample_python_cbor_enc.py b/support/testing/tests/package/sample_python_cbor_enc.py
index 1bde92bb9f..31d937d4b8 100644
--- a/support/testing/tests/package/sample_python_cbor_enc.py
+++ b/support/testing/tests/package/sample_python_cbor_enc.py
@@ -9,6 +9,6 @@ data = {
     }
 }
 serialized = cbor.dumps(data)
-print(serialized)
+print(serialized.decode(errors="ignore"))
 with open("/tmp/data.cbor", "wb") as f:
     f.write(serialized)
-- 
2.17.1

[Buildroot] [PATCH 3/5] support/testing: factor out run_cmd_on_host

[Ricardo Martincoski]

Currently many test cases call subprocess.check_output on their own.
Factor out that code to an infra method so the call get standardized.

This will be handful when switching the test infra to use Python 3.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
---
 support/testing/infra/__init__.py            | 13 +++++++++++--
 support/testing/tests/core/test_hardening.py |  6 +-----
 support/testing/tests/fs/test_ext.py         |  6 +-----
 support/testing/tests/fs/test_f2fs.py        |  5 +----
 support/testing/tests/fs/test_jffs2.py       |  6 ++----
 support/testing/tests/fs/test_squashfs.py    |  4 +---
 support/testing/tests/fs/test_ubi.py         |  4 +---
 7 files changed, 18 insertions(+), 26 deletions(-)

diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py
index 1d4d18bbe9..43045d0173 100644
--- a/support/testing/infra/__init__.py
+++ b/support/testing/infra/__init__.py
@@ -55,6 +55,15 @@ def download(dldir, filename):
     return finalpath
 
 
+def run_cmd_on_host(builddir, cmd):
+    """Call subprocess.check_output and return the text output."""
+    out = subprocess.check_output(cmd,
+                                  stderr=open(os.devnull, "w"),
+                                  cwd=builddir,
+                                  env={"LANG": "C"})
+    return out
+
+
 def get_elf_arch_tag(builddir, prefix, fpath, tag):
     """
     Runs the cross readelf on 'fpath', then extracts the value of tag 'tag'.
@@ -66,7 +75,7 @@ def get_elf_arch_tag(builddir, prefix, fpath, tag):
     """
     cmd = ["host/bin/{}-readelf".format(prefix),
            "-A", os.path.join("target", fpath)]
-    out = subprocess.check_output(cmd, cwd=builddir, env={"LANG": "C"})
+    out = run_cmd_on_host(builddir, cmd)
     regexp = re.compile("^  {}: (.*)$".format(tag))
     for line in out.splitlines():
         m = regexp.match(line)
@@ -93,7 +102,7 @@ def get_elf_prog_interpreter(builddir, prefix, fpath):
     """
     cmd = ["host/bin/{}-readelf".format(prefix),
            "-l", os.path.join("target", fpath)]
-    out = subprocess.check_output(cmd, cwd=builddir, env={"LANG": "C"})
+    out = run_cmd_on_host(builddir, cmd)
     regexp = re.compile("^ *\[Requesting program interpreter: (.*)\]$")
     for line in out.splitlines():
         m = regexp.match(line)
diff --git a/support/testing/tests/core/test_hardening.py b/support/testing/tests/core/test_hardening.py
index 4d19b9f96d..edfb3315db 100644
--- a/support/testing/tests/core/test_hardening.py
+++ b/support/testing/tests/core/test_hardening.py
@@ -1,5 +1,4 @@
 import os
-import subprocess
 import json
 
 import infra.basetest
@@ -30,10 +29,7 @@ class TestHardeningBase(infra.basetest.BRTest):
                "--file={}".format(filepath)]
         # Checksec is being used for elf file analysis only.  There are no
         # assumptions of target/run-time checks as part of this testing.
-        ret = subprocess.check_output(cmd,
-                                      stderr=open(os.devnull, "w"),
-                                      cwd=self.builddir,
-                                      env={"LANG": "C"})
+        ret = infra.run_cmd_on_host(self.builddir, cmd)
         return json.loads(ret)
 
 
diff --git a/support/testing/tests/fs/test_ext.py b/support/testing/tests/fs/test_ext.py
index f5f9e9fdf1..16b308cf6d 100644
--- a/support/testing/tests/fs/test_ext.py
+++ b/support/testing/tests/fs/test_ext.py
@@ -1,5 +1,4 @@
 import os
-import subprocess
 
 import infra.basetest
 
@@ -15,10 +14,7 @@ CHECK_FS_TYPE_CMD = "mount | grep '/dev/root on / type {}'"
 
 def dumpe2fs_run(builddir, image):
     cmd = ["host/sbin/dumpe2fs", os.path.join("images", image)]
-    ret = subprocess.check_output(cmd,
-                                  stderr=open(os.devnull, "w"),
-                                  cwd=builddir,
-                                  env={"LANG": "C"})
+    ret = infra.run_cmd_on_host(builddir, cmd)
     return ret.strip().splitlines()
 
 
diff --git a/support/testing/tests/fs/test_f2fs.py b/support/testing/tests/fs/test_f2fs.py
index 819c619a36..950d4cf25b 100644
--- a/support/testing/tests/fs/test_f2fs.py
+++ b/support/testing/tests/fs/test_f2fs.py
@@ -1,5 +1,4 @@
 import os
-import subprocess
 
 import infra.basetest
 
@@ -29,9 +28,7 @@ class TestF2FS(infra.basetest.BRTest):
 
     def test_run(self):
         img = os.path.join(self.builddir, "images", "rootfs.f2fs")
-        out = subprocess.check_output(["host/sbin/dump.f2fs", img],
-                                      cwd=self.builddir,
-                                      env={"LANG": "C"})
+        out = infra.run_cmd_on_host(self.builddir, ["host/sbin/dump.f2fs", img])
         out = out.splitlines()
         prop = dumpf2fs_getprop(out, "Info: total sectors")
         self.assertEqual(prop, "262144 (128 MB)")
diff --git a/support/testing/tests/fs/test_jffs2.py b/support/testing/tests/fs/test_jffs2.py
index 2ff5099180..f5066ff01f 100644
--- a/support/testing/tests/fs/test_jffs2.py
+++ b/support/testing/tests/fs/test_jffs2.py
@@ -1,5 +1,4 @@
 import os
-import subprocess
 
 import infra.basetest
 
@@ -30,9 +29,8 @@ class TestJffs2(infra.basetest.BRTest):
 
     def test_run(self):
         img = os.path.join(self.builddir, "images", "rootfs.jffs2")
-        out = subprocess.check_output(["host/sbin/jffs2dump", "-c", img],
-                                      cwd=self.builddir,
-                                      env={"LANG": "C"})
+        cmd = ["host/sbin/jffs2dump", "-c", img]
+        out = infra.run_cmd_on_host(self.builddir, cmd)
         out = out.splitlines()
         self.assertTrue(jffs2dump_find_file(out, "busybox"))
 
diff --git a/support/testing/tests/fs/test_squashfs.py b/support/testing/tests/fs/test_squashfs.py
index 066c054342..234f4944be 100644
--- a/support/testing/tests/fs/test_squashfs.py
+++ b/support/testing/tests/fs/test_squashfs.py
@@ -15,9 +15,7 @@ class TestSquashfs(infra.basetest.BRTest):
 
     def test_run(self):
         unsquashfs_cmd = ["host/bin/unsquashfs", "-s", "images/rootfs.squashfs"]
-        out = subprocess.check_output(unsquashfs_cmd,
-                                      cwd=self.builddir,
-                                      env={"LANG": "C"})
+        out = infra.run_cmd_on_host(self.builddir, unsquashfs_cmd)
         out = out.splitlines()
         self.assertEqual(out[0],
                          "Found a valid SQUASHFS 4:0 superblock on images/rootfs.squashfs.")
diff --git a/support/testing/tests/fs/test_ubi.py b/support/testing/tests/fs/test_ubi.py
index e024d417df..7321f83da9 100644
--- a/support/testing/tests/fs/test_ubi.py
+++ b/support/testing/tests/fs/test_ubi.py
@@ -21,9 +21,7 @@ class TestUbi(infra.basetest.BRTest):
     # To be investigated.
     def test_run(self):
         img = os.path.join(self.builddir, "images", "rootfs.ubi")
-        out = subprocess.check_output(["file", img],
-                                      cwd=self.builddir,
-                                      env={"LANG": "C"})
+        out = infra.run_cmd_on_host(self.builddir, ["file", img])
         out = out.splitlines()
         self.assertIn("UBI image, version 1", out[0])
 
-- 
2.17.1

[Buildroot] [PATCH 4/5] support/docker: add python3

[Ricardo Martincoski]

The test infra will soon be converted to Python 3 only.
So add the interpreter and also the Python 3 variant of modules nose2
and pexpect to the docker image used to run runtime tests.

Keep the Python 2 variant of those modules to allow a gradual
transition.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
---
 support/docker/Dockerfile | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile
index 36cd3b28a4..03acde85d2 100644
--- a/support/docker/Dockerfile
+++ b/support/docker/Dockerfile
@@ -39,6 +39,9 @@ RUN apt-get install -y --no-install-recommends \
         python-flake8 \
         python-nose2 \
         python-pexpect \
+        python3 \
+        python3-nose2 \
+        python3-pexpect \
         qemu-system-arm \
         qemu-system-x86 \
         rsync \
-- 
2.17.1

[Buildroot] [PATCH 4/5] support/docker: add python3

[Thomas Petazzoni]

Hello Ricardo,

On Thu,  8 Aug 2019 20:10:14 -0300
Ricardo Martincoski <ricardo.martincoski@gmail.com> wrote:

> The test infra will soon be converted to Python 3 only.
> So add the interpreter and also the Python 3 variant of modules nose2
> and pexpect to the docker image used to run runtime tests.
> 
> Keep the Python 2 variant of those modules to allow a gradual
> transition.
> 
> Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
> Cc: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
>  support/docker/Dockerfile | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile
> index 36cd3b28a4..03acde85d2 100644
> --- a/support/docker/Dockerfile
> +++ b/support/docker/Dockerfile
> @@ -39,6 +39,9 @@ RUN apt-get install -y --no-install-recommends \
>          python-flake8 \
>          python-nose2 \
>          python-pexpect \
> +        python3 \
> +        python3-nose2 \
> +        python3-pexpect \

But then, can we remove python-nose2 and python-pexpect at the same
time ?

Yann: how do we regenerate/reupload the Docker image after this commit ?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [PATCH 4/5] support/docker: add python3

[Ricardo Martincoski]

Hello,

On Mon, Sep 23, 2019 at 05:13 PM, Thomas Petazzoni wrote:

> On Thu,  8 Aug 2019 20:10:14 -0300
> Ricardo Martincoski <ricardo.martincoski@gmail.com> wrote:
> 
>> The test infra will soon be converted to Python 3 only.
>> So add the interpreter and also the Python 3 variant of modules nose2
>> and pexpect to the docker image used to run runtime tests.
>> 
>> Keep the Python 2 variant of those modules to allow a gradual
>> transition.
>> 
>> Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
>> Cc: Yann E. MORIN <yann.morin.1998@free.fr>
>> ---
>>  support/docker/Dockerfile | 3 +++
>>  1 file changed, 3 insertions(+)
>> 
>> diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile
>> index 36cd3b28a4..03acde85d2 100644
>> --- a/support/docker/Dockerfile
>> +++ b/support/docker/Dockerfile
>> @@ -39,6 +39,9 @@ RUN apt-get install -y --no-install-recommends \
>>          python-flake8 \
>>          python-nose2 \
>>          python-pexpect \
>> +        python3 \
>> +        python3-nose2 \
>> +        python3-pexpect \
> 
> But then, can we remove python-nose2 and python-pexpect at the same
> time ?

You are right, we can.
We just need to change the image name in .gitlab-ci.* in the same patch that
converts the infra to Python 3 only.
I just tested it (with a image without python-{nose2,pexpect}):
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/83943587/builds

> 
> Yann: how do we regenerate/reupload the Docker image after this commit ?


Regards,
Ricardo

[Buildroot] [PATCH 4/5] support/docker: add python3

[Yann E. MORIN]

Thomas, All,

On 2019-09-23 22:13 +0200, Thomas Petazzoni spake thusly:
> Yann: how do we regenerate/reupload the Docker image after this commit ?

I am supposed to still have credentials to push to the dockerhub.

When the commit is applied, ping me (preferably by mail *and* IRC, and I
will generate and push the image. Then I will send a patch to update the
version we use in our .gitlab-ci.yml.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 5/5] support/testing: switch to Python 3 only

[Ricardo Martincoski]

Python 2.7 will not be maintained past 2020.

Many scripts on the tree are used during the build and should keep
Python 2 compatibility for a while.
This is not the case for the runtime test infra. It's meant to be run in
modern distros only, so it can safely switch to support Python 3 only.

An advantage of this approach is to have less scenarios to test in.
Otherwise every change to the test infra or runtime tests would need to
be tested against both versions of the interpreter, increasing the
effort of the developers, to ensure the compatibility to Python 2 was
not broken.

In order to accomplish the change to Python 3:
 - change the shebang for run-tests;
 - use Python 3 urllib as a drop-in replacement for Python 2 urllib2;
 - when writing the downloaded binary files, explicitly open the output
   file as binary;
 - when subprocess is used to retrieve the text output from commands,
   explicitly ask for text output. For this, use 'universal_newlines'
   because 'text' was added only on Python 3.7;
 - when pexpect is used to retrieve the text output from qemu or git,
   explicitly ask for text output using 'encoding';
 - the code using csv currently follows the example in the documentation
   for the Python 2 module, change it to follow the example in the
   documentation for the Python 3 module;
 - fix the relative import for test_git.py to be Python 3 compliant.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yann E. MORIN <yann.morin.1998@free.fr>
---
NOTE: obviously this patch cannot be applied before:
 - patch 4 is applied;
 - the new docker image is generated and uploaded to dockerhub;
 - a patch changing the name of the docker image to use is created and applied;
---
 support/testing/infra/__init__.py                 | 8 +++++---
 support/testing/infra/emulator.py                 | 1 +
 support/testing/run-tests                         | 2 +-
 support/testing/tests/core/test_post_scripts.py   | 2 +-
 support/testing/tests/download/gitremote.py       | 3 ++-
 support/testing/tests/download/test_git.py        | 2 +-
 support/testing/tests/utils/test_check_package.py | 3 ++-
 7 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py
index 43045d0173..6392aa679b 100644
--- a/support/testing/infra/__init__.py
+++ b/support/testing/infra/__init__.py
@@ -3,7 +3,8 @@ import re
 import sys
 import tempfile
 import subprocess
-from urllib2 import urlopen, HTTPError, URLError
+from urllib.request import urlopen
+from urllib.error import HTTPError, URLError
 
 ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
 BASE_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), "../../.."))
@@ -44,7 +45,7 @@ def download(dldir, filename):
 
     try:
         url_fh = urlopen(os.path.join(ARTIFACTS_URL, filename))
-        with open(tmpfile, "w+") as tmpfile_fh:
+        with open(tmpfile, "w+b") as tmpfile_fh:
             tmpfile_fh.write(url_fh.read())
     except (HTTPError, URLError) as err:
         os.unlink(tmpfile)
@@ -60,7 +61,8 @@ def run_cmd_on_host(builddir, cmd):
     out = subprocess.check_output(cmd,
                                   stderr=open(os.devnull, "w"),
                                   cwd=builddir,
-                                  env={"LANG": "C"})
+                                  env={"LANG": "C"},
+                                  universal_newlines=True)
     return out
 
 
diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
index 093a643a8b..5611ec96e8 100644
--- a/support/testing/infra/emulator.py
+++ b/support/testing/infra/emulator.py
@@ -76,6 +76,7 @@ class Emulator(object):
         self.logfile.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd))
         self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:],
                                   timeout=5 * self.timeout_multiplier,
+                                  encoding='utf-8',
                                   env={"QEMU_AUDIO_DRV": "none"})
         # We want only stdout into the log to avoid double echo
         self.qemu.logfile_read = self.logfile
diff --git a/support/testing/run-tests b/support/testing/run-tests
index 813b927045..74741aee1b 100755
--- a/support/testing/run-tests
+++ b/support/testing/run-tests
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
 import argparse
 import sys
 import os
diff --git a/support/testing/tests/core/test_post_scripts.py b/support/testing/tests/core/test_post_scripts.py
index 40a36b7904..bc61f4af58 100644
--- a/support/testing/tests/core/test_post_scripts.py
+++ b/support/testing/tests/core/test_post_scripts.py
@@ -20,7 +20,7 @@ class TestPostScripts(infra.basetest.BRTest):
 
     def check_post_log_file(self, f, what, target_dir):
         lines = {}
-        with open(os.path.join(self.builddir, "build", f), 'rb') as csvfile:
+        with open(os.path.join(self.builddir, "build", f), newline='') as csvfile:
             r = csv.reader(csvfile, delimiter=',')
             for row in r:
                 lines[row[0]] = row[1]
diff --git a/support/testing/tests/download/gitremote.py b/support/testing/tests/download/gitremote.py
index 3b35456dd1..7df252d031 100644
--- a/support/testing/tests/download/gitremote.py
+++ b/support/testing/tests/download/gitremote.py
@@ -32,7 +32,8 @@ class GitRemote(object):
         for port in range(GIT_REMOTE_PORT_INITIAL, GIT_REMOTE_PORT_LAST + 1):
             cmd = daemon_cmd + ["--port={port}".format(port=port)]
             self.logfile.write("> starting git remote with '{}'\n".format(" ".join(cmd)))
-            self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile)
+            self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile,
+                                        encoding='utf-8')
             ret = self.daemon.expect(["Ready to rumble",
                                       "Address already in use"])
             if ret == 0:
diff --git a/support/testing/tests/download/test_git.py b/support/testing/tests/download/test_git.py
index 2455557298..ec5b8f3fdd 100644
--- a/support/testing/tests/download/test_git.py
+++ b/support/testing/tests/download/test_git.py
@@ -1,7 +1,7 @@
 import os
 import shutil
 
-from gitremote import GitRemote
+from tests.download.gitremote import GitRemote
 
 import infra
 
diff --git a/support/testing/tests/utils/test_check_package.py b/support/testing/tests/utils/test_check_package.py
index 17c2fcf3bc..c70ba02324 100644
--- a/support/testing/tests/utils/test_check_package.py
+++ b/support/testing/tests/utils/test_check_package.py
@@ -16,7 +16,8 @@ import infra
 def call_script(args, env, cwd):
     """Call a script and return stdout and stderr as lists."""
     out, err = subprocess.Popen(args, cwd=cwd, stdout=subprocess.PIPE,
-                                stderr=subprocess.PIPE, env=env).communicate()
+                                stderr=subprocess.PIPE, env=env,
+                                universal_newlines=True).communicate()
     return out.splitlines(), err.splitlines()
 
 
-- 
2.17.1

[Buildroot] [PATCH 5/5] support/testing: switch to Python 3 only

[Romain Naour]

Hi Ricardo,

Le 09/08/2019 ? 01:10, Ricardo Martincoski a ?crit?:
> Python 2.7 will not be maintained past 2020.
> 
> Many scripts on the tree are used during the build and should keep
> Python 2 compatibility for a while.
> This is not the case for the runtime test infra. It's meant to be run in
> modern distros only, so it can safely switch to support Python 3 only.
> 
> An advantage of this approach is to have less scenarios to test in.
> Otherwise every change to the test infra or runtime tests would need to
> be tested against both versions of the interpreter, increasing the
> effort of the developers, to ensure the compatibility to Python 2 was
> not broken.
> 
> In order to accomplish the change to Python 3:
>  - change the shebang for run-tests;
>  - use Python 3 urllib as a drop-in replacement for Python 2 urllib2;
>  - when writing the downloaded binary files, explicitly open the output
>    file as binary;
>  - when subprocess is used to retrieve the text output from commands,
>    explicitly ask for text output. For this, use 'universal_newlines'
>    because 'text' was added only on Python 3.7;
>  - when pexpect is used to retrieve the text output from qemu or git,
>    explicitly ask for text output using 'encoding';
>  - the code using csv currently follows the example in the documentation
>    for the Python 2 module, change it to follow the example in the
>    documentation for the Python 3 module;
>  - fix the relative import for test_git.py to be Python 3 compliant.

I'm using Fedora 30 and there no python2-nose2 package anymore.
So, I'm using this patch to avoid installing python2-nose2 from pypi.

Tested-by: Romain Naour <romain.naour@smile.fr>

Best regards,
Romain


> 
> Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
> NOTE: obviously this patch cannot be applied before:
>  - patch 4 is applied;
>  - the new docker image is generated and uploaded to dockerhub;
>  - a patch changing the name of the docker image to use is created and applied;
> ---
>  support/testing/infra/__init__.py                 | 8 +++++---
>  support/testing/infra/emulator.py                 | 1 +
>  support/testing/run-tests                         | 2 +-
>  support/testing/tests/core/test_post_scripts.py   | 2 +-
>  support/testing/tests/download/gitremote.py       | 3 ++-
>  support/testing/tests/download/test_git.py        | 2 +-
>  support/testing/tests/utils/test_check_package.py | 3 ++-
>  7 files changed, 13 insertions(+), 8 deletions(-)
> 
> diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py
> index 43045d0173..6392aa679b 100644
> --- a/support/testing/infra/__init__.py
> +++ b/support/testing/infra/__init__.py
> @@ -3,7 +3,8 @@ import re
>  import sys
>  import tempfile
>  import subprocess
> -from urllib2 import urlopen, HTTPError, URLError
> +from urllib.request import urlopen
> +from urllib.error import HTTPError, URLError
>  
>  ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
>  BASE_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), "../../.."))
> @@ -44,7 +45,7 @@ def download(dldir, filename):
>  
>      try:
>          url_fh = urlopen(os.path.join(ARTIFACTS_URL, filename))
> -        with open(tmpfile, "w+") as tmpfile_fh:
> +        with open(tmpfile, "w+b") as tmpfile_fh:
>              tmpfile_fh.write(url_fh.read())
>      except (HTTPError, URLError) as err:
>          os.unlink(tmpfile)
> @@ -60,7 +61,8 @@ def run_cmd_on_host(builddir, cmd):
>      out = subprocess.check_output(cmd,
>                                    stderr=open(os.devnull, "w"),
>                                    cwd=builddir,
> -                                  env={"LANG": "C"})
> +                                  env={"LANG": "C"},
> +                                  universal_newlines=True)
>      return out
>  
>  
> diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py
> index 093a643a8b..5611ec96e8 100644
> --- a/support/testing/infra/emulator.py
> +++ b/support/testing/infra/emulator.py
> @@ -76,6 +76,7 @@ class Emulator(object):
>          self.logfile.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd))
>          self.qemu = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:],
>                                    timeout=5 * self.timeout_multiplier,
> +                                  encoding='utf-8',
>                                    env={"QEMU_AUDIO_DRV": "none"})
>          # We want only stdout into the log to avoid double echo
>          self.qemu.logfile_read = self.logfile
> diff --git a/support/testing/run-tests b/support/testing/run-tests
> index 813b927045..74741aee1b 100755
> --- a/support/testing/run-tests
> +++ b/support/testing/run-tests
> @@ -1,4 +1,4 @@
> -#!/usr/bin/env python2
> +#!/usr/bin/env python3
>  import argparse
>  import sys
>  import os
> diff --git a/support/testing/tests/core/test_post_scripts.py b/support/testing/tests/core/test_post_scripts.py
> index 40a36b7904..bc61f4af58 100644
> --- a/support/testing/tests/core/test_post_scripts.py
> +++ b/support/testing/tests/core/test_post_scripts.py
> @@ -20,7 +20,7 @@ class TestPostScripts(infra.basetest.BRTest):
>  
>      def check_post_log_file(self, f, what, target_dir):
>          lines = {}
> -        with open(os.path.join(self.builddir, "build", f), 'rb') as csvfile:
> +        with open(os.path.join(self.builddir, "build", f), newline='') as csvfile:
>              r = csv.reader(csvfile, delimiter=',')
>              for row in r:
>                  lines[row[0]] = row[1]
> diff --git a/support/testing/tests/download/gitremote.py b/support/testing/tests/download/gitremote.py
> index 3b35456dd1..7df252d031 100644
> --- a/support/testing/tests/download/gitremote.py
> +++ b/support/testing/tests/download/gitremote.py
> @@ -32,7 +32,8 @@ class GitRemote(object):
>          for port in range(GIT_REMOTE_PORT_INITIAL, GIT_REMOTE_PORT_LAST + 1):
>              cmd = daemon_cmd + ["--port={port}".format(port=port)]
>              self.logfile.write("> starting git remote with '{}'\n".format(" ".join(cmd)))
> -            self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile)
> +            self.daemon = pexpect.spawn(cmd[0], cmd[1:], logfile=self.logfile,
> +                                        encoding='utf-8')
>              ret = self.daemon.expect(["Ready to rumble",
>                                        "Address already in use"])
>              if ret == 0:
> diff --git a/support/testing/tests/download/test_git.py b/support/testing/tests/download/test_git.py
> index 2455557298..ec5b8f3fdd 100644
> --- a/support/testing/tests/download/test_git.py
> +++ b/support/testing/tests/download/test_git.py
> @@ -1,7 +1,7 @@
>  import os
>  import shutil
>  
> -from gitremote import GitRemote
> +from tests.download.gitremote import GitRemote
>  
>  import infra
>  
> diff --git a/support/testing/tests/utils/test_check_package.py b/support/testing/tests/utils/test_check_package.py
> index 17c2fcf3bc..c70ba02324 100644
> --- a/support/testing/tests/utils/test_check_package.py
> +++ b/support/testing/tests/utils/test_check_package.py
> @@ -16,7 +16,8 @@ import infra
>  def call_script(args, env, cwd):
>      """Call a script and return stdout and stderr as lists."""
>      out, err = subprocess.Popen(args, cwd=cwd, stdout=subprocess.PIPE,
> -                                stderr=subprocess.PIPE, env=env).communicate()
> +                                stderr=subprocess.PIPE, env=env,
> +                                universal_newlines=True).communicate()
>      return out.splitlines(), err.splitlines()
>  
>  
> 

[Buildroot] [PATCH 0/5] test infra using Python 3

[Thomas Petazzoni]

Hello Ricardo,

As usual, thanks for this work on the test infrastructure!

On Thu,  8 Aug 2019 20:10:10 -0300
Ricardo Martincoski <ricardo.martincoski@gmail.com> wrote:
 
> Ricardo Martincoski (5):
>   support/testing: check ubi image before booting
>   support/testing: do not print non-utf-8 chars on cbor test
>   support/testing: factor out run_cmd_on_host

I have applied to next those 3 preparation patches.

>   support/docker: add python3
>   support/testing: switch to Python 3 only

These two patches sound all good to me, but I'll give it a bit more
time for other to comment before applying.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com