* [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile
@ 2022-10-19 21:57 Oguz Ozhan
2022-10-21 7:44 ` Thomas Petazzoni via buildroot
0 siblings, 1 reply; 11+ messages in thread
From: Oguz Ozhan @ 2022-10-19 21:57 UTC (permalink / raw)
To: buildroot; +Cc: Oguz Ozhan, Ricardo Martincoski
Signed-off-by: Oguz Ozhan <oguz.ozhan@mind.be>
---
support/docker/Dockerfile | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile
index f54c31b54a..fd1527b1f9 100644
--- a/support/docker/Dockerfile
+++ b/support/docker/Dockerfile
@@ -38,8 +38,8 @@ RUN apt-get install -y --no-install-recommends \
mercurial \
openssh-server \
python3 \
+ python3-pip \
python3-flake8 \
- python3-nose2 \
python3-pexpect \
python3-pytest \
qemu-system-arm \
@@ -53,6 +53,9 @@ RUN apt-get install -y --no-install-recommends \
apt-get -y autoremove && \
apt-get -y clean
+# To be able to run tests in parallel
+RUN pip install pytest-parallel
+
# To be able to generate a toolchain with locales, enable one UTF-8 locale
RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
/usr/sbin/locale-gen
--
2.34.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile
2022-10-19 21:57 Oguz Ozhan
@ 2022-10-21 7:44 ` Thomas Petazzoni via buildroot
2022-10-21 9:18 ` Oguz Ozhan
0 siblings, 1 reply; 11+ messages in thread
From: Thomas Petazzoni via buildroot @ 2022-10-21 7:44 UTC (permalink / raw)
To: Oguz Ozhan; +Cc: Ricardo Martincoski, buildroot
Hello Oguz,
On Wed, 19 Oct 2022 23:57:39 +0200
Oguz Ozhan <oguz.ozhan@mind.be> wrote:
> Signed-off-by: Oguz Ozhan <oguz.ozhan@mind.be>
Thanks a lot for your patch series. However, all commit logs are empty.
What is the rationale/motivation for the changes in this patch series?
Why is pytest better than nose2? Could you send a new iteration with a
detailed commit log for each commit that explains the motivation for
the change, and also describes the change itself, especially in PATCH
2/3, where test cases see changes. BTW, these changes in test cases
could be done in a separate preparation commit, as they would still
work with nose2 I believe.
Also another concern is that the series is not really bisectable, but
admittedly I'm not sure it's easy to achieve without doing all the
changes in one single patch.
Best regards,
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 11+ messages in thread
* [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile
@ 2022-10-21 9:15 Oguz Ozhan
2022-10-21 9:15 ` [Buildroot] [PATCH 2/3] support/testing: replace nose2 with pytest - run-tests Oguz Ozhan
` (3 more replies)
0 siblings, 4 replies; 11+ messages in thread
From: Oguz Ozhan @ 2022-10-21 9:15 UTC (permalink / raw)
To: buildroot; +Cc: Oguz Ozhan, Oguz Ozhan, Ricardo Martincoski
From: Oguz Ozhan <oguz.ozhan@mind.com>
- From web page of nose2:
(https://docs.nose2.io/en/latest/)
nose2 vs pytest:
- pytest is an excellent test framework and we encourage users to consider it for new projects.
- It has a bigger team of maintainers and a larger community of users.
- pytest is more robust and has more ability compared to nose2
Signed-off-by: Oguz Ozhan <oguz.ozhan@mind.be>
---
support/docker/Dockerfile | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile
index f54c31b54a..fd1527b1f9 100644
--- a/support/docker/Dockerfile
+++ b/support/docker/Dockerfile
@@ -38,8 +38,8 @@ RUN apt-get install -y --no-install-recommends \
mercurial \
openssh-server \
python3 \
+ python3-pip \
python3-flake8 \
- python3-nose2 \
python3-pexpect \
python3-pytest \
qemu-system-arm \
@@ -53,6 +53,9 @@ RUN apt-get install -y --no-install-recommends \
apt-get -y autoremove && \
apt-get -y clean
+# To be able to run tests in parallel
+RUN pip install pytest-parallel
+
# To be able to generate a toolchain with locales, enable one UTF-8 locale
RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
/usr/sbin/locale-gen
--
2.34.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [Buildroot] [PATCH 2/3] support/testing: replace nose2 with pytest - run-tests
2022-10-21 9:15 [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile Oguz Ozhan
@ 2022-10-21 9:15 ` Oguz Ozhan
2022-10-23 16:51 ` Yann E. MORIN
2022-10-30 22:20 ` Ricardo Martincoski
2022-10-21 9:15 ` [Buildroot] [PATCH 3/3] support/testing: replace nose2 with pytest - CI Oguz Ozhan
` (2 subsequent siblings)
3 siblings, 2 replies; 11+ messages in thread
From: Oguz Ozhan @ 2022-10-21 9:15 UTC (permalink / raw)
To: buildroot; +Cc: Oguz Ozhan, Oguz Ozhan, Ricardo Martincoski
From: Oguz Ozhan <oguz.ozhan@mind.com>
- From web page of nose2:
(https://docs.nose2.io/en/latest/)
nose2 vs pytest:
- pytest is an excellent test framework and we encourage users to consider it for new projects.
- It has a bigger team of maintainers and a larger community of users.
- pytest is more robust and has more ability compared to nose2
- Changes in this patch:
- run-tests modified to use pytest instead of nose2 to call tests. A corresponding Collector class added for pytest to list the available tests
- nose2 plugins removed which are not required anymore
- in test_iso9660.py, local test_ routine names changed to not be accepted as test_ cases via pytest
Signed-off-by: Oguz Ozhan <oguz.ozhan@mind.be>
---
support/testing/conf/unittest.cfg | 3 --
support/testing/run-tests | 28 ++++++++++-------
support/testing/tests/fs/test_iso9660.py | 40 ++++++++++++------------
3 files changed, 36 insertions(+), 35 deletions(-)
diff --git a/support/testing/conf/unittest.cfg b/support/testing/conf/unittest.cfg
index 4f516fb80a..0dfffcfbda 100644
--- a/support/testing/conf/unittest.cfg
+++ b/support/testing/conf/unittest.cfg
@@ -1,5 +1,2 @@
-[unittest]
-plugins = nose2.plugins.mp
-
[multiprocess]
always-on = True
diff --git a/support/testing/run-tests b/support/testing/run-tests
index bf40019362..f887bc6715 100755
--- a/support/testing/run-tests
+++ b/support/testing/run-tests
@@ -4,10 +4,16 @@ import multiprocessing
import os
import sys
-import nose2
+import pytest
from infra.basetest import BRConfigTest
+class PyTestCollector:
+ def __init__(self):
+ self.collected = []
+ def pytest_collection_modifyitems(self, items):
+ for item in items:
+ self.collected.append(item.nodeid)
def main():
parser = argparse.ArgumentParser(description='Run Buildroot tests')
@@ -42,12 +48,10 @@ def main():
BRConfigTest.logtofile = False
if args.list:
- print("List of tests")
- nose2.discover(argv=[script_path,
- "-s", test_dir,
- "-v",
- "--collect-only"],
- plugins=["nose2.plugins.collect"])
+ collect_plugin = PyTestCollector()
+ pytest.main(['--collect-only', '-p', 'no:terminal', test_dir], plugins=[collect_plugin])
+ for nodeid in collect_plugin.collected:
+ print(nodeid)
return 0
if args.download is None:
@@ -106,15 +110,15 @@ def main():
return 1
BRConfigTest.timeout_multiplier = args.timeout_multiplier
- nose2_args = ["-v",
- "-N", str(args.testcases),
- "-s", test_dir,
+ pytest_args = ["--workers", str(args.testcases),
+ "--rootdir", test_dir,
"-c", os.path.join(test_dir, "conf/unittest.cfg")]
if args.testname:
- nose2_args += args.testname
+ pytest_args += args.testname
+
+ pytest.main(pytest_args)
- nose2.discover(argv=nose2_args)
if __name__ == "__main__":
diff --git a/support/testing/tests/fs/test_iso9660.py b/support/testing/tests/fs/test_iso9660.py
index 692291267e..d2390cccf1 100644
--- a/support/testing/tests/fs/test_iso9660.py
+++ b/support/testing/tests/fs/test_iso9660.py
@@ -25,7 +25,7 @@ BASIC_CONFIG = \
""".format(infra.filepath("conf/minimal-x86-qemu-kernel.config"))
-def test_mount_internal_external(emulator, builddir, internal=True, efi=False):
+def do_test_mount_internal_external(emulator, builddir, internal=True, efi=False):
img = os.path.join(builddir, "images", "rootfs.iso9660")
if efi:
efi_img = os.path.join(builddir, "images", "OVMF.fd")
@@ -43,7 +43,7 @@ def test_mount_internal_external(emulator, builddir, internal=True, efi=False):
return exit_code
-def test_touch_file(emulator):
+def do_test_touch_file(emulator):
_, exit_code = emulator.run("touch test")
return exit_code
@@ -63,11 +63,11 @@ class TestIso9660Grub2External(infra.basetest.BRTest):
""".format(infra.filepath("conf/grub2.cfg"))
def test_run(self):
- exit_code = test_mount_internal_external(self.emulator,
+ exit_code = do_test_mount_internal_external(self.emulator,
self.builddir, internal=False)
self.assertEqual(exit_code, 0)
- exit_code = test_touch_file(self.emulator)
+ exit_code = do_test_touch_file(self.emulator)
self.assertEqual(exit_code, 1)
@@ -84,11 +84,11 @@ class TestIso9660Grub2ExternalCompress(infra.basetest.BRTest):
""".format(infra.filepath("conf/grub2.cfg"))
def test_run(self):
- exit_code = test_mount_internal_external(self.emulator,
+ exit_code = do_test_mount_internal_external(self.emulator,
self.builddir, internal=False)
self.assertEqual(exit_code, 0)
- exit_code = test_touch_file(self.emulator)
+ exit_code = do_test_touch_file(self.emulator)
self.assertEqual(exit_code, 1)
@@ -104,11 +104,11 @@ class TestIso9660Grub2Internal(infra.basetest.BRTest):
""".format(infra.filepath("conf/grub2.cfg"))
def test_run(self):
- exit_code = test_mount_internal_external(self.emulator,
+ exit_code = do_test_mount_internal_external(self.emulator,
self.builddir, internal=True)
self.assertEqual(exit_code, 0)
- exit_code = test_touch_file(self.emulator)
+ exit_code = do_test_touch_file(self.emulator)
self.assertEqual(exit_code, 0)
@@ -127,12 +127,12 @@ class TestIso9660Grub2EFI(infra.basetest.BRTest):
infra.filepath("conf/grub2.cfg"))
def test_run(self):
- exit_code = test_mount_internal_external(self.emulator,
+ exit_code = do_test_mount_internal_external(self.emulator,
self.builddir, internal=True,
efi=True)
self.assertEqual(exit_code, 0)
- exit_code = test_touch_file(self.emulator)
+ exit_code = do_test_touch_file(self.emulator)
self.assertEqual(exit_code, 0)
@@ -155,22 +155,22 @@ class TestIso9660Grub2Hybrid(infra.basetest.BRTest):
infra.filepath("conf/grub2.cfg"))
def test_run(self):
- exit_code = test_mount_internal_external(self.emulator,
+ exit_code = do_test_mount_internal_external(self.emulator,
self.builddir, internal=True,
efi=False)
self.assertEqual(exit_code, 0)
- exit_code = test_touch_file(self.emulator)
+ exit_code = do_test_touch_file(self.emulator)
self.assertEqual(exit_code, 0)
self.emulator.stop()
- exit_code = test_mount_internal_external(self.emulator,
+ exit_code = do_test_mount_internal_external(self.emulator,
self.builddir, internal=True,
efi=True)
self.assertEqual(exit_code, 0)
- exit_code = test_touch_file(self.emulator)
+ exit_code = do_test_touch_file(self.emulator)
self.assertEqual(exit_code, 0)
@@ -189,11 +189,11 @@ class TestIso9660SyslinuxExternal(infra.basetest.BRTest):
""".format(infra.filepath("conf/isolinux.cfg"))
def test_run(self):
- exit_code = test_mount_internal_external(self.emulator,
+ exit_code = do_test_mount_internal_external(self.emulator,
self.builddir, internal=False)
self.assertEqual(exit_code, 0)
- exit_code = test_touch_file(self.emulator)
+ exit_code = do_test_touch_file(self.emulator)
self.assertEqual(exit_code, 1)
@@ -209,11 +209,11 @@ class TestIso9660SyslinuxExternalCompress(infra.basetest.BRTest):
""".format(infra.filepath("conf/isolinux.cfg"))
def test_run(self):
- exit_code = test_mount_internal_external(self.emulator,
+ exit_code = do_test_mount_internal_external(self.emulator,
self.builddir, internal=False)
self.assertEqual(exit_code, 0)
- exit_code = test_touch_file(self.emulator)
+ exit_code = do_test_touch_file(self.emulator)
self.assertEqual(exit_code, 1)
@@ -228,9 +228,9 @@ class TestIso9660SyslinuxInternal(infra.basetest.BRTest):
""".format(infra.filepath("conf/isolinux.cfg"))
def test_run(self):
- exit_code = test_mount_internal_external(self.emulator,
+ exit_code = do_test_mount_internal_external(self.emulator,
self.builddir, internal=True)
self.assertEqual(exit_code, 0)
- exit_code = test_touch_file(self.emulator)
+ exit_code = do_test_touch_file(self.emulator)
self.assertEqual(exit_code, 0)
--
2.34.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [Buildroot] [PATCH 3/3] support/testing: replace nose2 with pytest - CI
2022-10-21 9:15 [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile Oguz Ozhan
2022-10-21 9:15 ` [Buildroot] [PATCH 2/3] support/testing: replace nose2 with pytest - run-tests Oguz Ozhan
@ 2022-10-21 9:15 ` Oguz Ozhan
2022-10-30 22:20 ` Ricardo Martincoski
2022-10-23 16:44 ` [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile Yann E. MORIN
2022-10-30 22:17 ` Ricardo Martincoski
3 siblings, 1 reply; 11+ messages in thread
From: Oguz Ozhan @ 2022-10-21 9:15 UTC (permalink / raw)
To: buildroot; +Cc: Oguz Ozhan, Oguz Ozhan, Ricardo Martincoski
From: Oguz Ozhan <oguz.ozhan@mind.com>
- From web page of nose2:
(https://docs.nose2.io/en/latest/)
nose2 vs pytest:
- pytest is an excellent test framework and we encourage users to consider it for new projects.
- It has a bigger team of maintainers and a larger community of users.
- pytest is more robust and has more ability compared to nose2
- Changes in this patch:
- the sed command was fine filtering the tests list with nose2 output but removing all operating on pytest output. With pytest no fine filtering is required to get the test names in correct format.
Signed-off-by: Oguz Ozhan <oguz.ozhan@mind.be>
---
support/scripts/generate-gitlab-ci-yml | 2 --
1 file changed, 2 deletions(-)
diff --git a/support/scripts/generate-gitlab-ci-yml b/support/scripts/generate-gitlab-ci-yml
index aa43aac019..bb4dcbbb22 100755
--- a/support/scripts/generate-gitlab-ci-yml
+++ b/support/scripts/generate-gitlab-ci-yml
@@ -31,7 +31,6 @@ gen_tests() {
defconfigs=( $(cd configs; LC_ALL=C ls -1 *_defconfig) )
runtimes=( $(./support/testing/run-tests -l 2>&1 \
- | sed -r -e '/^test_run \((.*)\).*/!d; s//\1/' \
| LC_ALL=C sort)
)
@@ -80,7 +79,6 @@ gen_tests() {
;;
(*-tests.*)
runtimes=( $(./support/testing/run-tests -l 2>&1 \
- | sed -r -e '/^test_run \((.*)\).*/!d; s//\1/' \
| LC_ALL=C sort \
| grep "^${CI_COMMIT_REF_NAME##*-}")
)
--
2.34.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile
2022-10-21 7:44 ` Thomas Petazzoni via buildroot
@ 2022-10-21 9:18 ` Oguz Ozhan
0 siblings, 0 replies; 11+ messages in thread
From: Oguz Ozhan @ 2022-10-21 9:18 UTC (permalink / raw)
To: Thomas Petazzoni; +Cc: Ricardo Martincoski, buildroot
[-- Attachment #1.1: Type: text/plain, Size: 1231 bytes --]
Hi Thomas.
Thanks a lot for your comments.
I inserted more information in the commit logs and re-sent the patches.
KR.
Oguz
On Fri, Oct 21, 2022 at 9:44 AM Thomas Petazzoni <
thomas.petazzoni@bootlin.com> wrote:
> Hello Oguz,
>
> On Wed, 19 Oct 2022 23:57:39 +0200
> Oguz Ozhan <oguz.ozhan@mind.be> wrote:
>
> > Signed-off-by: Oguz Ozhan <oguz.ozhan@mind.be>
>
> Thanks a lot for your patch series. However, all commit logs are empty.
> What is the rationale/motivation for the changes in this patch series?
> Why is pytest better than nose2? Could you send a new iteration with a
> detailed commit log for each commit that explains the motivation for
> the change, and also describes the change itself, especially in PATCH
> 2/3, where test cases see changes. BTW, these changes in test cases
> could be done in a separate preparation commit, as they would still
> work with nose2 I believe.
>
> Also another concern is that the series is not really bisectable, but
> admittedly I'm not sure it's easy to achieve without doing all the
> changes in one single patch.
>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com
>
[-- Attachment #1.2: Type: text/html, Size: 1916 bytes --]
[-- Attachment #2: Type: text/plain, Size: 150 bytes --]
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile
2022-10-21 9:15 [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile Oguz Ozhan
2022-10-21 9:15 ` [Buildroot] [PATCH 2/3] support/testing: replace nose2 with pytest - run-tests Oguz Ozhan
2022-10-21 9:15 ` [Buildroot] [PATCH 3/3] support/testing: replace nose2 with pytest - CI Oguz Ozhan
@ 2022-10-23 16:44 ` Yann E. MORIN
2022-10-30 22:17 ` Ricardo Martincoski
3 siblings, 0 replies; 11+ messages in thread
From: Yann E. MORIN @ 2022-10-23 16:44 UTC (permalink / raw)
To: Oguz Ozhan; +Cc: Oguz Ozhan, Ricardo Martincoski, buildroot
Oguz, All,
On 2022-10-21 11:15 +0200, Oguz Ozhan spake thusly:
> From: Oguz Ozhan <oguz.ozhan@mind.com>
>
> - From web page of nose2:
> (https://docs.nose2.io/en/latest/)
> nose2 vs pytest:
> - pytest is an excellent test framework and we encourage users to consider it for new projects.
> - It has a bigger team of maintainers and a larger community of users.
This clearly states "for new projects", but we are not new! ;-)
Why do we need to switch to pytest, and what does that bring us? (see
below too)
> - pytest is more robust and has more ability compared to nose2
You provide robustness and more abilities to advocate for switching to
pytest, however, you do not explain what is currently broken that would
be fixed by the pytest robustness, or what new features from pytest we
could leverage, and how.
Note that I am not opposed to switching, but we need a good rationale
for switching.
This commit should also come after we actually did the switch, i.e. we
should first convert support/testing/ over to pytest, then modify the
dockerfile accordingly, which allows to write a better coomit log.
Also, please organise your commit logs as (first line is the title):
support/docker: install pytest instead of nose2
Now that the tuntime test infra has switched to using pytest instead
of nose2, install the appropriate packages in the Dockerfile for our
reference build environment.
Note that pytest-parallel is not available i Debian Bullseye, so we
install it (and its depenencies) from Pypi instead.
Signed-off-by: You
See more below...
> Signed-off-by: Oguz Ozhan <oguz.ozhan@mind.be>
> ---
> support/docker/Dockerfile | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/support/docker/Dockerfile b/support/docker/Dockerfile
> index f54c31b54a..fd1527b1f9 100644
> --- a/support/docker/Dockerfile
> +++ b/support/docker/Dockerfile
> @@ -38,8 +38,8 @@ RUN apt-get install -y --no-install-recommends \
> mercurial \
> openssh-server \
> python3 \
> + python3-pip \
> python3-flake8 \
> - python3-nose2 \
> python3-pexpect \
> python3-pytest \
> qemu-system-arm \
> @@ -53,6 +53,9 @@ RUN apt-get install -y --no-install-recommends \
> apt-get -y autoremove && \
> apt-get -y clean
>
> +# To be able to run tests in parallel
> +RUN pip install pytest-parallel
Please, pin the version so that the image is reproducible.
Also, if pytest-parallel has dependencies, explicitly install those at
pinned versions too.
Also note in the commit log that pytest-parallel is not available in
bullseye, so we need to get it from Pypi instead (as in my suggestion).
Regards,
Yann E. MORIN.
> # To be able to generate a toolchain with locales, enable one UTF-8 locale
> RUN sed -i 's/# \(en_US.UTF-8\)/\1/' /etc/locale.gen && \
> /usr/sbin/locale-gen
> --
> 2.34.1
>
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Buildroot] [PATCH 2/3] support/testing: replace nose2 with pytest - run-tests
2022-10-21 9:15 ` [Buildroot] [PATCH 2/3] support/testing: replace nose2 with pytest - run-tests Oguz Ozhan
@ 2022-10-23 16:51 ` Yann E. MORIN
2022-10-30 22:20 ` Ricardo Martincoski
1 sibling, 0 replies; 11+ messages in thread
From: Yann E. MORIN @ 2022-10-23 16:51 UTC (permalink / raw)
To: Oguz Ozhan; +Cc: Oguz Ozhan, Ricardo Martincoski, buildroot
Oguz, All,
On 2022-10-21 11:15 +0200, Oguz Ozhan spake thusly:
> From: Oguz Ozhan <oguz.ozhan@mind.com>
>
> - From web page of nose2:
> (https://docs.nose2.io/en/latest/)
> nose2 vs pytest:
> - pytest is an excellent test framework and we encourage users to consider it for new projects.
> - It has a bigger team of maintainers and a larger community of users.
>
> - pytest is more robust and has more ability compared to nose2
Please see my comment in patch 1 about providing a rationale for the
switch.
Please fold your commit logs at 72 chars.
> - Changes in this patch:
> - run-tests modified to use pytest instead of nose2 to call tests. A corresponding Collector class added for pytest to list the available tests
> - nose2 plugins removed which are not required anymore
Don't describe the changes, explain them.
E.g. "nose2 plugins removed which are not required anymore" we can see
they get removed, and since the change is about switching away from
nose2, it is obvious they get removed because they are not used anymore.
> - in test_iso9660.py, local test_ routine names changed to not be accepted as test_ cases via pytest
This last part should be a patch by itself, coming first in the series,
as it can be applied without doing the conversion yet.
I have only had a cursory look at the changes, they look quite OK.
Be sure to run "make check-flake8", I think I could spot too many lines
at one place...
Regards,
Yann E. MORIN.
> Signed-off-by: Oguz Ozhan <oguz.ozhan@mind.be>
> ---
> support/testing/conf/unittest.cfg | 3 --
> support/testing/run-tests | 28 ++++++++++-------
> support/testing/tests/fs/test_iso9660.py | 40 ++++++++++++------------
> 3 files changed, 36 insertions(+), 35 deletions(-)
>
> diff --git a/support/testing/conf/unittest.cfg b/support/testing/conf/unittest.cfg
> index 4f516fb80a..0dfffcfbda 100644
> --- a/support/testing/conf/unittest.cfg
> +++ b/support/testing/conf/unittest.cfg
> @@ -1,5 +1,2 @@
> -[unittest]
> -plugins = nose2.plugins.mp
> -
> [multiprocess]
> always-on = True
> diff --git a/support/testing/run-tests b/support/testing/run-tests
> index bf40019362..f887bc6715 100755
> --- a/support/testing/run-tests
> +++ b/support/testing/run-tests
> @@ -4,10 +4,16 @@ import multiprocessing
> import os
> import sys
>
> -import nose2
> +import pytest
>
> from infra.basetest import BRConfigTest
>
> +class PyTestCollector:
> + def __init__(self):
> + self.collected = []
> + def pytest_collection_modifyitems(self, items):
> + for item in items:
> + self.collected.append(item.nodeid)
>
> def main():
> parser = argparse.ArgumentParser(description='Run Buildroot tests')
> @@ -42,12 +48,10 @@ def main():
> BRConfigTest.logtofile = False
>
> if args.list:
> - print("List of tests")
> - nose2.discover(argv=[script_path,
> - "-s", test_dir,
> - "-v",
> - "--collect-only"],
> - plugins=["nose2.plugins.collect"])
> + collect_plugin = PyTestCollector()
> + pytest.main(['--collect-only', '-p', 'no:terminal', test_dir], plugins=[collect_plugin])
> + for nodeid in collect_plugin.collected:
> + print(nodeid)
> return 0
>
> if args.download is None:
> @@ -106,15 +110,15 @@ def main():
> return 1
> BRConfigTest.timeout_multiplier = args.timeout_multiplier
>
> - nose2_args = ["-v",
> - "-N", str(args.testcases),
> - "-s", test_dir,
> + pytest_args = ["--workers", str(args.testcases),
> + "--rootdir", test_dir,
> "-c", os.path.join(test_dir, "conf/unittest.cfg")]
>
> if args.testname:
> - nose2_args += args.testname
> + pytest_args += args.testname
> +
> + pytest.main(pytest_args)
>
> - nose2.discover(argv=nose2_args)
>
>
> if __name__ == "__main__":
> diff --git a/support/testing/tests/fs/test_iso9660.py b/support/testing/tests/fs/test_iso9660.py
> index 692291267e..d2390cccf1 100644
> --- a/support/testing/tests/fs/test_iso9660.py
> +++ b/support/testing/tests/fs/test_iso9660.py
> @@ -25,7 +25,7 @@ BASIC_CONFIG = \
> """.format(infra.filepath("conf/minimal-x86-qemu-kernel.config"))
>
>
> -def test_mount_internal_external(emulator, builddir, internal=True, efi=False):
> +def do_test_mount_internal_external(emulator, builddir, internal=True, efi=False):
> img = os.path.join(builddir, "images", "rootfs.iso9660")
> if efi:
> efi_img = os.path.join(builddir, "images", "OVMF.fd")
> @@ -43,7 +43,7 @@ def test_mount_internal_external(emulator, builddir, internal=True, efi=False):
> return exit_code
>
>
> -def test_touch_file(emulator):
> +def do_test_touch_file(emulator):
> _, exit_code = emulator.run("touch test")
> return exit_code
>
> @@ -63,11 +63,11 @@ class TestIso9660Grub2External(infra.basetest.BRTest):
> """.format(infra.filepath("conf/grub2.cfg"))
>
> def test_run(self):
> - exit_code = test_mount_internal_external(self.emulator,
> + exit_code = do_test_mount_internal_external(self.emulator,
> self.builddir, internal=False)
> self.assertEqual(exit_code, 0)
>
> - exit_code = test_touch_file(self.emulator)
> + exit_code = do_test_touch_file(self.emulator)
> self.assertEqual(exit_code, 1)
>
>
> @@ -84,11 +84,11 @@ class TestIso9660Grub2ExternalCompress(infra.basetest.BRTest):
> """.format(infra.filepath("conf/grub2.cfg"))
>
> def test_run(self):
> - exit_code = test_mount_internal_external(self.emulator,
> + exit_code = do_test_mount_internal_external(self.emulator,
> self.builddir, internal=False)
> self.assertEqual(exit_code, 0)
>
> - exit_code = test_touch_file(self.emulator)
> + exit_code = do_test_touch_file(self.emulator)
> self.assertEqual(exit_code, 1)
>
>
> @@ -104,11 +104,11 @@ class TestIso9660Grub2Internal(infra.basetest.BRTest):
> """.format(infra.filepath("conf/grub2.cfg"))
>
> def test_run(self):
> - exit_code = test_mount_internal_external(self.emulator,
> + exit_code = do_test_mount_internal_external(self.emulator,
> self.builddir, internal=True)
> self.assertEqual(exit_code, 0)
>
> - exit_code = test_touch_file(self.emulator)
> + exit_code = do_test_touch_file(self.emulator)
> self.assertEqual(exit_code, 0)
>
>
> @@ -127,12 +127,12 @@ class TestIso9660Grub2EFI(infra.basetest.BRTest):
> infra.filepath("conf/grub2.cfg"))
>
> def test_run(self):
> - exit_code = test_mount_internal_external(self.emulator,
> + exit_code = do_test_mount_internal_external(self.emulator,
> self.builddir, internal=True,
> efi=True)
> self.assertEqual(exit_code, 0)
>
> - exit_code = test_touch_file(self.emulator)
> + exit_code = do_test_touch_file(self.emulator)
> self.assertEqual(exit_code, 0)
>
>
> @@ -155,22 +155,22 @@ class TestIso9660Grub2Hybrid(infra.basetest.BRTest):
> infra.filepath("conf/grub2.cfg"))
>
> def test_run(self):
> - exit_code = test_mount_internal_external(self.emulator,
> + exit_code = do_test_mount_internal_external(self.emulator,
> self.builddir, internal=True,
> efi=False)
> self.assertEqual(exit_code, 0)
>
> - exit_code = test_touch_file(self.emulator)
> + exit_code = do_test_touch_file(self.emulator)
> self.assertEqual(exit_code, 0)
>
> self.emulator.stop()
>
> - exit_code = test_mount_internal_external(self.emulator,
> + exit_code = do_test_mount_internal_external(self.emulator,
> self.builddir, internal=True,
> efi=True)
> self.assertEqual(exit_code, 0)
>
> - exit_code = test_touch_file(self.emulator)
> + exit_code = do_test_touch_file(self.emulator)
> self.assertEqual(exit_code, 0)
>
>
> @@ -189,11 +189,11 @@ class TestIso9660SyslinuxExternal(infra.basetest.BRTest):
> """.format(infra.filepath("conf/isolinux.cfg"))
>
> def test_run(self):
> - exit_code = test_mount_internal_external(self.emulator,
> + exit_code = do_test_mount_internal_external(self.emulator,
> self.builddir, internal=False)
> self.assertEqual(exit_code, 0)
>
> - exit_code = test_touch_file(self.emulator)
> + exit_code = do_test_touch_file(self.emulator)
> self.assertEqual(exit_code, 1)
>
>
> @@ -209,11 +209,11 @@ class TestIso9660SyslinuxExternalCompress(infra.basetest.BRTest):
> """.format(infra.filepath("conf/isolinux.cfg"))
>
> def test_run(self):
> - exit_code = test_mount_internal_external(self.emulator,
> + exit_code = do_test_mount_internal_external(self.emulator,
> self.builddir, internal=False)
> self.assertEqual(exit_code, 0)
>
> - exit_code = test_touch_file(self.emulator)
> + exit_code = do_test_touch_file(self.emulator)
> self.assertEqual(exit_code, 1)
>
>
> @@ -228,9 +228,9 @@ class TestIso9660SyslinuxInternal(infra.basetest.BRTest):
> """.format(infra.filepath("conf/isolinux.cfg"))
>
> def test_run(self):
> - exit_code = test_mount_internal_external(self.emulator,
> + exit_code = do_test_mount_internal_external(self.emulator,
> self.builddir, internal=True)
> self.assertEqual(exit_code, 0)
>
> - exit_code = test_touch_file(self.emulator)
> + exit_code = do_test_touch_file(self.emulator)
> self.assertEqual(exit_code, 0)
> --
> 2.34.1
>
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile
2022-10-21 9:15 [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile Oguz Ozhan
` (2 preceding siblings ...)
2022-10-23 16:44 ` [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile Yann E. MORIN
@ 2022-10-30 22:17 ` Ricardo Martincoski
3 siblings, 0 replies; 11+ messages in thread
From: Ricardo Martincoski @ 2022-10-30 22:17 UTC (permalink / raw)
To: oguz.ozhan; +Cc: buildroot
[-- Attachment #1: Type: text/plain, Size: 4435 bytes --]
Hello,
I am not against to switch to pytest.
But I didn't understand clearly yet the use case that motivated this series, in
which nose2 is not good and pytest is better. Could you elaborate?
Also, there are issues to be addressed before merging this series:
- the detection of a failed test on GitLab CI is broken, even tests failing
return exit code 0, so GitLab infra mark them as PASS, see [1];
- run-tests -s is broken (does not show the build and runtime console);
- support to ask a single runtime test to run on GitLab CI is broken due to a
regexp that needs to be adapted;
- the manual needs to be updated.
There are also some changes in behavior that would be nice to be highlighted:
- someone willing to find a test in the output of run-tests -l can do:
nose2: run-tests -l 2>&1 | grep pattern
pytest: run-tests -l | grep pattern
- total number of tests:
nose2: run-tests -l prints the number of tests at the end "Ran 481 tests"
pytest: run-tests -l does not
- the name of tests change (list)
nose2: test_run (tests.toolchain.test_external_bootlin.\
TestExternalToolchainBootlinPowerpc64lepower8GlibcBleedingEdge) ... ok
pytest: support/testing/tests/toolchain/test_external_bootlin.py::\
TestExternalToolchainBootlinPowerpc64lepower8GlibcBleedingEdge::test_run
- the name of tests change (local run)
nose2: support/testing/run-tests tests.init.test_busybox \
tests.package.test_php_lua.TestPhpLuaLuajit
pytest: support/testing/run-tests support/testing/tests/init/test_busybox.py \
support/testing/tests/package/test_php_lua.py::TestPhpLuaLua
- the name of tests change (gitlab test suite)
nose2: git push gitlab HEAD:foo-tests.init.test_busybox
see [2]
pytest: git push gitlab HEAD:foo-support/testing/tests/init/test_busybox.py
see [3]
- the name of tests change (gitlab test case)
nose2: git push gitlab HEAD:foo-tests.package.test_php_lua.TestPhpLuaLuajit
see [4]
pytest: git push gitlab \
HEAD:foo-support/testing/tests/package/test_php_lua.py--TestPhpLuaLuajit
see [5]
NOTICE: the use of '--' is just the first solution that came to mind.
- the name of tests are longer, so in the GitLab CI the display is a bit worse:
nose2: tests.init.test_busybox.TestInitSystemBusyboxRo
tests.init.test_busybox.TestInitSystemBusybox...
tests.init.test_busybox.TestInitSystemBusybox...
tests.init.test_busybox.TestInitSystemBusybox...
tests.package.test_php_lua.TestPhpLuaLua
tests.package.test_php_lua.TestPhpLuaLuajit
pytest: support/testing/tests/init/test_busybox.py::Tes...
support/testing/tests/init/test_busybox.py::Tes...
support/testing/tests/init/test_busybox.py::Tes...
support/testing/tests/init/test_busybox.py::Tes...
support/testing/tests/package/test_php_lua.p...
support/testing/tests/package/test_php_lua.p...
- bash completion for the name of a test suite
nose2: no completion
support/testing/run-tests tests.init.test_busybox
pytest: since it uses the path as the begin of test name one can type
'support/testing/run-tests support/testing' and use bash completion for
test files names:
support/testing/run-tests support/testing/tests/init/test_busybox.py
- cache of results
nose2: no cache
pytest: it creates a .pytest_cache/ with a cache of the results, so we could
later expose another parameter in run-tests that passes '-lf' to pytest,
which will re-run only the tests that failed on last run
- verbosity when collecting tests:
nose2: no output
pytest: prints a summary 'collected 1 item'
'pytest-parallel: 1 worker (process), 1 test per worker (thread)'
- verbosity while tests are running:
nose2: show states for each test (Starting, Building)
pytest: shows nothing before each test ends
- verbosity when tests fail:
nose2: verbose
pytest: more verbose than nose2
- verbosity when tests pass:
nose2: show states for each test (Starting, Building)
pytest: show only a '.' for each test
[1] https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/3248620433
[2] https://gitlab.com/RicardoMartincoski/buildroot/-/pipelines/680878149
[3] https://gitlab.com/RicardoMartincoski/buildroot/-/pipelines/680879324
[4] https://gitlab.com/RicardoMartincoski/buildroot/-/pipelines/680879308
[5] https://gitlab.com/RicardoMartincoski/buildroot/-/pipelines/680878335
Regards,
Ricardo
[-- Attachment #2: Type: text/plain, Size: 150 bytes --]
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Buildroot] [PATCH 2/3] support/testing: replace nose2 with pytest - run-tests
2022-10-21 9:15 ` [Buildroot] [PATCH 2/3] support/testing: replace nose2 with pytest - run-tests Oguz Ozhan
2022-10-23 16:51 ` Yann E. MORIN
@ 2022-10-30 22:20 ` Ricardo Martincoski
1 sibling, 0 replies; 11+ messages in thread
From: Ricardo Martincoski @ 2022-10-30 22:20 UTC (permalink / raw)
To: oguz.ozhan; +Cc: buildroot
[-- Attachment #1: Type: text/plain, Size: 2142 bytes --]
Hello,
On Fri, Oct 21, 2022 at 06:15 AM, Oguz Ozhan wrote:
> - From web page of nose2:
> (https://docs.nose2.io/en/latest/)
> nose2 vs pytest:
> - pytest is an excellent test framework and we encourage users to consider it for new projects.
> - It has a bigger team of maintainers and a larger community of users.
>
> - pytest is more robust and has more ability compared to nose2
>
> - Changes in this patch:
> - run-tests modified to use pytest instead of nose2 to call tests. A corresponding Collector class added for pytest to list the available tests
> - nose2 plugins removed which are not required anymore
> - in test_iso9660.py, local test_ routine names changed to not be accepted as test_ cases via pytest
Please split the changes to the tests in a new patch (first in the series).
While doing that, please fix flake8 warnings.
>
> Signed-off-by: Oguz Ozhan <oguz.ozhan@mind.be>
> ---
> support/testing/conf/unittest.cfg | 3 --
> support/testing/run-tests | 28 ++++++++++-------
Please fix flake8 warnings for run-tests as well.
> support/testing/tests/fs/test_iso9660.py | 40 ++++++++++++------------
> 3 files changed, 36 insertions(+), 35 deletions(-)
Please update the manual, section 'Using the runtime tests framework', in the
same commit that changes run-tests to use pytest.
[snip]
> @@ -106,15 +110,15 @@ def main():
> return 1
> BRConfigTest.timeout_multiplier = args.timeout_multiplier
>
> - nose2_args = ["-v",
> - "-N", str(args.testcases),
> - "-s", test_dir,
> + pytest_args = ["--workers", str(args.testcases),
> + "--rootdir", test_dir,
> "-c", os.path.join(test_dir, "conf/unittest.cfg")]
run-tests -s is broken: it does not show the build and runtime console
>
> if args.testname:
> - nose2_args += args.testname
> + pytest_args += args.testname
> +
> + pytest.main(pytest_args)
>
> - nose2.discover(argv=nose2_args)
The detection of a failed test on GitLab CI is broken, even when tests fail
run-tests returns exit code 0.
Regards,
Ricardo
[-- Attachment #2: Type: text/plain, Size: 150 bytes --]
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Buildroot] [PATCH 3/3] support/testing: replace nose2 with pytest - CI
2022-10-21 9:15 ` [Buildroot] [PATCH 3/3] support/testing: replace nose2 with pytest - CI Oguz Ozhan
@ 2022-10-30 22:20 ` Ricardo Martincoski
0 siblings, 0 replies; 11+ messages in thread
From: Ricardo Martincoski @ 2022-10-30 22:20 UTC (permalink / raw)
To: oguz.ozhan; +Cc: buildroot
[-- Attachment #1: Type: text/plain, Size: 2282 bytes --]
Hello,
IMO, this change should be merged into patch 2.
On Fri, Oct 21, 2022 at 06:15 AM, Oguz Ozhan wrote:
> - From web page of nose2:
> (https://docs.nose2.io/en/latest/)
> nose2 vs pytest:
> - pytest is an excellent test framework and we encourage users to consider it for new projects.
> - It has a bigger team of maintainers and a larger community of users.
>
> - pytest is more robust and has more ability compared to nose2
>
> - Changes in this patch:
> - the sed command was fine filtering the tests list with nose2 output but removing all operating on pytest output. With pytest no fine filtering is required to get the test names in correct format.
>
> Signed-off-by: Oguz Ozhan <oguz.ozhan@mind.be>
> ---
> support/scripts/generate-gitlab-ci-yml | 2 --
> 1 file changed, 2 deletions(-)
>
> diff --git a/support/scripts/generate-gitlab-ci-yml b/support/scripts/generate-gitlab-ci-yml
> index aa43aac019..bb4dcbbb22 100755
> --- a/support/scripts/generate-gitlab-ci-yml
> +++ b/support/scripts/generate-gitlab-ci-yml
> @@ -31,7 +31,6 @@ gen_tests() {
> defconfigs=( $(cd configs; LC_ALL=C ls -1 *_defconfig) )
>
> runtimes=( $(./support/testing/run-tests -l 2>&1 \
> - | sed -r -e '/^test_run \((.*)\).*/!d; s//\1/' \
> | LC_ALL=C sort)
> )
>
> @@ -80,7 +79,6 @@ gen_tests() {
> ;;
> (*-tests.*)
This regexp must be adapted, something like this:
(*-support*)
pattern="${CI_COMMIT_REF_NAME/--/::}"
runtimes=( $(./support/testing/run-tests -l 2>&1 \
| LC_ALL=C sort \
| grep "^${pattern##*-}")
Notice that ':' is not accepted on git branches, so we cannot use this branch
name to run a single test on GitLab CI:
git push gitlab \
HEAD:foo-support/testing/tests/package/test_php_lua.py::TestPhpLuaLuajit
so I guess we could use a pattern substitution, for instance:
git push gitlab \
HEAD:foo-support/testing/tests/package/test_php_lua.py--TestPhpLuaLuajit
and document it in the manual.
The manual must also be updated.
docs/manual/contribute.txt: $ git push gitlab HEAD:foo-tests.init.test_busybox
docs/manual/contribute.txt: $ git push gitlab HEAD:foo-tests.init
Regards,
Ricardo
[-- Attachment #2: Type: text/plain, Size: 150 bytes --]
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2022-10-30 22:21 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-10-21 9:15 [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile Oguz Ozhan
2022-10-21 9:15 ` [Buildroot] [PATCH 2/3] support/testing: replace nose2 with pytest - run-tests Oguz Ozhan
2022-10-23 16:51 ` Yann E. MORIN
2022-10-30 22:20 ` Ricardo Martincoski
2022-10-21 9:15 ` [Buildroot] [PATCH 3/3] support/testing: replace nose2 with pytest - CI Oguz Ozhan
2022-10-30 22:20 ` Ricardo Martincoski
2022-10-23 16:44 ` [Buildroot] [PATCH 1/3] support/testing: replace nose2 with pytest - Dockerfile Yann E. MORIN
2022-10-30 22:17 ` Ricardo Martincoski
-- strict thread matches above, loose matches on Subject: below --
2022-10-19 21:57 Oguz Ozhan
2022-10-21 7:44 ` Thomas Petazzoni via buildroot
2022-10-21 9:18 ` Oguz Ozhan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.