[Buildroot] [PATCH 0/3] support/testing: misc improvements (branch yem/runtime-test-ppd)

[Buildroot] [PATCH 0/3] support/testing: misc improvements (branch yem/runtime-test-ppd)

[Yann E. MORIN]

Hello All!

This small three-patch series brings in two improvements to the runtime
testing infrastructure:
  - enable top-level parallel build (TLPB) for tests that have
    per-package directories (PPD) enabled;
  - display the failed commands and its output on assertRunOk().

Finally, it enables PPD, and thus TLPB, for systemd test cases.

Regards,
Yann E. MORIN.


----------------------------------------------------------------
Yann E. MORIN (3):
      support/tests: print failed command and output on assertRunOK error
      support/tests: allow top-level parallel builds
      support/tests: enable PPD, and thus TLPB, for systemd tests

 support/testing/infra/basetest.py          | 13 +++++++++----
 support/testing/infra/builder.py           |  5 ++++-
 support/testing/tests/init/test_systemd.py |  1 +
 3 files changed, 14 insertions(+), 5 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 1/3] support/tests: print failed command and output on assertRunOK error

[Yann E. MORIN]

Currently, when asserting that a command succeeded, we just capture the
return code of the command. If that is not zero, the assertion fails,
but the error message is not very splicit:
    AssertionError: 1 != 0

Replace the error message with an explicit message that dumps the failed
command, the error code, and the resulting output.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/testing/infra/basetest.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/support/testing/infra/basetest.py b/support/testing/infra/basetest.py
index 96c6848dfc..45bcd4c2e2 100644
--- a/support/testing/infra/basetest.py
+++ b/support/testing/infra/basetest.py
@@ -88,7 +88,12 @@ class BRTest(BRConfigTest):
         super(BRTest, self).tearDown()
 
     # Run the given 'cmd' with a 'timeout' on the target and
-    # assert that the command succeeded
+    # assert that the command succeeded; on error, print the
+    # faulty command and its output
     def assertRunOk(self, cmd, timeout=-1):
-        _, exit_code = self.emulator.run(cmd, timeout)
-        self.assertEqual(exit_code, 0)
+        out, exit_code = self.emulator.run(cmd, timeout)
+        self.assertEqual(
+            exit_code,
+            0,
+            "\nFailed to run: {}\noutput was:\n{}".format(cmd, '  '+'\n  '.join(out))
+        )
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds

[Yann E. MORIN]

Running tests with top-level parallel builds can speed up running some
tests, expecially those that have a lot of packages like the systemd
init tests.

Trigger TLPB when the configuration enables per-package directories.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/testing/infra/basetest.py | 2 +-
 support/testing/infra/builder.py  | 5 ++++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/support/testing/infra/basetest.py b/support/testing/infra/basetest.py
index 45bcd4c2e2..670c7213d6 100644
--- a/support/testing/infra/basetest.py
+++ b/support/testing/infra/basetest.py
@@ -52,7 +52,7 @@ class BRConfigTest(unittest.TestCase):
 
     def setUp(self):
         self.show_msg("Starting")
-        self.b = Builder(self.config, self.builddir, self.logtofile)
+        self.b = Builder(self.config, self.builddir, self.logtofile, self.jlevel)
 
         if not self.keepbuilds:
             self.b.delete()
diff --git a/support/testing/infra/builder.py b/support/testing/infra/builder.py
index 922a707220..a2abb9ed89 100644
--- a/support/testing/infra/builder.py
+++ b/support/testing/infra/builder.py
@@ -6,11 +6,12 @@ import infra
 
 
 class Builder(object):
-    def __init__(self, config, builddir, logtofile):
+    def __init__(self, config, builddir, logtofile, jlevel=None):
         self.config = '\n'.join([line.lstrip() for line in
                                  config.splitlines()]) + '\n'
         self.builddir = builddir
         self.logfile = infra.open_log_file(builddir, "build", logtofile)
+        self.jlevel = jlevel
 
     def is_defconfig_valid(self, configfile, defconfig):
         """Check if the .config is contains all lines present in the defconfig."""
@@ -87,6 +88,8 @@ class Builder(object):
         env.update(make_extra_env)
 
         cmd = ["make", "-C", self.builddir]
+        if "BR2_PER_PACKAGE_DIRECTORIES=y" in self.config.splitlines() and self.jlevel:
+            cmd.append(f"-j{self.jlevel}")
         cmd += make_extra_opts
 
         ret = subprocess.call(cmd, stdout=self.logfile, stderr=self.logfile,
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 3/3] support/tests: enable PPD, and thus TLPB, for systemd tests

[Yann E. MORIN]

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
---
 support/testing/tests/init/test_systemd.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/support/testing/tests/init/test_systemd.py b/support/testing/tests/init/test_systemd.py
index 06c42736ae..ddc32b0838 100644
--- a/support/testing/tests/init/test_systemd.py
+++ b/support/testing/tests/init/test_systemd.py
@@ -13,6 +13,7 @@ class InitSystemSystemdBase(InitSystemBase):
         BR2_INIT_SYSTEMD=y
         BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0"
         # BR2_TARGET_ROOTFS_TAR is not set
+        BR2_PER_PACKAGE_DIRECTORIES=y
         """
 
     def check_systemd(self, fs):
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/3] support/tests: print failed command and output on assertRunOK error

[Thomas Petazzoni via buildroot]

On Sat, 24 Dec 2022 10:18:11 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Currently, when asserting that a command succeeded, we just capture the
> return code of the command. If that is not zero, the assertion fails,
> but the error message is not very splicit:
>     AssertionError: 1 != 0
> 
> Replace the error message with an explicit message that dumps the failed
> command, the error code, and the resulting output.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  support/testing/infra/basetest.py | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds

[Thomas Petazzoni via buildroot]

On Sat, 24 Dec 2022 10:18:12 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Running tests with top-level parallel builds can speed up running some
> tests, expecially those that have a lot of packages like the systemd
> init tests.
> 
> Trigger TLPB when the configuration enables per-package directories.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

I certainly support the idea of running some tests with TLPB. However,
this implementation makes a confusion between two different settings:

- The existing BRConfigTest.jlevel, which is set by run-tests -j, and
  used to define BR2_JLEVEL in the Buildroot configuration of each test
  case. This determines the number of parallel jobs used to build each
  independent package.

- Your new work, which uses "make -j" to do TLPB... but relies on the
  same above value, even though it's a completely different setting.

Is this expected?

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds

[Yann E. MORIN]

Thomas, All,

On 2022-12-27 21:45 +0100, Thomas Petazzoni via buildroot spake thusly:
> On Sat, 24 Dec 2022 10:18:12 +0100
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> > Running tests with top-level parallel builds can speed up running some
> > tests, expecially those that have a lot of packages like the systemd
> > init tests.
> > 
> > Trigger TLPB when the configuration enables per-package directories.
> > 
> > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> 
> I certainly support the idea of running some tests with TLPB. However,
> this implementation makes a confusion between two different settings:
> 
> - The existing BRConfigTest.jlevel, which is set by run-tests -j, and
>   used to define BR2_JLEVEL in the Buildroot configuration of each test
>   case. This determines the number of parallel jobs used to build each
>   independent package.
> 
> - Your new work, which uses "make -j" to do TLPB... but relies on the
>   same above value, even though it's a completely different setting.
> 
> Is this expected?

Yes, this is the intended behaviour, which I was explicitly seeking.

So, if one runs with PPD and TLPB (outside the run-time infra), one
would do something like:

    $ make -jN

This spawns a top-level make process that is parallel. In turn, in
rules, when we call $(MAKE), this is the magic that tells make that
it is recursive, but that it should use the jobserver from the calling
process.

So, in this case, the BR2_JLEVEL is unused by whatever uses the make
jobserver; only the number of jobs in the top-level jobserver is
meaningful, i.e. whatever we pass as -jN.

The exception, of course, is whatever uses BR2_JLEVEL but does not talk
to the jobserver, but this is mostly a few packages (scons, waf et al.).
Even ninja packages do talk to the top-level jobserver, now that we use
the ninja fork that knows to talk to it.

So, yes, using top-level -jN with the same value as BR2_JLEVEL is
exactly what I intended to do.

Unless I totally missed something...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds

[Yann E. MORIN]

Thomas, All,

On 2022-12-27 21:53 +0100, Yann E. MORIN spake thusly:
> On 2022-12-27 21:45 +0100, Thomas Petazzoni via buildroot spake thusly:
> > On Sat, 24 Dec 2022 10:18:12 +0100
> > "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> > > Running tests with top-level parallel builds can speed up running some
> > > tests, expecially those that have a lot of packages like the systemd
> > > init tests.
> > > 
> > > Trigger TLPB when the configuration enables per-package directories.
> > > 
> > > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> > > Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> > 
> > I certainly support the idea of running some tests with TLPB. However,
> > this implementation makes a confusion between two different settings:
> > 
> > - The existing BRConfigTest.jlevel, which is set by run-tests -j, and
> >   used to define BR2_JLEVEL in the Buildroot configuration of each test
> >   case. This determines the number of parallel jobs used to build each
> >   independent package.
> > 
> > - Your new work, which uses "make -j" to do TLPB... but relies on the
> >   same above value, even though it's a completely different setting.
> > 
> > Is this expected?
> 
> Yes, this is the intended behaviour, which I was explicitly seeking.
> 
> So, if one runs with PPD and TLPB (outside the run-time infra), one
> would do something like:
> 
>     $ make -jN
> 
> This spawns a top-level make process that is parallel. In turn, in
> rules, when we call $(MAKE), this is the magic that tells make that
> it is recursive, but that it should use the jobserver from the calling
> process.
> 
> So, in this case, the BR2_JLEVEL is unused by whatever uses the make
> jobserver; only the number of jobs in the top-level jobserver is
> meaningful, i.e. whatever we pass as -jN.

Slight correction: BR2_JLEVEL does have an actual effect, but only if its
value is lower than the one we pass as -jN.

So, in practice, we want to have BR2_JLEVEL equal to the top-level -jN.

Regards,
Yann E. MORIN.

> The exception, of course, is whatever uses BR2_JLEVEL but does not talk
> to the jobserver, but this is mostly a few packages (scons, waf et al.).
> Even ninja packages do talk to the top-level jobserver, now that we use
> the ninja fork that knows to talk to it.
> 
> So, yes, using top-level -jN with the same value as BR2_JLEVEL is
> exactly what I intended to do.
> 
> Unless I totally missed something...
> 
> Regards,
> Yann E. MORIN.
> 
> -- 
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/3] support/tests: allow top-level parallel builds

[Thomas Petazzoni via buildroot]

On Sat, 24 Dec 2022 10:18:12 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Running tests with top-level parallel builds can speed up running some
> tests, expecially those that have a lot of packages like the systemd
> init tests.
> 
> Trigger TLPB when the configuration enables per-package directories.
> 
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  support/testing/infra/basetest.py | 2 +-
>  support/testing/infra/builder.py  | 5 ++++-
>  2 files changed, 5 insertions(+), 2 deletions(-)

Thanks to the additional explanation you provided, I understood better
the idea, and therefore applied the patch. I updated the commit log a
little bit to summarize the explanation you gave, and applied. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 3/3] support/tests: enable PPD, and thus TLPB, for systemd tests

[Thomas Petazzoni via buildroot]

On Sat, 24 Dec 2022 10:18:13 +0100
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> ---
>  support/testing/tests/init/test_systemd.py | 1 +
>  1 file changed, 1 insertion(+)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot