[Buildroot] [RFC 0/4] suppport/testing: Perl, Lua and their modules

[Buildroot] [RFC 0/4] suppport/testing: Perl, Lua and their modules

[Francois Perrad]

The testing infrastructure seems not yet documented.
So, I use the recent work on Python2/3 modules as example.

On my local machine, I run successfully:
	flake8
	run-tests --list
So, the Python syntax is correct.
But, I don't know how to run a test on Qemu.

As future work, I think that scancpan could generate test for Perl/CPAN packages.
Loading a pure Perl module has no real added value.
So, test will be generated only for XS modules.

luainterpreter is a virtual package, provided by lua or luajit.
lua comes with 3 versions (5.1, 5.2, 5.3), 5.3 is the BR default.
On 32 bits target, BR enables by default the "32 bits numbers" compilation option.
I think that most of upstream authors never test their modules with a "32 bits numbers" lua.
Currently, test_lua uses this worst config (Lua 5.3 "32 bits numbers").
What do you think about shuffle various Lua interpreters ?

Francois Perrad (4):
  support/testing: add perl test
  support/testing: add perl-time-hires test
  support/testing: add lua test
  support/testing: add lpeg test

 .gitlab-ci.yml                                |  4 +++
 support/testing/tests/package/test_lua.py     | 33 +++++++++++++++++++
 .../testing/tests/package/test_lua_lpeg.py    | 13 ++++++++
 support/testing/tests/package/test_perl.py    | 33 +++++++++++++++++++
 .../tests/package/test_perl_time_hires.py     | 12 +++++++
 5 files changed, 95 insertions(+)
 create mode 100644 support/testing/tests/package/test_lua.py
 create mode 100644 support/testing/tests/package/test_lua_lpeg.py
 create mode 100644 support/testing/tests/package/test_perl.py
 create mode 100644 support/testing/tests/package/test_perl_time_hires.py

-- 
2.17.1

[Buildroot] [RFC 1/4] support/testing: add perl test

[Francois Perrad]

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
---
 .gitlab-ci.yml                             |  1 +
 support/testing/tests/package/test_perl.py | 33 ++++++++++++++++++++++
 2 files changed, 34 insertions(+)
 create mode 100644 support/testing/tests/package/test_perl.py

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 0dfb3537c..ae5d57026 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -312,6 +312,7 @@ tests.init.test_systemd.TestInitSystemSystemdRwNetworkd: *runtime_test
 tests.package.test_dropbear.TestDropbear: *runtime_test
 tests.package.test_ipython.TestIPythonPy2: *runtime_test
 tests.package.test_ipython.TestIPythonPy3: *runtime_test
+tests.package.test_perl.TestPerl: *runtime_test
 tests.package.test_python.TestPython2: *runtime_test
 tests.package.test_python.TestPython3: *runtime_test
 tests.package.test_python_autobahn.TestPythonPy2Autobahn: *runtime_test
diff --git a/support/testing/tests/package/test_perl.py b/support/testing/tests/package/test_perl.py
new file mode 100644
index 000000000..4f6e69075
--- /dev/null
+++ b/support/testing/tests/package/test_perl.py
@@ -0,0 +1,33 @@
+import os
+
+import infra.basetest
+
+
+class TestPerl(infra.basetest.BRTest):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+        """
+        BR2_TARGET_ROOTFS_CPIO=y
+        # BR2_TARGET_ROOTFS_TAR is not set
+        BR2_PACKAGE_PERL=y
+        """
+
+    def login(self):
+        cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio")
+        self.emulator.boot(arch="armv5",
+                           kernel="builtin",
+                           options=["-initrd", cpio_file])
+        self.emulator.login()
+
+    def version_test(self, timeout=-1):
+        cmd = "perl -v"
+        _, exit_code = self.emulator.run(cmd, timeout)
+        self.assertEqual(exit_code, 0)
+
+    def module_test(self, module, script="1", timeout=-1):
+        cmd = "perl -M{} -e '{}'".format(module, script)
+        _, exit_code = self.emulator.run(cmd, timeout)
+        self.assertEqual(exit_code, 0)
+
+    def test_run(self):
+        self.login()
+        self.version_test()
-- 
2.17.1

[Buildroot] [RFC 1/4] support/testing: add perl test

[Ricardo Martincoski]

Hello,

In overall, the series looks good.
Except for the general discussion in the cover letter about which path we want
to follow (add the infra to add a .pl to the image in build time or not), I have
only nits.

On Sat, Nov 17, 2018 at 03:52 PM, Francois Perrad wrote:

[snip]
> +class TestPerl(infra.basetest.BRTest):
> +    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> +        """
> +        BR2_TARGET_ROOTFS_CPIO=y
> +        # BR2_TARGET_ROOTFS_TAR is not set
> +        BR2_PACKAGE_PERL=y

nit: this is not the order after 'make savedefconfig'

> +        """
> +
[snip]
> +    def version_test(self, timeout=-1):

This timeout is always used with its default value, you can remove from here and
from the call to emulator.run 2 lines below.

> +        cmd = "perl -v"

What about using this instead?
        cmd = "perl -v | grep 'This is perl 5'"
I guess Perl 6 will take a long time to be supported in Buildroot.

> +        _, exit_code = self.emulator.run(cmd, timeout)
> +        self.assertEqual(exit_code, 0)
> +
> +    def module_test(self, module, script="1", timeout=-1):

If we end up using the extra logic (basically copied from TestPythonPackageBase)
and having the .pl file added to the tree, this method would not be needed.


Regards,
Ricardo

[Buildroot] [RFC 1/4] support/testing: add perl test

[François Perrad]

Le lun. 19 nov. 2018 ? 02:11, Ricardo Martincoski <
ricardo.martincoski@gmail.com> a ?crit :

> Hello,
>
> In overall, the series looks good.
> Except for the general discussion in the cover letter about which path we
> want
> to follow (add the infra to add a .pl to the image in build time or not),
> I have
> only nits.
>
> On Sat, Nov 17, 2018 at 03:52 PM, Francois Perrad wrote:
>
> [snip]
> > +class TestPerl(infra.basetest.BRTest):
> > +    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> > +        """
> > +        BR2_TARGET_ROOTFS_CPIO=y
> > +        # BR2_TARGET_ROOTFS_TAR is not set
> > +        BR2_PACKAGE_PERL=y
>
> nit: this is not the order after 'make savedefconfig'
>
> > +        """
> > +
> [snip]
> > +    def version_test(self, timeout=-1):
>
> This timeout is always used with its default value, you can remove from
> here and
> from the call to emulator.run 2 lines below.
>
> > +        cmd = "perl -v"
>
> What about using this instead?
>         cmd = "perl -v | grep 'This is perl 5'"
>

As tester, I don't like this kind of code, because I never know which exit
code is available, the one from perl or the one from grep.
Code written for test must be obvious.
See the following example:

#!/usr/bin/env python
import unittest
import subprocess

class DeveloperWay(unittest.TestCase):
    def testPerl(self):
        retcode = subprocess.call("perl -v | grep 'This is perl 5'",
shell=True)
        self.assertEqual(retcode, 0)

class TesterWay(unittest.TestCase):
    def testPerl(self):
        retcode = subprocess.call('perl -v', shell=True)
        self.assertEqual(retcode, 0)
        output = subprocess.check_output('perl -v', shell=True)
        self.assert_('This is perl 5' in output)

unittest.main()

Fran?ois

I guess Perl 6 will take a long time to be supported in Buildroot.
>
> > +        _, exit_code = self.emulator.run(cmd, timeout)
> > +        self.assertEqual(exit_code, 0)
> > +
> > +    def module_test(self, module, script="1", timeout=-1):
>
> If we end up using the extra logic (basically copied from
> TestPythonPackageBase)
> and having the .pl file added to the tree, this method would not be needed.
>
>
> Regards,
> Ricardo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20181121/13a37a58/attachment.html>

[Buildroot] [RFC 1/4] support/testing: add perl test

[Ricardo Martincoski]

Hello,

On Wed, Nov 21, 2018 at 02:26 PM, Fran?ois Perrad wrote:

> Le lun. 19 nov. 2018 ? 02:11, Ricardo Martincoski <
> ricardo.martincoski at gmail.com> a ?crit :
[snip]
>> > +        cmd = "perl -v"
>>
>> What about using this instead?
>>         cmd = "perl -v | grep 'This is perl 5'"
>>
> 
> As tester, I don't like this kind of code, because I never know which exit
> code is available, the one from perl or the one from grep.
> Code written for test must be obvious.
> See the following example:
> 
> #!/usr/bin/env python
> import unittest
> import subprocess
> 
> class DeveloperWay(unittest.TestCase):
>     def testPerl(self):
>         retcode = subprocess.call("perl -v | grep 'This is perl 5'",
> shell=True)
>         self.assertEqual(retcode, 0)
> 
> class TesterWay(unittest.TestCase):
>     def testPerl(self):
>         retcode = subprocess.call('perl -v', shell=True)
>         self.assertEqual(retcode, 0)
>         output = subprocess.check_output('perl -v', shell=True)
>         self.assert_('This is perl 5' in output)
> 
> unittest.main()

Sure. Much better to have an error message that makes obvious what failed.


Regards,
Ricardo

[Buildroot] [RFC 2/4] support/testing: add perl-time-hires test

[Francois Perrad]

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
---
 .gitlab-ci.yml                                       |  1 +
 .../testing/tests/package/test_perl_time_hires.py    | 12 ++++++++++++
 2 files changed, 13 insertions(+)
 create mode 100644 support/testing/tests/package/test_perl_time_hires.py

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index ae5d57026..b3d5616f1 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -313,6 +313,7 @@ tests.package.test_dropbear.TestDropbear: *runtime_test
 tests.package.test_ipython.TestIPythonPy2: *runtime_test
 tests.package.test_ipython.TestIPythonPy3: *runtime_test
 tests.package.test_perl.TestPerl: *runtime_test
+tests.package.test_perl_time_hires.TestPerlTimeHiRes: *runtime_test
 tests.package.test_python.TestPython2: *runtime_test
 tests.package.test_python.TestPython3: *runtime_test
 tests.package.test_python_autobahn.TestPythonPy2Autobahn: *runtime_test
diff --git a/support/testing/tests/package/test_perl_time_hires.py b/support/testing/tests/package/test_perl_time_hires.py
new file mode 100644
index 000000000..2f7e1d119
--- /dev/null
+++ b/support/testing/tests/package/test_perl_time_hires.py
@@ -0,0 +1,12 @@
+from tests.package.test_perl import TestPerl
+
+
+class TestPerlTimeHiRes(TestPerl):
+    config = TestPerl.config + \
+        """
+        BR2_PACKAGE_PERL_TIME_HIRES=y
+        """
+
+    def test_run(self):
+        self.login()
+        self.module_test("Time::HiRes")
-- 
2.17.1

[Buildroot] [RFC 2/4] support/testing: add perl-time-hires test

[Ricardo Martincoski]

Hello,

Just giving an example of what I mentioned in a reply to the cover letter...

On Sat, Nov 17, 2018 at 03:52 PM, Francois Perrad wrote:

[snip]
> +class TestPerlTimeHiRes(TestPerl):
> +    config = TestPerl.config + \
> +        """
> +        BR2_PACKAGE_PERL_TIME_HIRES=y
> +        """
> +
> +    def test_run(self):
> +        self.login()
> +        self.module_test("Time::HiRes")

Another way to test a perl package would be to have an script added to the image
in build time, something like this:
sample_perl_time_hires.pl:
|use Time::HiRes;
|
|Time::HiRes::usleep(1_000_000)

And call:
perl sample_perl_time_hires.pl

This way we could test scripts with a few lines of code and keep it readable in
the tree.


Regards,
Ricardo

[Buildroot] [RFC 2/4] support/testing: add perl-time-hires test

[François Perrad]

Le lun. 19 nov. 2018 ? 02:14, Ricardo Martincoski <
ricardo.martincoski@gmail.com> a ?crit :

> Hello,
>
> Just giving an example of what I mentioned in a reply to the cover
> letter...
>
> On Sat, Nov 17, 2018 at 03:52 PM, Francois Perrad wrote:
>
> [snip]
> > +class TestPerlTimeHiRes(TestPerl):
> > +    config = TestPerl.config + \
> > +        """
> > +        BR2_PACKAGE_PERL_TIME_HIRES=y
> > +        """
> > +
> > +    def test_run(self):
> > +        self.login()
> > +        self.module_test("Time::HiRes")
>
> Another way to test a perl package would be to have an script added to the
> image
> in build time, something like this:
> sample_perl_time_hires.pl:
> |use Time::HiRes;
> |
> |Time::HiRes::usleep(1_000_000)
>
> And call:
> perl sample_perl_time_hires.pl
>
> This way we could test scripts with a few lines of code and keep it
> readable in
> the tree.
>
>
With the current TestPerl class, this test could be written as:
        self.module_test("Time::HiRes", "Time::HiRes::usleep(1_000_000);")
A long oneliner script could be written with a multiline layout, like this:
        self.module_test("Time::HiRes", "use strict;" +
                                        "use warnings;" +
                                        "Time::HiRes::usleep(1_000_000);")
So, we don't need to handle sample_perl_*.pl scripts.
Same thing with Lua.

Fran?ois


>
> Regards,
> Ricardo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20181119/cf8991c4/attachment.html>

[Buildroot] [RFC 2/4] support/testing: add perl-time-hires test

[Ricardo Martincoski]

Hello,

On Mon, Nov 19, 2018 at 05:29 PM, Fran?ois Perrad wrote:

> Le lun. 19 nov. 2018 ? 02:14, Ricardo Martincoski <
> ricardo.martincoski at gmail.com> a ?crit :
[snip]
>> Another way to test a perl package would be to have an script added to the
>> image
>> in build time, something like this:
>> sample_perl_time_hires.pl:
>> |use Time::HiRes;
>> |
>> |Time::HiRes::usleep(1_000_000)
>>
>> And call:
>> perl sample_perl_time_hires.pl
>>
>> This way we could test scripts with a few lines of code and keep it
>> readable in
>> the tree.
>>
>>
> With the current TestPerl class, this test could be written as:
>         self.module_test("Time::HiRes", "Time::HiRes::usleep(1_000_000);")
> A long oneliner script could be written with a multiline layout, like this:
>         self.module_test("Time::HiRes", "use strict;" +
>                                         "use warnings;" +
>                                         "Time::HiRes::usleep(1_000_000);")
> So, we don't need to handle sample_perl_*.pl scripts.
> Same thing with Lua.

This is (kind of) how we were handling the tests for python packages before.

But you have a point.

Testing python packages got really messy when a class was needed in the script
because of the indentation. In those cases a oneliner was not enough.
Lua and Perl packages can always be tested with oneliners AFAIK.

Another issue that made python scripts less readable was the escaping for string
delimiters.
I guess that for lua and perl scripts we can always use those fancy quotes
([[string]], q{string}) when needed.

If Thomas and Arnout don't disagree, let's keep using oneliners to test lua and
perl packages, as you originally proposed.


Regards,
Ricardo

[Buildroot] [RFC 3/4] support/testing: add lua test

[Francois Perrad]

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
---
 .gitlab-ci.yml                            |  1 +
 support/testing/tests/package/test_lua.py | 33 +++++++++++++++++++++++
 2 files changed, 34 insertions(+)
 create mode 100644 support/testing/tests/package/test_lua.py

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index b3d5616f1..660199e8c 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -312,6 +312,7 @@ tests.init.test_systemd.TestInitSystemSystemdRwNetworkd: *runtime_test
 tests.package.test_dropbear.TestDropbear: *runtime_test
 tests.package.test_ipython.TestIPythonPy2: *runtime_test
 tests.package.test_ipython.TestIPythonPy3: *runtime_test
+tests.package.test_lua.TestLua: *runtime_test
 tests.package.test_perl.TestPerl: *runtime_test
 tests.package.test_perl_time_hires.TestPerlTimeHiRes: *runtime_test
 tests.package.test_python.TestPython2: *runtime_test
diff --git a/support/testing/tests/package/test_lua.py b/support/testing/tests/package/test_lua.py
new file mode 100644
index 000000000..3e5e6779d
--- /dev/null
+++ b/support/testing/tests/package/test_lua.py
@@ -0,0 +1,33 @@
+import os
+
+import infra.basetest
+
+
+class TestLua(infra.basetest.BRTest):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+        """
+        BR2_TARGET_ROOTFS_CPIO=y
+        # BR2_TARGET_ROOTFS_TAR is not set
+        BR2_PACKAGE_LUA=y
+        """
+
+    def login(self):
+        cpio_file = os.path.join(self.builddir, "images", "rootfs.cpio")
+        self.emulator.boot(arch="armv5",
+                           kernel="builtin",
+                           options=["-initrd", cpio_file])
+        self.emulator.login()
+
+    def version_test(self, timeout=-1):
+        cmd = "lua -v"
+        _, exit_code = self.emulator.run(cmd, timeout)
+        self.assertEqual(exit_code, 0)
+
+    def module_test(self, module, script="a=1", timeout=-1):
+        cmd = "lua -l {} -e '{}'".format(module, script)
+        _, exit_code = self.emulator.run(cmd, timeout)
+        self.assertEqual(exit_code, 0)
+
+    def test_run(self):
+        self.login()
+        self.version_test()
-- 
2.17.1

[Buildroot] [RFC 4/4] support/testing: add lpeg test

[Francois Perrad]

Signed-off-by: Francois Perrad <francois.perrad@gadz.org>
---
 .gitlab-ci.yml                                 |  1 +
 support/testing/tests/package/test_lua_lpeg.py | 13 +++++++++++++
 2 files changed, 14 insertions(+)
 create mode 100644 support/testing/tests/package/test_lua_lpeg.py

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 660199e8c..625f486e9 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -313,6 +313,7 @@ tests.package.test_dropbear.TestDropbear: *runtime_test
 tests.package.test_ipython.TestIPythonPy2: *runtime_test
 tests.package.test_ipython.TestIPythonPy3: *runtime_test
 tests.package.test_lua.TestLua: *runtime_test
+tests.package.test_lua_lpeg.TestLuaLpeg: *runtime_test
 tests.package.test_perl.TestPerl: *runtime_test
 tests.package.test_perl_time_hires.TestPerlTimeHiRes: *runtime_test
 tests.package.test_python.TestPython2: *runtime_test
diff --git a/support/testing/tests/package/test_lua_lpeg.py b/support/testing/tests/package/test_lua_lpeg.py
new file mode 100644
index 000000000..5e1d9efeb
--- /dev/null
+++ b/support/testing/tests/package/test_lua_lpeg.py
@@ -0,0 +1,13 @@
+from tests.package.test_lua import TestLua
+
+
+class TestLuaLpeg(TestLua):
+    config = TestLua.config + \
+        """
+        BR2_PACKAGE_LPEG=y
+        """
+
+    def test_run(self):
+        self.login()
+        self.module_test("lpeg")
+        self.module_test("lpeg", "print(require[[lpeg]].version())")
-- 
2.17.1

[Buildroot] [RFC 0/4] suppport/testing: Perl, Lua and their modules

[Ricardo Martincoski]

Hello,

+ Arnout
+ Thomas
+ Yann

On Sat, Nov 17, 2018 at 03:52 PM, Francois Perrad wrote:

> The testing infrastructure seems not yet documented.

Not yet. I know Yann has some work in progress.

> So, I use the recent work on Python2/3 modules as example.
> 
> On my local machine, I run successfully:
> 	flake8
> 	run-tests --list
> So, the Python syntax is correct.

And they run well in the GitLab CI:
https://gitlab.com/RicardoMartincoski/buildroot/pipelines/37026720

> But, I don't know how to run a test on Qemu.

You can run it locally:

$ support/testing/run-tests -o test-output tests.package.test_lua

Hint 1: use -k when developing tests, specially the commands that will be issued
to the target. This option keeps any successful build and just execute the
commands.

Hint 2: -s is also very helpful when developing test cases, sending all the logs
to stdout.

Hint 3: the first line of the -run.log contains the command that the infra calls
to start qemu. You can run the same command and enter qemu.

> 
> As future work, I think that scancpan could generate test for Perl/CPAN packages.

Nice.

> Loading a pure Perl module has no real added value.
> So, test will be generated only for XS modules.
> 
> luainterpreter is a virtual package, provided by lua or luajit.
> lua comes with 3 versions (5.1, 5.2, 5.3), 5.3 is the BR default.

I would create a TestLuaPackageBase class that does the same of
TestPythonPackageBase, adding a script in build time to the image, and calling
it in runtime. The script is then stored in the tree in a separate .lua file.
And with this, we can also easily create 4 test cases for each lua package, one
for each version.
Arnout,
Thomas,
Do you think it is too much?
Notice the 4 test cases added in this series take respectively (perl) 9, 12,
(lua) 4 and 4 minutes to run.

> On 32 bits target, BR enables by default the "32 bits numbers" compilation option.
> I think that most of upstream authors never test their modules with a "32 bits numbers" lua.
> Currently, test_lua uses this worst config (Lua 5.3 "32 bits numbers").
> What do you think about shuffle various Lua interpreters ?

I am not sure if I understood, sorry.
Do you mean who is writing the test case randomly chooses a version and creates
a test case for it?
Or do you mean to have a code in the test infra that randomly selects a version
when testing?

The later is not good, IMO. The results would not be reproducible.
I would prefer to add one test case for each one of them.
And if we think it is too much (I don't, it seems to me these test cases would
run in few minutes), we can choose a subset, perhaps 2 of the 4 variants per lua
package.


Regards,
Ricardo