[Buildroot] testing/infra: generating tests for python packages

[Yann E. MORIN <yann.morin.1998@free.fr>]

Ricardo, Thomas, All,

On 2017-09-13 00:29 -0300, Ricardo Martincoski spake thusly:
> Thomas mentioned [1] you guys talked about tests for python packages being
> generated by the test infra.
> 
> I performed an experiment [2] ignoring where the data is coming from (whether
> it comes from the package recipe or from inside the infra) and I start this
> thread to follow up the discussion.

I also performed an experiment, which failed because I am no python
expert, but basically it was something like (in pseudo code because I
lost it and it anyway did not work):

    support/testing/tests/package/test_python-modules.py

        for dir, files, _ = os.walk(os.path.join(buildroot_top_dir,'package'):
            pkg = os.path.last_component(dir)
            for f in files:
                if re.match('{}-test.py'.format(pkg),f):
                    import_module(os.path.join(dir,f))

which basically scans the Buildroot package/ subdir to search for files
matching the glob 'PKG_NAME-test.py' (with PKG_NAME replaced by the
package name, obviously), and imports them one by one.

In turn, each of those files would define standard test classes, for
example:

    package/python-cryptography/python-cryptography-test.py

        import os
        import tests.package.testpythonbase

        class TestPythonCryptography(tests.package.testpythonbase.BaseClass):
            config = "blabla"

            def test_run(self):
                emulator.exec("python -c 'import cryptography'")

Of course, one may be more inventive with the tests.

And in the end, it would have been all integrated in the current infra.

But alas, that import stuff I could not make to work... :-( Maybe there
is a node function to tell it where to load extra tests from?

An alternative would be to go with per-package test files, like
explained above, but have the support/testing/run-test script do the can
before calling nose, and for each file if finds, do a symlink ( or a
copy) in the testing infra, in a special directory that is git-ignored ?

*That* would be pretty trivial (even I could do it, that's saying
something!).

To be honest, I would highly prefer that we go this kind of route rather
than generate the tests from higher-level metadata. Especially, I'm not
fond of adding FOO_TEST_CMDS in the .mk files.

Now, completely unrelated to the above, when we want to test each module
to ensure they are correct and have the required dependencies, we need
to have a configuration for each of the modules, which means doing one
build per module we want to test, each build including building the
python interpreter. This will make for a very long test campaign,
indeed... :-/

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'