[Buildroot] testing/infra: generating tests for python packages

[Buildroot] testing/infra: generating tests for python packages

[Ricardo Martincoski]

Yann,

Thomas mentioned [1] you guys talked about tests for python packages being
generated by the test infra.

I performed an experiment [2] ignoring where the data is coming from (whether
it comes from the package recipe or from inside the infra) and I start this
thread to follow up the discussion.

It consist of a simple templating scheme using sed (of course it could be done
by some fancy templating like pystache, jinja, ... but I like sed) and
inheritance for a generic method that tests a command, and it tries to generate
about the same test cases Yegor created [3].

We cannot use TestPython{2,3} classes as-is since nose2 (or one of its
dependencies) would call its test_run method twice. I did not investigate the
reason, just worked around it for now.

Together to testing for exit code 0 to determine Pass/Fail, I added regexes for
the output of the command, maybe it can become handy (for example, hashlib was
just added as dependency to python-cryptography with Python 2 but when it was
missing the test did not fail since the exit code was still 0).

Since .gitlab-ci.yml cannot yet be dynamically generated (see [4] from Arnout
and its linked-list of similar issues reported) and we want to keep separate
results to each test, it looks to me a good place to call the test case
generation would be just before the handling of --list option in run-tests.
This way 'make .gitlab-ci.yml' and 'run-tests --list' always keep working as
expected and also the test case is correctly generated in the docker runner.

It seems to me that besides the package + the test command, some interesting
variables to generate the test cases are:
- if the package supports Python 2, 3 or both;
- if testing for exit code 0 is enough, and the regex to determine pass/fail
  when appropriate;
- the timeout to the command;
- the non-python dependencies of the package (python-slob needs gcc 4.9 so as a
  dirty hack I changed the base config, of course we could try to find a good
  default for all python packages and use it in the TestPython2Package class);

Of course we can draw a line and choose to cover most of, but not all, python
packages and avoid or replace some of these variables by good defaults.

Now concerning the choice between keeping the data in the package recipe or in
the test infra I don't have strong opinion. It looks nice to have it in the
package recipe, but maybe it could be too complicated or even not feasible
using make targets due to the Python2/Python3/both variable. Perhaps externally
parsing package/python-*/{Config.in,*.mk} could do the job.

[1] http://lists.busybox.net/pipermail/buildroot/2017-September/201766.html
[2] https://gitlab.com/RicardoMartincoski/buildroot/commit/41da07c352dc3ad45af5a447cbabf8bef1245f98
[3] http://patchwork.ozlabs.org/patch/811011/
[4] https://gitlab.com/gitlab-org/gitlab-ce/issues/34595

Regards,
Ricardo

[Buildroot] testing/infra: generating tests for python packages

[Yann E. MORIN]

Ricardo, Thomas, All,

On 2017-09-13 00:29 -0300, Ricardo Martincoski spake thusly:
> Thomas mentioned [1] you guys talked about tests for python packages being
> generated by the test infra.
> 
> I performed an experiment [2] ignoring where the data is coming from (whether
> it comes from the package recipe or from inside the infra) and I start this
> thread to follow up the discussion.

I also performed an experiment, which failed because I am no python
expert, but basically it was something like (in pseudo code because I
lost it and it anyway did not work):

    support/testing/tests/package/test_python-modules.py

        for dir, files, _ = os.walk(os.path.join(buildroot_top_dir,'package'):
            pkg = os.path.last_component(dir)
            for f in files:
                if re.match('{}-test.py'.format(pkg),f):
                    import_module(os.path.join(dir,f))

which basically scans the Buildroot package/ subdir to search for files
matching the glob 'PKG_NAME-test.py' (with PKG_NAME replaced by the
package name, obviously), and imports them one by one.

In turn, each of those files would define standard test classes, for
example:

    package/python-cryptography/python-cryptography-test.py

        import os
        import tests.package.testpythonbase

        class TestPythonCryptography(tests.package.testpythonbase.BaseClass):
            config = "blabla"

            def test_run(self):
                emulator.exec("python -c 'import cryptography'")

Of course, one may be more inventive with the tests.

And in the end, it would have been all integrated in the current infra.

But alas, that import stuff I could not make to work... :-( Maybe there
is a node function to tell it where to load extra tests from?

An alternative would be to go with per-package test files, like
explained above, but have the support/testing/run-test script do the can
before calling nose, and for each file if finds, do a symlink ( or a
copy) in the testing infra, in a special directory that is git-ignored ?

*That* would be pretty trivial (even I could do it, that's saying
something!).

To be honest, I would highly prefer that we go this kind of route rather
than generate the tests from higher-level metadata. Especially, I'm not
fond of adding FOO_TEST_CMDS in the .mk files.

Now, completely unrelated to the above, when we want to test each module
to ensure they are correct and have the required dependencies, we need
to have a configuration for each of the modules, which means doing one
build per module we want to test, each build including building the
python interpreter. This will make for a very long test campaign,
indeed... :-/

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] testing/infra: generating tests for python packages

[Arnout Vandecappelle]

On 13-09-17 18:41, Yann E. MORIN wrote:
> Ricardo, Thomas, All,
> 
> On 2017-09-13 00:29 -0300, Ricardo Martincoski spake thusly:
>> Thomas mentioned [1] you guys talked about tests for python packages being
>> generated by the test infra.
>>
>> I performed an experiment [2] ignoring where the data is coming from (whether
>> it comes from the package recipe or from inside the infra) and I start this
>> thread to follow up the discussion.

 I didn't look at your solution yet, Ricardo, but here are my high-level
observations.


> I also performed an experiment, which failed because I am no python
> expert, but basically it was something like (in pseudo code because I
> lost it and it anyway did not work):
> 
>     support/testing/tests/package/test_python-modules.py
> 
>         for dir, files, _ = os.walk(os.path.join(buildroot_top_dir,'package'):
>             pkg = os.path.last_component(dir)
>             for f in files:
>                 if re.match('{}-test.py'.format(pkg),f):
>                     import_module(os.path.join(dir,f))
> 
> which basically scans the Buildroot package/ subdir to search for files
> matching the glob 'PKG_NAME-test.py' (with PKG_NAME replaced by the
> package name, obviously), and imports them one by one.

 Looks like the right thing to do to me. However, to be consistent with the
existing tests, the pattern should be test_pkg.py, and the package name should
go through - to _ conversion.

> In turn, each of those files would define standard test classes, for
> example:
> 
>     package/python-cryptography/python-cryptography-test.py
> 
>         import os
>         import tests.package.testpythonbase
> 
>         class TestPythonCryptography(tests.package.testpythonbase.BaseClass):
>             config = "blabla"
> 
>             def test_run(self):
>                 emulator.exec("python -c 'import cryptography'")

 Yes, we want to be explicit, so this looks good. Only in the config part we
could perhaps do some automagic, i.e. automagically selecting python and the
package itself.


> Of course, one may be more inventive with the tests.
> 
> And in the end, it would have been all integrated in the current infra.
> 
> But alas, that import stuff I could not make to work... :-( Maybe there
> is a node function to tell it where to load extra tests from?

 We can just add support/testing to sys.path.

> 
> An alternative would be to go with per-package test files, like
> explained above, but have the support/testing/run-test script do the can
> before calling nose, and for each file if finds, do a symlink ( or a
> copy) in the testing infra, in a special directory that is git-ignored ?

 Bwerk, hack...

> 
> *That* would be pretty trivial (even I could do it, that's saying
> something!).
> 
> To be honest, I would highly prefer that we go this kind of route rather
> than generate the tests from higher-level metadata. Especially, I'm not
> fond of adding FOO_TEST_CMDS in the .mk files.

 Absolutely. We certainly want to be able to do more things than a simple import
so we want a full .py file. And also we want to be explicit about things.


> Now, completely unrelated to the above, when we want to test each module
> to ensure they are correct and have the required dependencies, we need
> to have a configuration for each of the modules, which means doing one
> build per module we want to test, each build including building the
> python interpreter. This will make for a very long test campaign,
> indeed... :-/

 We don't care that it takes a long time. It's not intended to be run
sequentially anyway. And in gitlab, all tests run in parallel (but only on 4
builders IIRC). As long as the tests on gitlab finish within a day or so, we're
golden.

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] testing/infra: generating tests for python packages

[Yann E. MORIN]

Arnout, All,

On 2017-09-13 23:04 +0200, Arnout Vandecappelle spake thusly:
> 
> 
> On 13-09-17 18:41, Yann E. MORIN wrote:
> > Ricardo, Thomas, All,
> > 
> > On 2017-09-13 00:29 -0300, Ricardo Martincoski spake thusly:
> >> Thomas mentioned [1] you guys talked about tests for python packages being
> >> generated by the test infra.
> >>
> >> I performed an experiment [2] ignoring where the data is coming from (whether
> >> it comes from the package recipe or from inside the infra) and I start this
> >> thread to follow up the discussion.
> 
>  I didn't look at your solution yet, Ricardo, but here are my high-level
> observations.
> 
> > I also performed an experiment, which failed because I am no python
> > expert, but basically it was something like (in pseudo code because I
> > lost it and it anyway did not work):
> > 
> >     support/testing/tests/package/test_python-modules.py
> > 
> >         for dir, files, _ = os.walk(os.path.join(buildroot_top_dir,'package'):
> >             pkg = os.path.last_component(dir)
> >             for f in files:
> >                 if re.match('{}-test.py'.format(pkg),f):
> >                     import_module(os.path.join(dir,f))
> > 
> > which basically scans the Buildroot package/ subdir to search for files
> > matching the glob 'PKG_NAME-test.py' (with PKG_NAME replaced by the
> > package name, obviously), and imports them one by one.
> 
>  Looks like the right thing to do to me. However, to be consistent with the
> existing tests, the pattern should be test_pkg.py, and the package name should
> go through - to _ conversion.

And to be consistent with the existing .mk and .hash files, the test
file should start with the package name. So, we have two incompatible
consistency requirements. One will have to win and the other to lose.
And I think the best would be to keep the .mk and .hash scheme, because
this is in the package directory.

> > Of course, one may be more inventive with the tests.
> > 
> > And in the end, it would have been all integrated in the current infra.
> > 
> > But alas, that import stuff I could not make to work... :-( Maybe there
> > is a node function to tell it where to load extra tests from?
> 
>  We can just add support/testing to sys.path.

Except that did not work when I tried... :-/

But I also tried setting package/ in sys.path and it did not work
either... Meh, someone will have to teach me some python magic... ;-)

> > An alternative would be to go with per-package test files, like
> > explained above, but have the support/testing/run-test script do the can
> > before calling nose, and for each file if finds, do a symlink ( or a
> > copy) in the testing infra, in a special directory that is git-ignored ?
> 
>  Bwerk, hack...

Maybe, but very, very easy and trivial!

We can even keep the PKG_NAME-test.py (or PKG_NAME.test) and symlink
that as test_PKG_NAME.py to respect the nose naming scheme.

> > Now, completely unrelated to the above, when we want to test each module
> > to ensure they are correct and have the required dependencies, we need
> > to have a configuration for each of the modules, which means doing one
> > build per module we want to test, each build including building the
> > python interpreter. This will make for a very long test campaign,
> > indeed... :-/
> 
>  We don't care that it takes a long time. It's not intended to be run
> sequentially anyway. And in gitlab, all tests run in parallel (but only on 4
> builders IIRC). As long as the tests on gitlab finish within a day or so, we're
> golden.

Yep, but we currently have about 224 python packages, so if each needs
30 minutes to build and run, that's about 5 days to run the full
suite... Just sayin'...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] testing/infra: generating tests for python packages

[Ricardo Martincoski]

Hello,

On Wed, Sep 13, 2017 at 06:31 PM, Yann E. MORIN wrote:

> On 2017-09-13 23:04 +0200, Arnout Vandecappelle spake thusly:
>> 
>> On 13-09-17 18:41, Yann E. MORIN wrote:
>> > 
>> > On 2017-09-13 00:29 -0300, Ricardo Martincoski spake thusly:
>> >> Thomas mentioned [1] you guys talked about tests for python packages being
>> >> generated by the test infra.
>> >>
>> >> I performed an experiment [2] ignoring where the data is coming from (whether
>> >> it comes from the package recipe or from inside the infra) and I start this
>> >> thread to follow up the discussion.
>> 
>>  I didn't look at your solution yet, Ricardo, but here are my high-level
>> observations.

Not really a solution. It was just an experiment/hack so don't bother to look.

The solution discussed in this thread looks much better. It gives the developer
of the test all the freedom regarding all the variables I mentioned (timeout,
regex, python2/python3/both, gcc version, ...).

>> 
>> > I also performed an experiment, which failed because I am no python
>> > expert, but basically it was something like (in pseudo code because I
>> > lost it and it anyway did not work):
>> > 
>> >     support/testing/tests/package/test_python-modules.py
>> > 
>> >         for dir, files, _ = os.walk(os.path.join(buildroot_top_dir,'package'):
>> >             pkg = os.path.last_component(dir)
>> >             for f in files:
>> >                 if re.match('{}-test.py'.format(pkg),f):
>> >                     import_module(os.path.join(dir,f))
>> > 
>> > which basically scans the Buildroot package/ subdir to search for files

So you were not only trying to create a solution for python packages but one to
any package that needs runtime tests. Nice! So eventually test_dropbear could
be moved to package/dropbear.

And as a consequence, a patch adding a new package for which having a runtime
test makes sense would look like this:

package/<package>/Config.in ++++++
package/<package>/<package>.mk ++++++
package/<package>/<package>.hash +++++++
package/<package>/<package>.test or test_<package>.py or <package>.py ++++
package/Config.in +1
DEVELOPERS +1
.gitlab-ci.yml +1

I think this beat the fact the test data is spread over the tree.

We can even call flake8 from check-package for .py/.test files, as once Arnout
suggested.

>> > matching the glob 'PKG_NAME-test.py' (with PKG_NAME replaced by the
>> > package name, obviously), and imports them one by one.
>> 
>>  Looks like the right thing to do to me. However, to be consistent with the
>> existing tests, the pattern should be test_pkg.py, and the package name should
>> go through - to _ conversion.
> 
> And to be consistent with the existing .mk and .hash files, the test
> file should start with the package name. So, we have two incompatible
> consistency requirements. One will have to win and the other to lose.
> And I think the best would be to keep the .mk and .hash scheme, because
> this is in the package directory.

nose2 defaults to look for test_*.py but it can be customized adding
'test-file-pattern = *.py' to unittest.cfg

> 
>> > Of course, one may be more inventive with the tests.
>> > 
>> > And in the end, it would have been all integrated in the current infra.
>> > 
>> > But alas, that import stuff I could not make to work... :-( Maybe there
>> > is a node function to tell it where to load extra tests from?

There is a 'code-directories' option to be added to unittest.cfg but I couldn't
get it working easily. Needs further investigation.
I guess it needs to be relative to the path passed to -t that defaults to the
path passed to -s, but it then requires that all current imports inside test
infra be changed to relative or to contain the new full path, i.e. 'import
support.testing.infra.basetest' and probably it needs some __init__.py files to
be created.

>> 
>>  We can just add support/testing to sys.path.
> 
> Except that did not work when I tried... :-/
> 
> But I also tried setting package/ in sys.path and it did not work
> either... Meh, someone will have to teach me some python magic... ;-)
> 
>> > An alternative would be to go with per-package test files, like
>> > explained above, but have the support/testing/run-test script do the can
>> > before calling nose, and for each file if finds, do a symlink ( or a
>> > copy) in the testing infra, in a special directory that is git-ignored ?
>> 
>>  Bwerk, hack...

Agree, but beware of the possible overpopulation of __init__.py, see below.
Perhaps there is a way to avoid it, but I don't know one.

> 
> Maybe, but very, very easy and trivial!
> 
> We can even keep the PKG_NAME-test.py (or PKG_NAME.test) and symlink
> that as test_PKG_NAME.py to respect the nose naming scheme.

That should work if we symlink each .py file to a git-ignored path that have
__init__.py

Other way around would be to have a symlink to package/, but that AFAIK would
require a __init__.py file in the package/ dir and also in each path leading to
the test files. So a lot of empty files, at least one per package that have
tests.

> 
>> > Now, completely unrelated to the above, when we want to test each module
>> > to ensure they are correct and have the required dependencies, we need
>> > to have a configuration for each of the modules, which means doing one
>> > build per module we want to test, each build including building the
>> > python interpreter. This will make for a very long test campaign,
>> > indeed... :-/
>> 
>>  We don't care that it takes a long time. It's not intended to be run
>> sequentially anyway. And in gitlab, all tests run in parallel (but only on 4
>> builders IIRC). As long as the tests on gitlab finish within a day or so, we're
>> golden.
> 
> Yep, but we currently have about 224 python packages, so if each needs
> 30 minutes to build and run, that's about 5 days to run the full
> suite... Just sayin'...

I think Arnout has a point here. With the 4 free runners in parallel it should
end in 28 hours.

Regards,
Ricardo