Python3 ptest and unittest

Python3 ptest and unittest

[Paul Barker]

I've just thrown together a couple of things which may be useful.
They're currently slightly hackish but I could improve them and share
them/submit them as patches if wanted:

1) I've wrote a custom python test runner which runs test suites and
outputs the format expected by ptest natively instead of needing the
sed magic in "openembedded-core/meta/recipes-devtools/python/python/run-ptest".
I'm using it in my own project, might move the opkg test suite over to
it if I have time, and it might be useful for any other python test
suites. It's <50 lines of python :)

2) I've wrote a script which patches this test runner into python3's
own testsuite then runs the suite. You don't even to patch the
Makefile from the python source tree and install it (as in the python
recipe in openembedded-core). It should run on anything with python3
installed with the python standard library (as the standard library
already includes all the tests). This may be a good option for adding
ptest support to the python3 recipe - it'd just be a single 50-line
'run-ptest' script written in python.

Does that sound interesting to anyone else?

-- 
Paul Barker

Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk

Re: Python3 ptest and unittest

[Randy MacLeod]

On 14-02-18 04:27 PM, Paul Barker wrote:
> I've just thrown together a couple of things which may be useful.
> They're currently slightly hackish but I could improve them and share
> them/submit them as patches if wanted:
>
> 1) I've wrote a custom python test runner which runs test suites and
> outputs the format expected by ptest natively instead of needing the
> sed magic in "openembedded-core/meta/recipes-devtools/python/python/run-ptest".
> I'm using it in my own project, might move the opkg test suite over to
> it if I have time, and it might be useful for any other python test
> suites. It's <50 lines of python :)
>
> 2) I've wrote a script which patches this test runner into python3's
> own testsuite then runs the suite. You don't even to patch the
> Makefile from the python source tree and install it (as in the python
> recipe in openembedded-core). It should run on anything with python3
> installed with the python standard library (as the standard library
> already includes all the tests). This may be a good option for adding
> ptest support to the python3 recipe - it'd just be a single 50-line
> 'run-ptest' script written in python.
>
> Does that sound interesting to anyone else?


Yes, this seems to be very useful. Can you send a patch for review?

>


-- 
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350

Re: Python3 ptest and unittest

[Paul Barker]

On 21 February 2014 16:08, Randy MacLeod <randy.macleod@windriver.com> wrote:
> On 14-02-18 04:27 PM, Paul Barker wrote:
>>
>> I've just thrown together a couple of things which may be useful.
>> They're currently slightly hackish but I could improve them and share
>> them/submit them as patches if wanted:
>>
>> 1) I've wrote a custom python test runner which runs test suites and
>> outputs the format expected by ptest natively instead of needing the
>> sed magic in
>> "openembedded-core/meta/recipes-devtools/python/python/run-ptest".
>> I'm using it in my own project, might move the opkg test suite over to
>> it if I have time, and it might be useful for any other python test
>> suites. It's <50 lines of python :)
>>
>> 2) I've wrote a script which patches this test runner into python3's
>> own testsuite then runs the suite. You don't even to patch the
>> Makefile from the python source tree and install it (as in the python
>> recipe in openembedded-core). It should run on anything with python3
>> installed with the python standard library (as the standard library
>> already includes all the tests). This may be a good option for adding
>> ptest support to the python3 recipe - it'd just be a single 50-line
>> 'run-ptest' script written in python.
>>
>> Does that sound interesting to anyone else?
>
> Yes, this seems to be very useful. Can you send a patch for review?
>

I've never worked with the ptest system before but I'm giving it a
go... I'm mostly relying on https://wiki.yoctoproject.org/wiki/Ptest
for info.

It does look like the python3 test suite crashes with out-of-memory
errors on qemux86 with the default configuration of 256 MB RAM. I've
bumped it up to 1 GB and it seems to be working. Is there any way to
note this in the ptest system so that this test suite is skipped if
there isn't enough memory?

Also, I'm getting several failures within the testsuite which I'm not
seeing when I run it on my desktop. Do we keep data on whether each
ptest package is expected to pass or has known failures?

Cheers,

-- 
Paul Barker

Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk

Re: Python3 ptest and unittest

[Randy MacLeod]

On 14-02-22 08:07 PM, Paul Barker wrote:
> On 21 February 2014 16:08, Randy MacLeod <randy.macleod@windriver.com> wrote:
>> On 14-02-18 04:27 PM, Paul Barker wrote:
>>>
>>> I've just thrown together a couple of things which may be useful.
>>> They're currently slightly hackish but I could improve them and share
>>> them/submit them as patches if wanted:
>>>
>>> 1) I've wrote a custom python test runner which runs test suites and
>>> outputs the format expected by ptest natively instead of needing the
>>> sed magic in
>>> "openembedded-core/meta/recipes-devtools/python/python/run-ptest".
>>> I'm using it in my own project, might move the opkg test suite over to
>>> it if I have time, and it might be useful for any other python test
>>> suites. It's <50 lines of python :)
>>>
>>> 2) I've wrote a script which patches this test runner into python3's
>>> own testsuite then runs the suite. You don't even to patch the
>>> Makefile from the python source tree and install it (as in the python
>>> recipe in openembedded-core). It should run on anything with python3
>>> installed with the python standard library (as the standard library
>>> already includes all the tests). This may be a good option for adding
>>> ptest support to the python3 recipe - it'd just be a single 50-line
>>> 'run-ptest' script written in python.
>>>
>>> Does that sound interesting to anyone else?
>>
>> Yes, this seems to be very useful. Can you send a patch for review?
>>
>
> I've never worked with the ptest system before but I'm giving it a
> go... I'm mostly relying on https://wiki.yoctoproject.org/wiki/Ptest
> for info.
>
> It does look like the python3 test suite crashes with out-of-memory
> errors on qemux86 with the default configuration of 256 MB RAM. I've
> bumped it up to 1 GB and it seems to be working.

I've seen similar problems and dealt with it the same way.
Some tests deliberately try to cause an OOM and I assert that
they don't belong in ptest runs.

> Is there any way to
> note this in the ptest system so that this test suite is skipped if
> there isn't enough memory?

Not that I'm aware of. Add to docs?

>
> Also, I'm getting several failures within the testsuite which I'm not
> seeing when I run it on my desktop. Do we keep data on whether each
> ptest package is expected to pass or has known failures?

Not that I know of. We should.

That's something that I've been meaning to do but I won't get to it
for until another project is done so don't wait for me. :)

../Randy

>
> Cheers,
>


-- 
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350

Re: Python3 ptest and unittest

[Paul Barker]

On 23 February 2014 21:50, Randy MacLeod <randy.macleod@windriver.com> wrote:
> On 14-02-22 08:07 PM, Paul Barker wrote:
>>
>> I've never worked with the ptest system before but I'm giving it a
>> go... I'm mostly relying on https://wiki.yoctoproject.org/wiki/Ptest
>> for info.
>>
>> It does look like the python3 test suite crashes with out-of-memory
>> errors on qemux86 with the default configuration of 256 MB RAM. I've
>> bumped it up to 1 GB and it seems to be working.
>
>
> I've seen similar problems and dealt with it the same way.
> Some tests deliberately try to cause an OOM and I assert that
> they don't belong in ptest runs.
>
>
>> Is there any way to
>> note this in the ptest system so that this test suite is skipped if
>> there isn't enough memory?
>
>
> Not that I'm aware of. Add to docs?
>
>
>>
>> Also, I'm getting several failures within the testsuite which I'm not
>> seeing when I run it on my desktop. Do we keep data on whether each
>> ptest package is expected to pass or has known failures?
>
>
> Not that I know of. We should.
>
> That's something that I've been meaning to do but I won't get to it
> for until another project is done so don't wait for me. :)
>

I've submitted the patch to oe-core.

-- 
Paul Barker

Email: paul@paulbarker.me.uk
http://www.paulbarker.me.uk