[PATCH] oeqa/ptest: print a warning if ptests failed

[PATCH] oeqa/ptest: print a warning if ptests failed

[Alexander Kanavin]

This allows spotting ptest regressions without having hard ptest failures
(for that full ptest stability should be achieved).

Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
---
 meta/lib/oeqa/runtime/cases/ptest.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/lib/oeqa/runtime/cases/ptest.py b/meta/lib/oeqa/runtime/cases/ptest.py
index a9572c81f0..0800f3c27f 100644
--- a/meta/lib/oeqa/runtime/cases/ptest.py
+++ b/meta/lib/oeqa/runtime/cases/ptest.py
@@ -108,4 +108,5 @@ class PtestRunnerTest(OERuntimeTestCase):
             failmsg = failmsg + "Failed ptests:\n%s" % pprint.pformat(failed_tests)
 
         if failmsg:
+            self.logger.warning("There were failing ptests.")
             self.fail(failmsg)
-- 
2.29.2

Re: [OE-core] [PATCH] oeqa/ptest: print a warning if ptests failed

[Richard Purdie]

On Sat, 2020-12-19 at 18:29 +0100, Alexander Kanavin wrote:
> This allows spotting ptest regressions without having hard ptest failures
> (for that full ptest stability should be achieved).
> 
> Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
> ---
>  meta/lib/oeqa/runtime/cases/ptest.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/meta/lib/oeqa/runtime/cases/ptest.py b/meta/lib/oeqa/runtime/cases/ptest.py
> index a9572c81f0..0800f3c27f 100644
> --- a/meta/lib/oeqa/runtime/cases/ptest.py
> +++ b/meta/lib/oeqa/runtime/cases/ptest.py
> @@ -108,4 +108,5 @@ class PtestRunnerTest(OERuntimeTestCase):
>              failmsg = failmsg + "Failed ptests:\n%s" % pprint.pformat(failed_tests)
>  
>          if failmsg:
> +            self.logger.warning("There were failing ptests.")
>              self.fail(failmsg)

Have you looked at this on the arm worker? That seems to have four
failing tests over three recipes:

{'libinput': ['libinput-test-suite'],
 'strace': ['qual_fault-syscall.test', 'qual_fault.test'],
 'valgrind': ['drd/tests/std_list']}

https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1355

Not sure if these always fail or are intermittent?

Cheers,

Richard

Re: [OE-core] [PATCH] oeqa/ptest: print a warning if ptests failed

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1777 bytes --]

On Sun, 20 Dec 2020 at 00:56, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Sat, 2020-12-19 at 18:29 +0100, Alexander Kanavin wrote:
> > This allows spotting ptest regressions without having hard ptest failures
> > (for that full ptest stability should be achieved).
> >
> > Signed-off-by: Alexander Kanavin <alex.kanavin@gmail.com>
> > ---
> >  meta/lib/oeqa/runtime/cases/ptest.py | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/meta/lib/oeqa/runtime/cases/ptest.py
> b/meta/lib/oeqa/runtime/cases/ptest.py
> > index a9572c81f0..0800f3c27f 100644
> > --- a/meta/lib/oeqa/runtime/cases/ptest.py
> > +++ b/meta/lib/oeqa/runtime/cases/ptest.py
> > @@ -108,4 +108,5 @@ class PtestRunnerTest(OERuntimeTestCase):
> >              failmsg = failmsg + "Failed ptests:\n%s" %
> pprint.pformat(failed_tests)
> >
> >          if failmsg:
> > +            self.logger.warning("There were failing ptests.")
> >              self.fail(failmsg)
>
> Have you looked at this on the arm worker? That seems to have four
> failing tests over three recipes:
>
> {'libinput': ['libinput-test-suite'],
>  'strace': ['qual_fault-syscall.test', 'qual_fault.test'],
>  'valgrind': ['drd/tests/std_list']}
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1355
>
> Not sure if these always fail or are intermittent?
>

libinput failure should be addressed by Bruce's patch to enable UINPUT
kernel option (I think I saw that in master-next, but now I don't).

The rest are intermittent, as latest master build only shows one different
intermittent failure in glib (which I'm trying to reproduce with additional
diagnostics):
https://autobuilder.yoctoproject.org/typhoon/#/builders/82/builds/1356

Alex

[-- Attachment #2: Type: text/html, Size: 2620 bytes --]