[OE-core][PATCH] selftest: allow '-R' and '-r' be used together

[OE-core][PATCH] selftest: allow '-R' and '-r' be used together

[Chen Qi]

For now, there's no easy way to skip some specified tests when
using 'oe-selftest -r'. This is because '-R' and '-r' are in the
mutually exclusive group. Looking at the codes, the codes have already
allowed running specified tests and skipping specified tests at
the same time. So move '-R' out of the mutually exclusive group
and change its help message.

After this change, the following command could be used.
  oe-selftest -r A -R A.B.C

This does introduce a behavior regression.
Previous 'oe-selftest -R xxx' needs to now be changed to
'oe-selftest -a -R xxx'.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/lib/oeqa/selftest/context.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oeqa/selftest/context.py b/meta/lib/oeqa/selftest/context.py
index 78c7a467e2..c7dd03ce37 100644
--- a/meta/lib/oeqa/selftest/context.py
+++ b/meta/lib/oeqa/selftest/context.py
@@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
         group.add_argument('-a', '--run-all-tests', default=False,
                 action="store_true", dest="run_all_tests",
                 help='Run all (unhidden) tests')
-        group.add_argument('-R', '--skip-tests', required=False, action='store',
-                nargs='+', dest="skips", default=None,
-                help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]')
         group.add_argument('-r', '--run-tests', required=False, action='store',
                 nargs='+', dest="run_tests", default=None,
                 help='Select what tests to run (modules, classes or test methods). Format should be: <module>.<class>.<test_method>')
@@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
                 action="store_true", default=False,
                 help='List all available tests.')
 
+        parser.add_argument('-R', '--skip-tests', required=False, action='store',
+                nargs='+', dest="skips", default=None,
+                help='Skip the tests specified. Format should be <module>[.<class>[.<test_method>]]')
         parser.add_argument('-j', '--num-processes', dest='processes', action='store',
                 type=int, help="number of processes to execute in parallel with")
 
-- 
2.37.1

Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

[Alexandre Belloni]

Hello,

On 23/11/2022 18:55:28-0800, Chen Qi wrote:
> For now, there's no easy way to skip some specified tests when
> using 'oe-selftest -r'. This is because '-R' and '-r' are in the
> mutually exclusive group. Looking at the codes, the codes have already
> allowed running specified tests and skipping specified tests at
> the same time. So move '-R' out of the mutually exclusive group
> and change its help message.
> 
> After this change, the following command could be used.
>   oe-selftest -r A -R A.B.C
> 
> This does introduce a behavior regression.
> Previous 'oe-selftest -R xxx' needs to now be changed to
> 'oe-selftest -a -R xxx'.
> 

This fails on the autobuilders which uses:

oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4395/steps/14/logs/stdio

> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  meta/lib/oeqa/selftest/context.py | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/lib/oeqa/selftest/context.py b/meta/lib/oeqa/selftest/context.py
> index 78c7a467e2..c7dd03ce37 100644
> --- a/meta/lib/oeqa/selftest/context.py
> +++ b/meta/lib/oeqa/selftest/context.py
> @@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
>          group.add_argument('-a', '--run-all-tests', default=False,
>                  action="store_true", dest="run_all_tests",
>                  help='Run all (unhidden) tests')
> -        group.add_argument('-R', '--skip-tests', required=False, action='store',
> -                nargs='+', dest="skips", default=None,
> -                help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]')
>          group.add_argument('-r', '--run-tests', required=False, action='store',
>                  nargs='+', dest="run_tests", default=None,
>                  help='Select what tests to run (modules, classes or test methods). Format should be: <module>.<class>.<test_method>')
> @@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
>                  action="store_true", default=False,
>                  help='List all available tests.')
>  
> +        parser.add_argument('-R', '--skip-tests', required=False, action='store',
> +                nargs='+', dest="skips", default=None,
> +                help='Skip the tests specified. Format should be <module>[.<class>[.<test_method>]]')
>          parser.add_argument('-j', '--num-processes', dest='processes', action='store',
>                  type=int, help="number of processes to execute in parallel with")
>  
> -- 
> 2.37.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#173732): https://lists.openembedded.org/g/openembedded-core/message/173732
> Mute This Topic: https://lists.openembedded.org/mt/95231396/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

RE: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

[Chen, Qi]

Hi Alexandre,

Yes, I'm aware of this behavior regression.
With this patch, 'oe-selftest -R xxx' needs to be changed to 'oe-selftest -a -R xxx'.

I'm not sure if this is useful for other people. Because how people use oe-selftest is unknown to me.
For me, I want to run something like 'oe-selftest -r A -R A.B.C'.

Regards,
Qi

-----Original Message-----
From: Alexandre Belloni <alexandre.belloni@bootlin.com> 
Sent: Thursday, November 24, 2022 7:35 PM
To: Chen, Qi <Qi.Chen@windriver.com>
Cc: openembedded-core@lists.openembedded.org
Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

Hello,

On 23/11/2022 18:55:28-0800, Chen Qi wrote:
> For now, there's no easy way to skip some specified tests when using 
> 'oe-selftest -r'. This is because '-R' and '-r' are in the mutually 
> exclusive group. Looking at the codes, the codes have already allowed 
> running specified tests and skipping specified tests at the same time. 
> So move '-R' out of the mutually exclusive group and change its help 
> message.
> 
> After this change, the following command could be used.
>   oe-selftest -r A -R A.B.C
> 
> This does introduce a behavior regression.
> Previous 'oe-selftest -R xxx' needs to now be changed to 'oe-selftest 
> -a -R xxx'.
> 

This fails on the autobuilders which uses:

oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4395/steps/14/logs/stdio

> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  meta/lib/oeqa/selftest/context.py | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/lib/oeqa/selftest/context.py 
> b/meta/lib/oeqa/selftest/context.py
> index 78c7a467e2..c7dd03ce37 100644
> --- a/meta/lib/oeqa/selftest/context.py
> +++ b/meta/lib/oeqa/selftest/context.py
> @@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
>          group.add_argument('-a', '--run-all-tests', default=False,
>                  action="store_true", dest="run_all_tests",
>                  help='Run all (unhidden) tests')
> -        group.add_argument('-R', '--skip-tests', required=False, action='store',
> -                nargs='+', dest="skips", default=None,
> -                help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]')
>          group.add_argument('-r', '--run-tests', required=False, action='store',
>                  nargs='+', dest="run_tests", default=None,
>                  help='Select what tests to run (modules, classes or 
> test methods). Format should be: <module>.<class>.<test_method>') @@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
>                  action="store_true", default=False,
>                  help='List all available tests.')
>  
> +        parser.add_argument('-R', '--skip-tests', required=False, action='store',
> +                nargs='+', dest="skips", default=None,
> +                help='Skip the tests specified. Format should be 
> + <module>[.<class>[.<test_method>]]')
>          parser.add_argument('-j', '--num-processes', dest='processes', action='store',
>                  type=int, help="number of processes to execute in 
> parallel with")
>  
> --
> 2.37.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#173732): https://lists.openembedded.org/g/openembedded-core/message/173732
> Mute This Topic: https://lists.openembedded.org/mt/95231396/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

[Alexandre Belloni]

On 24/11/2022 12:24:38+0000, Chen Qi wrote:
> Hi Alexandre,
> 
> Yes, I'm aware of this behavior regression.
> With this patch, 'oe-selftest -R xxx' needs to be changed to 'oe-selftest -a -R xxx'.
> 

Yes, my point is that there is no -R in the command line used by the AB

> I'm not sure if this is useful for other people. Because how people use oe-selftest is unknown to me.
> For me, I want to run something like 'oe-selftest -r A -R A.B.C'.
> 
> Regards,
> Qi
> 
> -----Original Message-----
> From: Alexandre Belloni <alexandre.belloni@bootlin.com> 
> Sent: Thursday, November 24, 2022 7:35 PM
> To: Chen, Qi <Qi.Chen@windriver.com>
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together
> 
> Hello,
> 
> On 23/11/2022 18:55:28-0800, Chen Qi wrote:
> > For now, there's no easy way to skip some specified tests when using 
> > 'oe-selftest -r'. This is because '-R' and '-r' are in the mutually 
> > exclusive group. Looking at the codes, the codes have already allowed 
> > running specified tests and skipping specified tests at the same time. 
> > So move '-R' out of the mutually exclusive group and change its help 
> > message.
> > 
> > After this change, the following command could be used.
> >   oe-selftest -r A -R A.B.C
> > 
> > This does introduce a behavior regression.
> > Previous 'oe-selftest -R xxx' needs to now be changed to 'oe-selftest 
> > -a -R xxx'.
> > 
> 
> This fails on the autobuilders which uses:
> 
> oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4395/steps/14/logs/stdio
> 
> > Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> > ---
> >  meta/lib/oeqa/selftest/context.py | 6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> > 
> > diff --git a/meta/lib/oeqa/selftest/context.py 
> > b/meta/lib/oeqa/selftest/context.py
> > index 78c7a467e2..c7dd03ce37 100644
> > --- a/meta/lib/oeqa/selftest/context.py
> > +++ b/meta/lib/oeqa/selftest/context.py
> > @@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
> >          group.add_argument('-a', '--run-all-tests', default=False,
> >                  action="store_true", dest="run_all_tests",
> >                  help='Run all (unhidden) tests')
> > -        group.add_argument('-R', '--skip-tests', required=False, action='store',
> > -                nargs='+', dest="skips", default=None,
> > -                help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]')
> >          group.add_argument('-r', '--run-tests', required=False, action='store',
> >                  nargs='+', dest="run_tests", default=None,
> >                  help='Select what tests to run (modules, classes or 
> > test methods). Format should be: <module>.<class>.<test_method>') @@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
> >                  action="store_true", default=False,
> >                  help='List all available tests.')
> >  
> > +        parser.add_argument('-R', '--skip-tests', required=False, action='store',
> > +                nargs='+', dest="skips", default=None,
> > +                help='Skip the tests specified. Format should be 
> > + <module>[.<class>[.<test_method>]]')
> >          parser.add_argument('-j', '--num-processes', dest='processes', action='store',
> >                  type=int, help="number of processes to execute in 
> > parallel with")
> >  
> > --
> > 2.37.1
> > 
> 
> > 
> > 
> > 
> 
> 
> -- 
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#173743): https://lists.openembedded.org/g/openembedded-core/message/173743
> Mute This Topic: https://lists.openembedded.org/mt/95231396/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

RE: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

[Chen, Qi]

Hi Alexandre,

The command line is: oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15

The '--skip-tests' option is the long format of '-R'.

Regards,
Qi

-----Original Message-----
From: Alexandre Belloni <alexandre.belloni@bootlin.com> 
Sent: Thursday, November 24, 2022 9:02 PM
To: Chen, Qi <Qi.Chen@windriver.com>
Cc: openembedded-core@lists.openembedded.org
Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

On 24/11/2022 12:24:38+0000, Chen Qi wrote:
> Hi Alexandre,
> 
> Yes, I'm aware of this behavior regression.
> With this patch, 'oe-selftest -R xxx' needs to be changed to 'oe-selftest -a -R xxx'.
> 

Yes, my point is that there is no -R in the command line used by the AB

> I'm not sure if this is useful for other people. Because how people use oe-selftest is unknown to me.
> For me, I want to run something like 'oe-selftest -r A -R A.B.C'.
> 
> Regards,
> Qi
> 
> -----Original Message-----
> From: Alexandre Belloni <alexandre.belloni@bootlin.com>
> Sent: Thursday, November 24, 2022 7:35 PM
> To: Chen, Qi <Qi.Chen@windriver.com>
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used 
> together
> 
> Hello,
> 
> On 23/11/2022 18:55:28-0800, Chen Qi wrote:
> > For now, there's no easy way to skip some specified tests when using 
> > 'oe-selftest -r'. This is because '-R' and '-r' are in the mutually 
> > exclusive group. Looking at the codes, the codes have already 
> > allowed running specified tests and skipping specified tests at the same time.
> > So move '-R' out of the mutually exclusive group and change its help 
> > message.
> > 
> > After this change, the following command could be used.
> >   oe-selftest -r A -R A.B.C
> > 
> > This does introduce a behavior regression.
> > Previous 'oe-selftest -R xxx' needs to now be changed to 
> > 'oe-selftest -a -R xxx'.
> > 
> 
> This fails on the autobuilders which uses:
> 
> oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg 
> buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T 
> machine -T toolchain-user -T toolchain-system -j 15
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4395
> /steps/14/logs/stdio
> 
> > Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> > ---
> >  meta/lib/oeqa/selftest/context.py | 6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> > 
> > diff --git a/meta/lib/oeqa/selftest/context.py
> > b/meta/lib/oeqa/selftest/context.py
> > index 78c7a467e2..c7dd03ce37 100644
> > --- a/meta/lib/oeqa/selftest/context.py
> > +++ b/meta/lib/oeqa/selftest/context.py
> > @@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
> >          group.add_argument('-a', '--run-all-tests', default=False,
> >                  action="store_true", dest="run_all_tests",
> >                  help='Run all (unhidden) tests')
> > -        group.add_argument('-R', '--skip-tests', required=False, action='store',
> > -                nargs='+', dest="skips", default=None,
> > -                help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]')
> >          group.add_argument('-r', '--run-tests', required=False, action='store',
> >                  nargs='+', dest="run_tests", default=None,
> >                  help='Select what tests to run (modules, classes or 
> > test methods). Format should be: <module>.<class>.<test_method>') @@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
> >                  action="store_true", default=False,
> >                  help='List all available tests.')
> >  
> > +        parser.add_argument('-R', '--skip-tests', required=False, action='store',
> > +                nargs='+', dest="skips", default=None,
> > +                help='Skip the tests specified. Format should be
> > + <module>[.<class>[.<test_method>]]')
> >          parser.add_argument('-j', '--num-processes', dest='processes', action='store',
> >                  type=int, help="number of processes to execute in 
> > parallel with")
> >  
> > --
> > 2.37.1
> > 
> 
> > 
> > 
> > 
> 
> 
> --
> Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel 
> engineering https://bootlin.com

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#173743): 
> https://lists.openembedded.org/g/openembedded-core/message/173743
> Mute This Topic: https://lists.openembedded.org/mt/95231396/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
> [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


--
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com

Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

[Alexandre Belloni]

On 24/11/2022 13:10:51+0000, Chen Qi wrote:
> Hi Alexandre,
> 
> The command line is: oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15
> 
> The '--skip-tests' option is the long format of '-R'.
> 

Right, sorry about that :) I actually checked we didn't have any -R
before sending the patch to the AB... I'll discuss that with RP.

> Regards,
> Qi
> 
> -----Original Message-----
> From: Alexandre Belloni <alexandre.belloni@bootlin.com> 
> Sent: Thursday, November 24, 2022 9:02 PM
> To: Chen, Qi <Qi.Chen@windriver.com>
> Cc: openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together
> 
> On 24/11/2022 12:24:38+0000, Chen Qi wrote:
> > Hi Alexandre,
> > 
> > Yes, I'm aware of this behavior regression.
> > With this patch, 'oe-selftest -R xxx' needs to be changed to 'oe-selftest -a -R xxx'.
> > 
> 
> Yes, my point is that there is no -R in the command line used by the AB
> 
> > I'm not sure if this is useful for other people. Because how people use oe-selftest is unknown to me.
> > For me, I want to run something like 'oe-selftest -r A -R A.B.C'.
> > 
> > Regards,
> > Qi
> > 
> > -----Original Message-----
> > From: Alexandre Belloni <alexandre.belloni@bootlin.com>
> > Sent: Thursday, November 24, 2022 7:35 PM
> > To: Chen, Qi <Qi.Chen@windriver.com>
> > Cc: openembedded-core@lists.openembedded.org
> > Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used 
> > together
> > 
> > Hello,
> > 
> > On 23/11/2022 18:55:28-0800, Chen Qi wrote:
> > > For now, there's no easy way to skip some specified tests when using 
> > > 'oe-selftest -r'. This is because '-R' and '-r' are in the mutually 
> > > exclusive group. Looking at the codes, the codes have already 
> > > allowed running specified tests and skipping specified tests at the same time.
> > > So move '-R' out of the mutually exclusive group and change its help 
> > > message.
> > > 
> > > After this change, the following command could be used.
> > >   oe-selftest -r A -R A.B.C
> > > 
> > > This does introduce a behavior regression.
> > > Previous 'oe-selftest -R xxx' needs to now be changed to 
> > > 'oe-selftest -a -R xxx'.
> > > 
> > 
> > This fails on the autobuilders which uses:
> > 
> > oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg 
> > buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T 
> > machine -T toolchain-user -T toolchain-system -j 15
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4395
> > /steps/14/logs/stdio
> > 
> > > Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> > > ---
> > >  meta/lib/oeqa/selftest/context.py | 6 +++---
> > >  1 file changed, 3 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/meta/lib/oeqa/selftest/context.py
> > > b/meta/lib/oeqa/selftest/context.py
> > > index 78c7a467e2..c7dd03ce37 100644
> > > --- a/meta/lib/oeqa/selftest/context.py
> > > +++ b/meta/lib/oeqa/selftest/context.py
> > > @@ -154,9 +154,6 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
> > >          group.add_argument('-a', '--run-all-tests', default=False,
> > >                  action="store_true", dest="run_all_tests",
> > >                  help='Run all (unhidden) tests')
> > > -        group.add_argument('-R', '--skip-tests', required=False, action='store',
> > > -                nargs='+', dest="skips", default=None,
> > > -                help='Run all (unhidden) tests except the ones specified. Format should be <module>[.<class>[.<test_method>]]')
> > >          group.add_argument('-r', '--run-tests', required=False, action='store',
> > >                  nargs='+', dest="run_tests", default=None,
> > >                  help='Select what tests to run (modules, classes or 
> > > test methods). Format should be: <module>.<class>.<test_method>') @@ -171,6 +168,9 @@ class OESelftestTestContextExecutor(OETestContextExecutor):
> > >                  action="store_true", default=False,
> > >                  help='List all available tests.')
> > >  
> > > +        parser.add_argument('-R', '--skip-tests', required=False, action='store',
> > > +                nargs='+', dest="skips", default=None,
> > > +                help='Skip the tests specified. Format should be
> > > + <module>[.<class>[.<test_method>]]')
> > >          parser.add_argument('-j', '--num-processes', dest='processes', action='store',
> > >                  type=int, help="number of processes to execute in 
> > > parallel with")
> > >  
> > > --
> > > 2.37.1
> > > 
> > 
> > > 
> > > 
> > > 
> > 
> > 
> > --
> > Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel 
> > engineering https://bootlin.com
> 
> > 
> > 
> > 
> 
> 
> --
> Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#173748): https://lists.openembedded.org/g/openembedded-core/message/173748
> Mute This Topic: https://lists.openembedded.org/mt/95231396/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

[Richard Purdie]

On Thu, 2022-11-24 at 14:35 +0100, Alexandre Belloni via
lists.openembedded.org wrote:
> On 24/11/2022 13:10:51+0000, Chen Qi wrote:
> > Hi Alexandre,
> > 
> > The command line is: oe-selftest --skip-tests distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible -T machine -T toolchain-user -T toolchain-system -j 15
> > 
> > The '--skip-tests' option is the long format of '-R'.
> > 
> 
> Right, sorry about that :) I actually checked we didn't have any -R
> before sending the patch to the AB... I'll discuss that with RP.

The autobuilder is not running "all" tests though, it is running two
groups of tests, -T machine and -T toolchain-user. Adding -a here
doesn't really make sense.

Cheers,

Richard

RE: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

[Chen, Qi]

Hi Richard,

The AB is actually running 'all' tests.

The '--skip-tests (-R)' option means 'Run all (unhidden) tests except the ones specified.', according to its help message. This is also its actual effect for now. You can see there's an implicit 'run all tests' meaning in this option.
The '-T (--exclude-tag)' options means 'Exclude all (unhidden) tests that match any of the specified tag(s)', according to its help message.

So the AB's oe-selftest command means: execute all tests except the ones tagged with 'machine' and 'toolchain-user', and also skip  'distrodata.Distrodata.test_checkpkg buildoptions.SourceMirroring.test_yocto_source_mirror reproducible'.

Regards,
Qi

-----Original Message-----
From: Richard Purdie <richard.purdie@linuxfoundation.org> 
Sent: Thursday, November 24, 2022 9:47 PM
To: alexandre.belloni@bootlin.com; Chen, Qi <Qi.Chen@windriver.com>
Cc: openembedded-core@lists.openembedded.org
Subject: Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

On Thu, 2022-11-24 at 14:35 +0100, Alexandre Belloni via lists.openembedded.org wrote:
> On 24/11/2022 13:10:51+0000, Chen Qi wrote:
> > Hi Alexandre,
> > 
> > The command line is: oe-selftest --skip-tests 
> > distrodata.Distrodata.test_checkpkg 
> > buildoptions.SourceMirroring.test_yocto_source_mirror reproducible 
> > -T machine -T toolchain-user -T toolchain-system -j 15
> > 
> > The '--skip-tests' option is the long format of '-R'.
> > 
> 
> Right, sorry about that :) I actually checked we didn't have any -R 
> before sending the patch to the AB... I'll discuss that with RP.

The autobuilder is not running "all" tests though, it is running two groups of tests, -T machine and -T toolchain-user. Adding -a here doesn't really make sense.

Cheers,

Richard

Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

[Richard Purdie]

Hi Qi,

On Fri, 2022-11-25 at 05:56 +0000, Chen Qi wrote:
> The AB is actually running 'all' tests.
> 
> The '--skip-tests (-R)' option means 'Run all (unhidden) tests except
> the ones specified.', according to its help message. This is also its
> actual effect for now. You can see there's an implicit 'run all
> tests' meaning in this option.
> The '-T (--exclude-tag)' options means 'Exclude all (unhidden) tests
> that match any of the specified tag(s)', according to its help
> message.
> 
> So the AB's oe-selftest command means: execute all tests except the
> ones tagged with 'machine' and 'toolchain-user', and also skip 
> 'distrodata.Distrodata.test_checkpkg
> buildoptions.SourceMirroring.test_yocto_source_mirror reproducible'.

You're right, a different build has -t machine and I'm confusing the
two, sorry.

I don't really like making changes which require lockstep changes to
the autobuilder configuration but I can see why we could do with doing
so here.

I was thinking there should be symettry between -r and -R like there is
with -T and -t but that also probably doesn't make sense when you think
about it more.

Alex: We'll probably have to work out how to make this change...

Cheers,

Richard

Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

[Randy MacLeod]

On 2022-11-25 08:00, Richard Purdie via lists.openembedded.org wrote:
> Hi Qi,
> 
> On Fri, 2022-11-25 at 05:56 +0000, Chen Qi wrote:
>> The AB is actually running 'all' tests.
>>
>> The '--skip-tests (-R)' option means 'Run all (unhidden) tests except
>> the ones specified.', according to its help message. This is also its
>> actual effect for now. You can see there's an implicit 'run all
>> tests' meaning in this option.
>> The '-T (--exclude-tag)' options means 'Exclude all (unhidden) tests
>> that match any of the specified tag(s)', according to its help
>> message.
>>
>> So the AB's oe-selftest command means: execute all tests except the
>> ones tagged with 'machine' and 'toolchain-user', and also skip
>> 'distrodata.Distrodata.test_checkpkg
>> buildoptions.SourceMirroring.test_yocto_source_mirror reproducible'.
> 
> You're right, a different build has -t machine and I'm confusing the
> two, sorry.
> 
> I don't really like making changes which require lockstep changes to
> the autobuilder configuration but I can see why we could do with doing
> so here.
> 
> I was thinking there should be symettry between -r and -R like there is
> with -T and -t but that also probably doesn't make sense when you think
> about it more.
> 
> Alex: We'll probably have to work out how to make this change...

Nudge and Happy Holidays.

../Randy


> 
> Cheers,
> 
> Richard
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#173766): https://lists.openembedded.org/g/openembedded-core/message/173766
> Mute This Topic: https://lists.openembedded.org/mt/95231396/3616765
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [randy.macleod@windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

-- 
# Randy MacLeod
# Wind River Linux

Re: [OE-core][PATCH] selftest: allow '-R' and '-r' be used together

[Alexandre Belloni]

On 25/12/2022 19:13:43-0500, Randy MacLeod wrote:
> On 2022-11-25 08:00, Richard Purdie via lists.openembedded.org wrote:
> > Hi Qi,
> > 
> > On Fri, 2022-11-25 at 05:56 +0000, Chen Qi wrote:
> > > The AB is actually running 'all' tests.
> > > 
> > > The '--skip-tests (-R)' option means 'Run all (unhidden) tests except
> > > the ones specified.', according to its help message. This is also its
> > > actual effect for now. You can see there's an implicit 'run all
> > > tests' meaning in this option.
> > > The '-T (--exclude-tag)' options means 'Exclude all (unhidden) tests
> > > that match any of the specified tag(s)', according to its help
> > > message.
> > > 
> > > So the AB's oe-selftest command means: execute all tests except the
> > > ones tagged with 'machine' and 'toolchain-user', and also skip
> > > 'distrodata.Distrodata.test_checkpkg
> > > buildoptions.SourceMirroring.test_yocto_source_mirror reproducible'.
> > 
> > You're right, a different build has -t machine and I'm confusing the
> > two, sorry.
> > 
> > I don't really like making changes which require lockstep changes to
> > the autobuilder configuration but I can see why we could do with doing
> > so here.
> > 
> > I was thinking there should be symettry between -r and -R like there is
> > with -T and -t but that also probably doesn't make sense when you think
> > about it more.
> > 
> > Alex: We'll probably have to work out how to make this change...
> 
> Nudge and Happy Holidays.

Yes, this is still on my radar.


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com