[PATCHv2] test-manual/ptest: Add info about fail state of ptest-runner/ptests

[PATCHv2] test-manual/ptest: Add info about fail state of ptest-runner/ptests

[Anibal Limon]

On the ML was pointed about no specific information about
ptest-runner/ptests so improve docs around.

https://lists.yoctoproject.org/g/yocto-patches/message/1622

[YOCTO #15832]

Signed-off-by: Anibal Limon <anibal@limonsoftware.com>
---
 documentation/test-manual/ptest.rst | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/documentation/test-manual/ptest.rst b/documentation/test-manual/ptest.rst
index 2c021af51..d01735c08 100644
--- a/documentation/test-manual/ptest.rst
+++ b/documentation/test-manual/ptest.rst
@@ -46,13 +46,19 @@ Running ptest
 =============
 
 The ``ptest-runner`` package installs a shell script that loops through
-all installed ptest test suites and runs them in sequence. Consequently,
-you might want to add this package to your image.
+all installed ptest test suites and runs them in sequence.
+
+During the execution ``ptest-runner`` keeps counter of failed ``ptests``
+and use it as exit value, failing the execution if any of the ``run-ptest``
+scripts fails.
+
+Consequently, you might want to add this package to your image.
+
 
 Getting Your Package Ready
 ==========================
 
-In order to enable a recipe to run installed ptests on target hardware,
+In order to enable a recipe to run installed ``ptests`` on target hardware,
 you need to prepare the recipes that build the packages you want to
 test. Here is what you have to do for each recipe:
 
@@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
 
 -  *Create run-ptest:* This script starts your test. Locate the
    script where you will refer to it using
-   :term:`SRC_URI`. Here is an
-   example that starts a test for ``dbus``::
+   :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it
+   as succesfully executed.
+   Here is an example that starts a test for ``dbus``::
 
       #!/bin/sh
       cd test
-- 
2.39.5

Re: [PATCHv2] test-manual/ptest: Add info about fail state of ptest-runner/ptests

[Antonin Godard]

On Tue Jun 10, 2025 at 3:51 AM CEST, Anibal Limon wrote:
> On the ML was pointed about no specific information about
> ptest-runner/ptests so improve docs around.
>
> https://lists.yoctoproject.org/g/yocto-patches/message/1622
>
> [YOCTO #15832]
>
> Signed-off-by: Anibal Limon <anibal@limonsoftware.com>
> ---
>  documentation/test-manual/ptest.rst | 17 ++++++++++++-----
>  1 file changed, 12 insertions(+), 5 deletions(-)
>
> diff --git a/documentation/test-manual/ptest.rst b/documentation/test-manual/ptest.rst
> index 2c021af51..d01735c08 100644
> --- a/documentation/test-manual/ptest.rst
> +++ b/documentation/test-manual/ptest.rst
> @@ -46,13 +46,19 @@ Running ptest
>  =============
>  
>  The ``ptest-runner`` package installs a shell script that loops through
> -all installed ptest test suites and runs them in sequence. Consequently,
> -you might want to add this package to your image.
> +all installed ptest test suites and runs them in sequence.
> +
> +During the execution ``ptest-runner`` keeps counter of failed ``ptests``
> +and use it as exit value, failing the execution if any of the ``run-ptest``
> +scripts fails.
> +
> +Consequently, you might want to add this package to your image.
> +
>  
>  Getting Your Package Ready
>  ==========================
>  
> -In order to enable a recipe to run installed ptests on target hardware,
> +In order to enable a recipe to run installed ``ptests`` on target hardware,
>  you need to prepare the recipes that build the packages you want to
>  test. Here is what you have to do for each recipe:
>  
> @@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
>  
>  -  *Create run-ptest:* This script starts your test. Locate the
>     script where you will refer to it using
> -   :term:`SRC_URI`. Here is an
> -   example that starts a test for ``dbus``::
> +   :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it

I think Yoann suggested that the run-ptest script should return 1 in case of
failure. Can you also mention that? We need to be clear that the run-ptest
script must return a static code and not any non-zero code, as it could clash
with ptest-runner's own exit code (e.g. 127), if I understood correctly.

> +   as succesfully executed.

s/succesfully/sucessfully/

> +   Here is an example that starts a test for ``dbus``::
>  
>        #!/bin/sh
>        cd test

Thanks,
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [PATCHv2] test-manual/ptest: Add info about fail state of ptest-runner/ptests

[Anibal Limon]

[-- Attachment #1: Type: text/plain, Size: 3020 bytes --]

On Tue, Jun 10, 2025 at 9:31 AM Antonin Godard <antonin.godard@bootlin.com>
wrote:

> On Tue Jun 10, 2025 at 3:51 AM CEST, Anibal Limon wrote:
> > On the ML was pointed about no specific information about
> > ptest-runner/ptests so improve docs around.
> >
> > https://lists.yoctoproject.org/g/yocto-patches/message/1622
> >
> > [YOCTO #15832]
> >
> > Signed-off-by: Anibal Limon <anibal@limonsoftware.com>
> > ---
> >  documentation/test-manual/ptest.rst | 17 ++++++++++++-----
> >  1 file changed, 12 insertions(+), 5 deletions(-)
> >
> > diff --git a/documentation/test-manual/ptest.rst
> b/documentation/test-manual/ptest.rst
> > index 2c021af51..d01735c08 100644
> > --- a/documentation/test-manual/ptest.rst
> > +++ b/documentation/test-manual/ptest.rst
> > @@ -46,13 +46,19 @@ Running ptest
> >  =============
> >
> >  The ``ptest-runner`` package installs a shell script that loops through
> > -all installed ptest test suites and runs them in sequence. Consequently,
> > -you might want to add this package to your image.
> > +all installed ptest test suites and runs them in sequence.
> > +
> > +During the execution ``ptest-runner`` keeps counter of failed ``ptests``
> > +and use it as exit value, failing the execution if any of the
> ``run-ptest``
> > +scripts fails.
> > +
> > +Consequently, you might want to add this package to your image.
> > +
> >
> >  Getting Your Package Ready
> >  ==========================
> >
> > -In order to enable a recipe to run installed ptests on target hardware,
> > +In order to enable a recipe to run installed ``ptests`` on target
> hardware,
> >  you need to prepare the recipes that build the packages you want to
> >  test. Here is what you have to do for each recipe:
> >
> > @@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
> >
> >  -  *Create run-ptest:* This script starts your test. Locate the
> >     script where you will refer to it using
> > -   :term:`SRC_URI`. Here is an
> > -   example that starts a test for ``dbus``::
> > +   :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it
>
> I think Yoann suggested that the run-ptest script should return 1 in case
> of
> failure. Can you also mention that? We need to be clear that the run-ptest
> script must return a static code and not any non-zero code, as it could
> clash
> with ptest-runner's own exit code (e.g. 127), if I understood correctly.
>

Current logic is any non-zero is taken as a failure, so at level of the
ptest_runner only increments the counter + 1 when
tests fail, at this stage the PATCHv3 matches the current behaviour.


>
> > +   as succesfully executed.
>
> s/succesfully/sucessfully/
>

Fixed on v3.

Regards!,
Anibal

>
> > +   Here is an example that starts a test for ``dbus``::
> >
> >        #!/bin/sh
> >        cd test
>
> Thanks,
> Antonin
>
> --
> Antonin Godard, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>
>

[-- Attachment #2: Type: text/html, Size: 4287 bytes --]