Re: [PATCHv2] test-manual/ptest: Add info about fail state of ptest-runner/ptests

["Antonin Godard" <antonin.godard@bootlin.com>]

On Tue Jun 10, 2025 at 3:51 AM CEST, Anibal Limon wrote:
> On the ML was pointed about no specific information about
> ptest-runner/ptests so improve docs around.
>
> https://lists.yoctoproject.org/g/yocto-patches/message/1622
>
> [YOCTO #15832]
>
> Signed-off-by: Anibal Limon <anibal@limonsoftware.com>
> ---
>  documentation/test-manual/ptest.rst | 17 ++++++++++++-----
>  1 file changed, 12 insertions(+), 5 deletions(-)
>
> diff --git a/documentation/test-manual/ptest.rst b/documentation/test-manual/ptest.rst
> index 2c021af51..d01735c08 100644
> --- a/documentation/test-manual/ptest.rst
> +++ b/documentation/test-manual/ptest.rst
> @@ -46,13 +46,19 @@ Running ptest
>  =============
>  
>  The ``ptest-runner`` package installs a shell script that loops through
> -all installed ptest test suites and runs them in sequence. Consequently,
> -you might want to add this package to your image.
> +all installed ptest test suites and runs them in sequence.
> +
> +During the execution ``ptest-runner`` keeps counter of failed ``ptests``
> +and use it as exit value, failing the execution if any of the ``run-ptest``
> +scripts fails.
> +
> +Consequently, you might want to add this package to your image.
> +
>  
>  Getting Your Package Ready
>  ==========================
>  
> -In order to enable a recipe to run installed ptests on target hardware,
> +In order to enable a recipe to run installed ``ptests`` on target hardware,
>  you need to prepare the recipes that build the packages you want to
>  test. Here is what you have to do for each recipe:
>  
> @@ -77,8 +83,9 @@ test. Here is what you have to do for each recipe:
>  
>  -  *Create run-ptest:* This script starts your test. Locate the
>     script where you will refer to it using
> -   :term:`SRC_URI`. Here is an
> -   example that starts a test for ``dbus``::
> +   :term:`SRC_URI`. Be sure your ``run-ptest`` exit with 0 to mark it

I think Yoann suggested that the run-ptest script should return 1 in case of
failure. Can you also mention that? We need to be clear that the run-ptest
script must return a static code and not any non-zero code, as it could clash
with ptest-runner's own exit code (e.g. 127), if I understood correctly.

> +   as succesfully executed.

s/succesfully/sucessfully/

> +   Here is an example that starts a test for ``dbus``::
>  
>        #!/bin/sh
>        cd test

Thanks,
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com