[LTP] [RFC PATCH] ltp-pan: Report failure if testcases failed to run

[LTP] [RFC PATCH] ltp-pan: Report failure if testcases failed to run

[Sagi Nakash]

In case a test fails to run in run_child() for some reason (signaled via
SIGTERM/SIGTOP, execve() failure, etc.), runltp still reports success:
"INFO: ltp-pan reported all tests PASS".

Failed tests are reported via check_pids(), by checking their exit
status, yet it doesn't catch tests which weren't able to run at all.

exit_stat holds the exit status code: 0 in case of success in all tests,
otherwise, the number of failed tests.

Fix this by increasing exit_stat also when a test fails to run.

Signed-off-by: Sagi Nakash <saginakash@gmail.com>
---
 pan/ltp-pan.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/pan/ltp-pan.c b/pan/ltp-pan.c
index 0bdb51477..eff9a867d 100644
--- a/pan/ltp-pan.c
+++ b/pan/ltp-pan.c
@@ -551,6 +551,8 @@ int main(int argc, char **argv)
 				++num_active;
 			if ((cpid != -1 || sequential) && starts > 0)
 				--starts;
+			if (cpid == -1 && track_exit_stats)
+				exit_stat++;
 
 			if (sequential)
 				if (++c >= coll->cnt)
-- 
2.25.1


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] ltp-pan: Report failure if testcases failed to run

[Sagi Nakash]

[-- Attachment #1.1: Type: text/plain, Size: 1287 bytes --]

Ping

On Sat, Nov 27, 2021, 14:16 Sagi Nakash <saginakash@gmail.com> wrote:

> In case a test fails to run in run_child() for some reason (signaled via
> SIGTERM/SIGTOP, execve() failure, etc.), runltp still reports success:
> "INFO: ltp-pan reported all tests PASS".
>
> Failed tests are reported via check_pids(), by checking their exit
> status, yet it doesn't catch tests which weren't able to run at all.
>
> exit_stat holds the exit status code: 0 in case of success in all tests,
> otherwise, the number of failed tests.
>
> Fix this by increasing exit_stat also when a test fails to run.
>
> Signed-off-by: Sagi Nakash <saginakash@gmail.com>
> ---
>  pan/ltp-pan.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/pan/ltp-pan.c b/pan/ltp-pan.c
> index 0bdb51477..eff9a867d 100644
> --- a/pan/ltp-pan.c
> +++ b/pan/ltp-pan.c
> @@ -551,6 +551,8 @@ int main(int argc, char **argv)
>                                 ++num_active;
>                         if ((cpid != -1 || sequential) && starts > 0)
>                                 --starts;
> +                       if (cpid == -1 && track_exit_stats)
> +                               exit_stat++;
>
>                         if (sequential)
>                                 if (++c >= coll->cnt)
> --
> 2.25.1
>
>

[-- Attachment #1.2: Type: text/html, Size: 1880 bytes --]

[-- Attachment #2: Type: text/plain, Size: 60 bytes --]


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] ltp-pan: Report failure if testcases failed to run

[Sagi Nakash]

[-- Attachment #1.1: Type: text/plain, Size: 1403 bytes --]

Ping

On Fri, Dec 3, 2021, 18:41 Sagi Nakash <saginakash@gmail.com> wrote:

> Ping
>
> On Sat, Nov 27, 2021, 14:16 Sagi Nakash <saginakash@gmail.com> wrote:
>
>> In case a test fails to run in run_child() for some reason (signaled via
>> SIGTERM/SIGTOP, execve() failure, etc.), runltp still reports success:
>> "INFO: ltp-pan reported all tests PASS".
>>
>> Failed tests are reported via check_pids(), by checking their exit
>> status, yet it doesn't catch tests which weren't able to run at all.
>>
>> exit_stat holds the exit status code: 0 in case of success in all tests,
>> otherwise, the number of failed tests.
>>
>> Fix this by increasing exit_stat also when a test fails to run.
>>
>> Signed-off-by: Sagi Nakash <saginakash@gmail.com>
>> ---
>>  pan/ltp-pan.c | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/pan/ltp-pan.c b/pan/ltp-pan.c
>> index 0bdb51477..eff9a867d 100644
>> --- a/pan/ltp-pan.c
>> +++ b/pan/ltp-pan.c
>> @@ -551,6 +551,8 @@ int main(int argc, char **argv)
>>                                 ++num_active;
>>                         if ((cpid != -1 || sequential) && starts > 0)
>>                                 --starts;
>> +                       if (cpid == -1 && track_exit_stats)
>> +                               exit_stat++;
>>
>>                         if (sequential)
>>                                 if (++c >= coll->cnt)
>> --
>> 2.25.1
>>
>>

[-- Attachment #1.2: Type: text/html, Size: 2262 bytes --]

[-- Attachment #2: Type: text/plain, Size: 60 bytes --]


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] ltp-pan: Report failure if testcases failed to run

[Petr Vorel]

Hi,

Reviewed-by: Petr Vorel <pvorel@suse.cz>

@all: Do we want to include this one in the release?

Kind regards,
Petr

> In case a test fails to run in run_child() for some reason (signaled via
> SIGTERM/SIGTOP, execve() failure, etc.), runltp still reports success:
> "INFO: ltp-pan reported all tests PASS".

> Failed tests are reported via check_pids(), by checking their exit
> status, yet it doesn't catch tests which weren't able to run at all.

> exit_stat holds the exit status code: 0 in case of success in all tests,
> otherwise, the number of failed tests.

> Fix this by increasing exit_stat also when a test fails to run.

> Signed-off-by: Sagi Nakash <saginakash@gmail.com>
> ---
>  pan/ltp-pan.c | 2 ++
>  1 file changed, 2 insertions(+)

> diff --git a/pan/ltp-pan.c b/pan/ltp-pan.c
> index 0bdb51477..eff9a867d 100644
> --- a/pan/ltp-pan.c
> +++ b/pan/ltp-pan.c
> @@ -551,6 +551,8 @@ int main(int argc, char **argv)
>  				++num_active;
>  			if ((cpid != -1 || sequential) && starts > 0)
>  				--starts;
> +			if (cpid == -1 && track_exit_stats)
> +				exit_stat++;

>  			if (sequential)
>  				if (++c >= coll->cnt)

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [RFC PATCH] ltp-pan: Report failure if testcases failed to run

[Cyril Hrubis]

Hi!
> Reviewed-by: Petr Vorel <pvorel@suse.cz>
> 
> @all: Do we want to include this one in the release?

I do not think that this is complete enough. If we wanted to fix this we
whould have to change the code to write a record to the failcmdfile as
well.

Also increasing the variable on each failure seems wrong as well, it may
as well overflow on large enough runtest file...

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp