[LTP] [PATCH] consolidate 4 TCs message_queue_test_02_* into message_queue_test_02.sh to avoid fail in case of random execution

[LTP] [PATCH] consolidate 4 TCs message_queue_test_02_* into message_queue_test_02.sh to avoid fail in case of random execution

[Nga Hoang]

Signed-off-by: Hoang Thi Thuy Nga (RBVH/ENG2) <ngh1hc@HCUT2832C.hc.apac.bosch.com>
---
 runtest/ipc                                              | 5 +----
 testcases/kernel/ipc/ipc_stress/Makefile                 | 2 +-
 testcases/kernel/ipc/ipc_stress/message_queue_test_02.sh | 6 ++++++
 3 files changed, 8 insertions(+), 5 deletions(-)
 create mode 100644 testcases/kernel/ipc/ipc_stress/message_queue_test_02.sh

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-consolidate-4-TCs-message_queue_test_02_-into-messag.patch
Type: text/x-patch
Size: 1534 bytes
Desc: not available
URL: <http://lists.linux.it/pipermail/ltp/attachments/20160715/edef22ab/attachment.bin>

[LTP] [PATCH] consolidate 4 TCs message_queue_test_02_* into message_queue_test_02.sh to avoid fail in case of random execution

[Cyril Hrubis]

Hi!
> diff --git a/runtest/ipc b/runtest/ipc
> index 50f072a..7c6fed5 100644
> --- a/runtest/ipc
> +++ b/runtest/ipc
> @@ -32,10 +32,7 @@ sem01 sem01
>  sem02 sem02
>  
>  message_queue_test_01		message_queue_test_01
> -message_queue_test_02_get	message_queue_test_02_get
> -message_queue_test_02_snd	message_queue_test_02_snd
> -message_queue_test_02_rcv	message_queue_test_02_rcv
> -message_queue_test_02_ctl	message_queue_test_02_ctl -r
> +message_queue_test_02		message_queue_test_02.sh
>  message_queue_test_04		message_queue_test_04
>  message_queue_test_05		message_queue_test_05
>  pipe_test_01			pipe_test_01

This is exactly against our guidelines, see:

https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines#211-runtest-files

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [PATCH] consolidate 4 TCs message_queue_test_02_* into message_queue_test_02.sh to avoid fail in case of random execution

[FIXED-TERM Rempel Oleksij]

Hi Cyril,

> -----Original Message-----
> From: ltp [mailto:ltp-bounces+fixed-
> term.oleksij.rempel=de.bosch.com@lists.linux.it] On Behalf Of Cyril Hrubis
> Sent: Friday, July 15, 2016 11:56 AM
> To: Hoang Thi Thuy Nga (RBVH/ENG2) <Nga.HoangThiThuy@vn.bosch.com>
> Cc: Hoang Thi Thuy Nga (RBVH/ENG2)
> <ngh1hc@HCUT2832C.hc.apac.bosch.com>; ltp@lists.linux.it
> Subject: Re: [LTP] [PATCH] consolidate 4 TCs message_queue_test_02_* into
> message_queue_test_02.sh to avoid fail in case of random execution
> 
> Hi!
> > diff --git a/runtest/ipc b/runtest/ipc index 50f072a..7c6fed5 100644
> > --- a/runtest/ipc
> > +++ b/runtest/ipc
> > @@ -32,10 +32,7 @@ sem01 sem01
> >  sem02 sem02
> >
> >  message_queue_test_01		message_queue_test_01
> > -message_queue_test_02_get	message_queue_test_02_get
> > -message_queue_test_02_snd	message_queue_test_02_snd
> > -message_queue_test_02_rcv	message_queue_test_02_rcv
> > -message_queue_test_02_ctl	message_queue_test_02_ctl -r
> > +message_queue_test_02		message_queue_test_02.sh
> >  message_queue_test_04		message_queue_test_04
> >  message_queue_test_05		message_queue_test_05
> >  pipe_test_01			pipe_test_01
> 
> This is exactly against our guidelines, see:
> 
> https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines#211-runtest-
> files

What exactly do you mean?

To avoid confusions I need to explain background of this patch:
We working on remote execution of LTPs on the target. It means, some script is taking test list and execute each of entry over ssh.
In this case we are able to track resets and continue testing after it.

So, now we found that some resets are not actially triggered by some specific test. So we decided to randomize test order.
This allowed us to find some more bugs on the system, but introduced issues with tests which depend on each other. For example message_queue_test_02_* can be executed only in some specific order.

What would be the proper way to solve this issue?

[LTP] [PATCH] consolidate 4 TCs message_queue_test_02_* into message_queue_test_02.sh to avoid fail in case of random execution

[Cyril Hrubis]

Hi!
> > >  sem02 sem02
> > >
> > >  message_queue_test_01		message_queue_test_01
> > > -message_queue_test_02_get	message_queue_test_02_get
> > > -message_queue_test_02_snd	message_queue_test_02_snd
> > > -message_queue_test_02_rcv	message_queue_test_02_rcv
> > > -message_queue_test_02_ctl	message_queue_test_02_ctl -r
> > > +message_queue_test_02		message_queue_test_02.sh
> > >  message_queue_test_04		message_queue_test_04
> > >  message_queue_test_05		message_queue_test_05
> > >  pipe_test_01			pipe_test_01
> > 
> > This is exactly against our guidelines, see:
> > 
> > https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines#211-runtest-
> > files
> 
> What exactly do you mean?

What I mean and is written in the guidelines is:

The runtest files should have one entry per a test. Creating a wrapper
that runs all your tests and adding it as a single test into runtest
file is strongly discouraged.

> To avoid confusions I need to explain background of this patch: We
> working on remote execution of LTPs on the target. It means, some
> script is taking test list and execute each of entry over ssh.  In
> this case we are able to track resets and continue testing after it.
>
> So, now we found that some resets are not actially triggered by some
> specific test. So we decided to randomize test order.  This allowed us
> to find some more bugs on the system, but introduced issues with tests
> which depend on each other. For example message_queue_test_02_* can be
> executed only in some specific order.

Ah, so they depend on each other. That is a valid reason for executing
them in a defined order.

You should have written better patch description since this is not
exactly clear.

> What would be the proper way to solve this issue?

Looking at the code we should probably avoid running these testcasese in
ipc runtest file, since they were designed to be executed in a loop to
stress the target system. They does not seem very useful when they are
executed just once as from the runtest file.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [PATCH] consolidate 4 TCs message_queue_test_02_* into message_queue_test_02.sh to avoid fail in case of random execution

[fixed-term.Oleksij.Rempel]

On 18.07.2016 13:59, Cyril Hrubis wrote:
> Hi!
>>>>  sem02 sem02
>>>>
>>>>  message_queue_test_01		message_queue_test_01
>>>> -message_queue_test_02_get	message_queue_test_02_get
>>>> -message_queue_test_02_snd	message_queue_test_02_snd
>>>> -message_queue_test_02_rcv	message_queue_test_02_rcv
>>>> -message_queue_test_02_ctl	message_queue_test_02_ctl -r
>>>> +message_queue_test_02		message_queue_test_02.sh
>>>>  message_queue_test_04		message_queue_test_04
>>>>  message_queue_test_05		message_queue_test_05
>>>>  pipe_test_01			pipe_test_01
>>>
>>> This is exactly against our guidelines, see:
>>>
>>> https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines#211-runtest-
>>> files
>>
>> What exactly do you mean?
> 
> What I mean and is written in the guidelines is:
> 
> The runtest files should have one entry per a test. Creating a wrapper
> that runs all your tests and adding it as a single test into runtest
> file is strongly discouraged.
> 
>> To avoid confusions I need to explain background of this patch: We
>> working on remote execution of LTPs on the target. It means, some
>> script is taking test list and execute each of entry over ssh.  In
>> this case we are able to track resets and continue testing after it.
>>
>> So, now we found that some resets are not actially triggered by some
>> specific test. So we decided to randomize test order.  This allowed us
>> to find some more bugs on the system, but introduced issues with tests
>> which depend on each other. For example message_queue_test_02_* can be
>> executed only in some specific order.
> 
> Ah, so they depend on each other. That is a valid reason for executing
> them in a defined order.
> 
> You should have written better patch description since this is not
> exactly clear.

Yea, we are learning to work with upstream directly :)
It will take some time until the patches will be good from first try.

>> What would be the proper way to solve this issue?
> 
> Looking at the code we should probably avoid running these testcasese in
> ipc runtest file, since they were designed to be executed in a loop to
> stress the target system. They does not seem very useful when they are
> executed just once as from the runtest file.

Hmm.. so, they can be removed for now?
Or there is some other place for this kind of tests?
How about cpuhotplug* tests? Some times they fail, but since they are
executed only once, i would expect bad reproducibility.