[LTP] open posix testsuite and names of testcases

[LTP] open posix testsuite and names of testcases

[Jan Stancek]

Hi,

this testsuite has many testcases, where binary name is in format: [number]-[number].run-test
This is not very helpful when trying to find out, which one actually
crashed or which one is responsible for kernel oops/hung tasks.
For example if I get in dmesg:
[  702.018706] 1-5.run-test[10140]: segfault at 7f84e724bed8 ip 0000000000402208 sp 00007f84e724bed0 error 6 in 1-5.run-test[400000+4000]
it's not clear, which testcase is responsible.

Does anyone have some ideas?

I had these two:
1. rename all testcases - lot of work, they are also referenced in some .sh scripts
2. modify run-tests.sh to copy it under more verbose name and run that one

diff --git a/testcases/open_posix_testsuite/bin/run-tests.sh b/testcases/open_posix_testsuite/bin/run-tests.sh
index bd8ca14..ddb6e0b 100755
--- a/testcases/open_posix_testsuite/bin/run-tests.sh
+++ b/testcases/open_posix_testsuite/bin/run-tests.sh
@@ -46,8 +46,11 @@ run_test() {
 
 	complog=$(basename $testname).log.$$
 
-	sh -c "$SCRIPT_DIR/t0 $TIMEOUT_VAL ./$1 $(cat ./$(echo "$1" | sed 's,\.[^\.]*,,').args 2>/dev/null)" > $complog 2>&1
+	# create a copy with more verbose name
+	new_testname=$(basename $TEST_PATH)-$1
+	cp -f ./$1 $new_testname
 
+	sh -c "$SCRIPT_DIR/t0 $TIMEOUT_VAL ./$new_testname $(cat ./$(echo "$1" | sed 's,\.[^\.]*,,').args 2>/dev/null)" > $complog 2>&1
 	ret_code=$?
 
 	if [ "$ret_code" = "0" ]; then

Which makes logs a bit more helpful:
[179468.329814] User process fault: interruption code 0x60011 in pthread_create-1-5.run-test[80000000+5000]

Regards,
Jan

------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] open posix testsuite and names of testcases

[chrubis]

Hi!
> this testsuite has many testcases, where binary name is in format: [number]-[number].run-test
> This is not very helpful when trying to find out, which one actually
> crashed or which one is responsible for kernel oops/hung tasks.
> For example if I get in dmesg:
> [  702.018706] 1-5.run-test[10140]: segfault at 7f84e724bed8 ip 0000000000402208 sp 00007f84e724bed0 error 6 in 1-5.run-test[400000+4000]
> it's not clear, which testcase is responsible.

Here in suse we use CTCS2 to run the testcases and we have a perl script
that generates the the test names accordingly to the directory structure
which doesn't help when looking into the kernel log but the segfault
is logged into the test results anyway.

And mostly because of that the current test execution framework both for
LTP and Open Posix Testsuite is a little out of my radar, although I
would like to have them fixed too.

> Does anyone have some ideas?
> 
> I had these two:
> 1. rename all testcases - lot of work, they are also referenced in some .sh scripts
> 2. modify run-tests.sh to copy it under more verbose name and run that one

Well I don't think that the option one is that hard, at least I'm not
aware of much cases of hardcoded names in the Open Posix Testsuite. On
the other hand having better test driver that executes the tests and
presents the result in more convinient way would be more appealing. Or
we can do both of them.

I was thinking of creating a relatively simply test driver that runs the
tests, takes care of timeouts (and anything else) and logs information
in some reasonable format (I used JSON in similar case). And then about
some tools that could compare the results, print list of failed
testcases and so on. Does that sound reasonable?

And on more general note we surely need to decide how to do things
there. There are some howtos in the Documentation directory none of them
are up to date.

> diff --git a/testcases/open_posix_testsuite/bin/run-tests.sh b/testcases/open_posix_testsuite/bin/run-tests.sh
> index bd8ca14..ddb6e0b 100755
> --- a/testcases/open_posix_testsuite/bin/run-tests.sh
> +++ b/testcases/open_posix_testsuite/bin/run-tests.sh
> @@ -46,8 +46,11 @@ run_test() {
>  
>  	complog=$(basename $testname).log.$$
>  
> -	sh -c "$SCRIPT_DIR/t0 $TIMEOUT_VAL ./$1 $(cat ./$(echo "$1" | sed 's,\.[^\.]*,,').args 2>/dev/null)" > $complog 2>&1
> +	# create a copy with more verbose name
> +	new_testname=$(basename $TEST_PATH)-$1
> +	cp -f ./$1 $new_testname
>  
> +	sh -c "$SCRIPT_DIR/t0 $TIMEOUT_VAL ./$new_testname $(cat ./$(echo "$1" | sed 's,\.[^\.]*,,').args 2>/dev/null)" > $complog 2>&1
>  	ret_code=$?
>  
>  	if [ "$ret_code" = "0" ]; then
> 
> Which makes logs a bit more helpful:
> [179468.329814] User process fault: interruption code 0x60011 in pthread_create-1-5.run-test[80000000+5000]

It's rather crude hack, but I guess that it's better than nothing. And
you should probably remove the file after the test has finished.

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] open posix testsuite and names of testcases

[chrubis]

Hi!
> > Well I don't think that the option one is that hard, at least I'm not
> > aware of much cases of hardcoded names in the Open Posix Testsuite.
> > On
> > the other hand having better test driver that executes the tests and
> > presents the result in more convinient way would be more appealing.
> > Or
> > we can do both of them.
> 
> I think we should do the renaming anyway.
> Would the following format work for everyone? "testdirname-[number]-[number].c"
> For example: pthread_create-1-5.c
> It's unique, and it's easy to tell what old name was.

I'm ok with that.

> > I was thinking of creating a relatively simply test driver that runs
> > the
> > tests, takes care of timeouts (and anything else) and logs
> > information
> > in some reasonable format (I used JSON in similar case). And then
> > about
> > some tools that could compare the results, print list of failed
> > testcases and so on. Does that sound reasonable?
> 
> How would this be related to ltp-pan? Would this be only for open posix testsuite
> or also for LTP?

Well, that is a good question. The differencies between Open Posix
Testsuite and the rest of the LTP aren't too big so it may be possible
to create a test driver for both and share most of the code. On the
other hand. To just run the tests you just need to export right
VARIABLES and modify PATH accordingly. To collect the result you need to
translate the test return values.

On the other hand I'm not even sure about all the features that ltp-pan
has and what subset of them is used by different people. But I will
surely like to have more up to date test driver that produces more
usefull data.

There are couple possibilities what we can do.

We may start creating something that could run the Open Posix Testsuite
and then gradually extend it to work with the rest of the LTP adding all
the features we would need while deprecating the ltp-pan and removing it
once we have full replacement.

Or we may look if there is something to fit our meets out there
allready and modify it to meet our needs.

Or anything that any of you may come up with.

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] open posix testsuite and names of testcases

[Wanlong Gao]

On 12/06/2012 08:06 PM, Jan Stancek wrote:
> Hi,
> 
> this testsuite has many testcases, where binary name is in format: [number]-[number].run-test
> This is not very helpful when trying to find out, which one actually
> crashed or which one is responsible for kernel oops/hung tasks.
> For example if I get in dmesg:
> [  702.018706] 1-5.run-test[10140]: segfault at 7f84e724bed8 ip 0000000000402208 sp 00007f84e724bed0 error 6 in 1-5.run-test[400000+4000]
> it's not clear, which testcase is responsible.

I met the same problem before because running the open_posix on recent RHEL6.4alpha and beta
will cause kernel panic...

Thanks,
Wanlong Gao

> 
> Does anyone have some ideas?
> 
> I had these two:
> 1. rename all testcases - lot of work, they are also referenced in some .sh scripts
> 2. modify run-tests.sh to copy it under more verbose name and run that one
> 
> diff --git a/testcases/open_posix_testsuite/bin/run-tests.sh b/testcases/open_posix_testsuite/bin/run-tests.sh
> index bd8ca14..ddb6e0b 100755
> --- a/testcases/open_posix_testsuite/bin/run-tests.sh
> +++ b/testcases/open_posix_testsuite/bin/run-tests.sh
> @@ -46,8 +46,11 @@ run_test() {
>  
>  	complog=$(basename $testname).log.$$
>  
> -	sh -c "$SCRIPT_DIR/t0 $TIMEOUT_VAL ./$1 $(cat ./$(echo "$1" | sed 's,\.[^\.]*,,').args 2>/dev/null)" > $complog 2>&1
> +	# create a copy with more verbose name
> +	new_testname=$(basename $TEST_PATH)-$1
> +	cp -f ./$1 $new_testname
>  
> +	sh -c "$SCRIPT_DIR/t0 $TIMEOUT_VAL ./$new_testname $(cat ./$(echo "$1" | sed 's,\.[^\.]*,,').args 2>/dev/null)" > $complog 2>&1
>  	ret_code=$?
>  
>  	if [ "$ret_code" = "0" ]; then
> 
> Which makes logs a bit more helpful:
> [179468.329814] User process fault: interruption code 0x60011 in pthread_create-1-5.run-test[80000000+5000]
> 
> Regards,
> Jan
> 
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
> _______________________________________________
> Ltp-list mailing list
> Ltp-list@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/ltp-list
> 


------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] open posix testsuite and names of testcases

[chrubis]

Hi!
> > this testsuite has many testcases, where binary name is in format: [number]-[number].run-test
> > This is not very helpful when trying to find out, which one actually
> > crashed or which one is responsible for kernel oops/hung tasks.
> > For example if I get in dmesg:
> > [  702.018706] 1-5.run-test[10140]: segfault at 7f84e724bed8 ip 0000000000402208 sp 00007f84e724bed0 error 6 in 1-5.run-test[400000+4000]
> > it's not clear, which testcase is responsible.
> 
> I met the same problem before because running the open_posix on recent RHEL6.4alpha and beta
> will cause kernel panic...

Which could still be fixed by a testdriver that ensures that the logfile
was written to the disk before each test is executed. But the renaming
is still a good idea because there may be a rare case where the
filesystem will be trashed.

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] open posix testsuite and names of testcases

[chrubis]

Hi!
Just recently I've found that one of the posix tests stays running after
the testsuite has exited (is reparented to init and runs happily ever
after). Which is another usecase for this rename.

But the original patch was rather hack (and it doesn't work for me
anyway).

So what about renaming the sources itself? Do we agree on this?

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
Free Next-Gen Firewall Hardware Offer
Buy your Sophos next-gen firewall before the end March 2013 
and get the hardware for free! Learn more.
http://p.sf.net/sfu/sophos-d2d-feb
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] open posix testsuite and names of testcases

[chrubis]

Hi!
> > Just recently I've found that one of the posix tests stays running
> > after
> > the testsuite has exited (is reparented to init and runs happily ever
> > after). Which is another usecase for this rename.
> > 
> > But the original patch was rather hack (and it doesn't work for me
> > anyway).
> > 
> > So what about renaming the sources itself? Do we agree on this?
> 
> I'm not against renaming sources, just thinking if there is less
> intrusive way to achieve same result. I believe all Makefiles are generated,
> so other option that comes to mind would be to modify scripts/generate-makefiles.sh.
> Any thoughts about this approach?

We can hack the script to generate Makefiles that include the dir name
into the binary name which would be less intrusive but on the other hand
that would break the rule that foo.c generates foo.run_test...

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
Free Next-Gen Firewall Hardware Offer
Buy your Sophos next-gen firewall before the end March 2013 
and get the hardware for free! Learn more.
http://p.sf.net/sfu/sophos-d2d-feb
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] open posix testsuite and names of testcases

[chrubis]

Hi!
> >>> I'm not against renaming sources, just thinking if there is less
> >>> intrusive way to achieve same result. I believe all Makefiles are
> >>> generated,
> >>> so other option that comes to mind would be to modify
> >>> scripts/generate-makefiles.sh.
> >>> Any thoughts about this approach?
> >>
> >> We can hack the script to generate Makefiles that include the dir
> >> name
> >> into the binary name which would be less intrusive but on the other
> >> hand
> >> that would break the rule that foo.c generates foo.run_test...
> > 
> > Caspar, Wanlong, any preference / thoughts?
> 
> I vote to changing the rule of generating the Makefiles, it's simple
> and the simple rule change is easy to be accepted. Agree?

Ok. I will look into this and hopefully commit a patch soon enough.

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] open posix testsuite and names of testcases

[chrubis]

Hi!
> > I vote to changing the rule of generating the Makefiles, it's simple
> > and the simple rule change is easy to be accepted. Agree?
> 
> Ok. I will look into this and hopefully commit a patch soon enough.

I tried to change the generate-makefiles.sh script, but there are some
problems.

Some tests are shell scripts that doesn't have build phase. I've
workarounded them by adding build rules that just copy them to
name_x-y.sh in the build phase, which unfortunatelly haven't fixed all
the problems.

We have quite quite a lot of tests (~270) that are shell scripts that
prints "This is tested implicitly via assertion xyz" and returns zero.
I'm quite tempted to remove them. These just increase tests run time,
log sizes, etc. but doesn't do any actual work.

Then there are more problematic ones, shell tests that calls one binary
with different parameters. As in sigignore for example. The paths and
names are obviously hardcoded, which makes them fail. The fix for such
cases will depend case by case but what I will probably do is to make
each test self-contained, but I haven't analyzed all the failures yet.

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_d2d_feb
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] open posix testsuite and names of testcases

[chrubis]

> I tried to change the generate-makefiles.sh script, but there are some
> problems.
> 
> Some tests are shell scripts that doesn't have build phase. I've
> workarounded them by adding build rules that just copy them to
> name_x-y.sh in the build phase, which unfortunatelly haven't fixed all
> the problems.
> 
> We have quite quite a lot of tests (~270) that are shell scripts that
> prints "This is tested implicitly via assertion xyz" and returns zero.
> I'm quite tempted to remove them. These just increase tests run time,
> log sizes, etc. but doesn't do any actual work.
> 
> Then there are more problematic ones, shell tests that calls one binary
> with different parameters. As in sigignore for example. The paths and
> names are obviously hardcoded, which makes them fail. The fix for such
> cases will depend case by case but what I will probably do is to make
> each test self-contained, but I haven't analyzed all the failures yet.


Good news everyone, I've just commited a patch that modifies the script
that generates the Makefiles (since I've fixed all the issues that
blocked this change).

Now the test binaries and scripts are prefixed with the directory such
as 'sigaction_'. I've did some testruns and everything seems fine.

Please tests.

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service 
that delivers powerful full stack analytics. Optimize and monitor your
browser, app, & servers with just a few lines of code. Try New Relic
and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] open posix testsuite and names of testcases

[chrubis]

Hi!
> I think I didn't see these ones failing before:
> Failed testcases:
> stress/mqueues/mqueues_multi_send_rev_1
> stress/mqueues/mqueues_multi_send_rev_2
> 
> from logfile.stress-test:
> stress/mqueues/mqueues_multi_send_rev_1: execution: FAILED: Output:
> Usage: ./mqueues_multi_send_rev_1.run-test number_of_threads
> stress/mqueues/mqueues_multi_send_rev_2: execution: FAILED: Output:
> Usage: ./mqueues_multi_send_rev_2.run-test number_of_threads
> 
> Tweaking bin/run-tests.sh or just renaming those ".args" files should do the trick.
> 
> # find . | grep .args
> ./stress/mqueues/multi_send_rev_1.args
> ./stress/mqueues/multi_send_rev_2.args

Thanks I've missed this.

Given that these are only two occurencies I've commited
a patch that just renames the arg files.

-- 
Cyril Hrubis
chrubis@suse.cz

------------------------------------------------------------------------------
Try New Relic Now & We'll Send You this Cool Shirt
New Relic is the only SaaS-based application performance monitoring service 
that delivers powerful full stack analytics. Optimize and monitor your
browser, app, & servers with just a few lines of code. Try New Relic
and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_apr
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list