* [LTP] [PATCH 1/2] pan/ltp-pan.c: fix compilation warnings @ 2014-06-23 8:59 Xiaoguang Wang 2014-06-23 8:59 ` [LTP] [PATCH 2/2] pan/ltp-pan.c: add statistics about not fully tested testcases Xiaoguang Wang 2014-06-23 13:48 ` [LTP] [PATCH 1/2] pan/ltp-pan.c: fix compilation warnings chrubis 0 siblings, 2 replies; 4+ messages in thread From: Xiaoguang Wang @ 2014-06-23 8:59 UTC (permalink / raw) To: ltp-list Delete the useless 'char *av[2];' definition. Fix the 'ignoring return value of read(2)' compilation warnings. Signed-off-by: Xiaoguang Wang <wangxg.fnst@cn.fujitsu.com> --- pan/ltp-pan.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pan/ltp-pan.c b/pan/ltp-pan.c index 6a1a797..c36a9ee 100644 --- a/pan/ltp-pan.c +++ b/pan/ltp-pan.c @@ -453,12 +453,10 @@ int main(int argc, char **argv) * this is an "active file cleanliness" thing */ { - char *av[2], bigarg[82]; + char bigarg[82]; memset(bigarg, '.', 81); bigarg[81] = '\0'; - av[0] = bigarg; - av[1] = NULL; for (c = 0; c < keep_active; c++) { if (zoo_mark_cmdline(zoofile, c, panname, "")) { @@ -864,6 +862,7 @@ run_child(struct coll_entry *colle, struct tag_pgrp *active, int quiet_mode, static long cmdno = 0; int errpipe[2]; /* way to communicate to parent that the tag */ char errbuf[1024]; /* didn't actually start */ + int ret; /* Try to open the file that will be stdout for the test */ if (test_out_dir) { @@ -1026,7 +1025,8 @@ run_child(struct coll_entry *colle, struct tag_pgrp *active, int quiet_mode, char *termtype; struct tms notime = { 0, 0, 0, 0 }; - read(errpipe[0], errbuf, errlen); + ret = read(errpipe[0], errbuf, errlen); + (void)ret; close(errpipe[0]); errbuf[errlen] = '\0'; /* fprintf(stderr, "%s", errbuf); */ -- 1.8.2.1 ------------------------------------------------------------------------------ HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions Find What Matters Most in Your Big Data with HPCC Systems Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. Leverages Graph Analysis for Fast Processing & Easy Data Exploration http://p.sf.net/sfu/hpccsystems _______________________________________________ Ltp-list mailing list Ltp-list@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ltp-list ^ permalink raw reply related [flat|nested] 4+ messages in thread
* [LTP] [PATCH 2/2] pan/ltp-pan.c: add statistics about not fully tested testcases 2014-06-23 8:59 [LTP] [PATCH 1/2] pan/ltp-pan.c: fix compilation warnings Xiaoguang Wang @ 2014-06-23 8:59 ` Xiaoguang Wang 2014-06-23 14:50 ` chrubis 2014-06-23 13:48 ` [LTP] [PATCH 1/2] pan/ltp-pan.c: fix compilation warnings chrubis 1 sibling, 1 reply; 4+ messages in thread From: Xiaoguang Wang @ 2014-06-23 8:59 UTC (permalink / raw) To: ltp-list Currently numbers of testcases in LTP will return TCONF when they are not appropriate to run. But when users execute './runltp' to run the default test suite towards an linux distribution or upstream kernel, if testcases return TCONF, ultimately they will print TPASS to users, then users will not know the real LTP test coverage. Here we change this behaviour and show users testcases which return TCONF, meaning that these testcases are not fully tested. Here is a sample: 1. Before this patch, The output will be: Test Start Time: Mon Jun 23 16:06:07 2014 ----------------------------------------- Testcase Result Exit Value -------- ------ ---------- access01 PASS 0 access02 PASS 0 access03 PASS 0 getxattr01 PASS 0 getxattr02 PASS 0 access04 PASS 0 access05 PASS 0 access06 PASS 0 ----------------------------------------------- Total Tests: 8 Total Failures: 0 Kernel Version: 3.16.0-rc1+ Machine Architecture: x86_64 Hostname: localhost.localdomain 2. After this patch, the output will be: Test Start Time: Mon Jun 23 16:17:50 2014 ----------------------------------------- Testcase Result Exit Value -------- ------ ---------- access01 PASS 0 access02 PASS 0 access03 PASS 0 getxattr01 Not Fully Tested 32 getxattr02 Not Fully Tested 32 access04 PASS 0 access05 PASS 0 access06 PASS 0 ----------------------------------------------- Total Tests: 8 Total Test Not Fully Tested: 2 Total Failures: 0 Kernel Version: 3.16.0-rc1+ Machine Architecture: x86_64 Hostname: localhost.localdomain Signed-off-by: Xiaoguang Wang <wangxg.fnst@cn.fujitsu.com> --- lib/tst_res.c | 2 +- pan/ltp-pan.c | 46 +++++++++++++++++++++++++++++++++++----------- 2 files changed, 36 insertions(+), 12 deletions(-) diff --git a/lib/tst_res.c b/lib/tst_res.c index c179e31..31186e0 100644 --- a/lib/tst_res.c +++ b/lib/tst_res.c @@ -563,7 +563,7 @@ void tst_exit(void) tst_flush(); /* Mask out TRETR, TINFO, and TCONF results from the exit status. */ - exit(T_exitval & ~(TRETR | TINFO | TCONF)); + exit(T_exitval & ~(TRETR | TINFO)); } pid_t tst_fork(void) diff --git a/pan/ltp-pan.c b/pan/ltp-pan.c index c36a9ee..679822a 100644 --- a/pan/ltp-pan.c +++ b/pan/ltp-pan.c @@ -68,6 +68,17 @@ #include "splitstr.h" #include "zoolib.h" +/* Use low 6 bits to encode test result type */ +#define TTYPE_MASK 0x3f +#define TPASS 0 /* Test passed flag */ +#define TFAIL 1 /* Test failed flag */ +#define TBROK 2 /* Test broken flag */ +#define TWARN 4 /* Test warning flag */ +#define TRETR 8 /* Test retire flag */ +#define TINFO 16 /* Test information flag */ +#define TCONF 32 /* Test not appropriate for configuration flag */ +#define TTYPE_RESULT(ttype) ((ttype) & TTYPE_MASK) + /* One entry in the command line collection. */ struct coll_entry { char *name; /* tag name */ @@ -104,7 +115,7 @@ static void pids_running(struct tag_pgrp *running, int keep_active); static int check_pids(struct tag_pgrp *running, int *num_active, int keep_active, FILE * logfile, FILE * failcmdfile, struct orphan_pgrp *orphans, int fmt_print, - int *failcnt, int quiet_mode); + int *failcnt, int *not_fully_run_cases, int quiet_mode); static void propagate_signal(struct tag_pgrp *running, int keep_active, struct orphan_pgrp *orphans); static void dump_coll(struct collection *coll); @@ -160,6 +171,8 @@ int main(int argc, char **argv) int keep_active = 1; int num_active = 0; int failcnt = 0; /* count of total testcases that failed. */ + /* count of total testcases that are not fully run*/ + int not_fully_run_cases = 0; int err, i; int starts = -1; int timed = 0; @@ -332,9 +345,9 @@ int main(int argc, char **argv) fprintf(logfile, "Test Start Time: %s\n", s); fprintf(logfile, "-----------------------------------------\n"); - fprintf(logfile, "%-30.20s %-10.10s %-10.10s\n", + fprintf(logfile, "%-30.20s %-15.15s %-10.10s\n", "Testcase", "Result", "Exit Value"); - fprintf(logfile, "%-30.20s %-10.10s %-10.10s\n", + fprintf(logfile, "%-30.20s %-15.15s %-10.10s\n", "--------", "------", "------------"); } fflush(logfile); @@ -568,7 +581,7 @@ int main(int argc, char **argv) err = check_pids(running, &num_active, keep_active, logfile, failcmdfile, orphans, fmt_print, &failcnt, - quiet_mode); + ¬_fully_run_cases, quiet_mode); if (Debug & Drunning) { pids_running(running, keep_active); orphans_running(orphans); @@ -628,6 +641,8 @@ int main(int argc, char **argv) fprintf(logfile, "\n-----------------------------------------------\n"); fprintf(logfile, "Total Tests: %d\n", coll->cnt); + fprintf(logfile, "Total Test Not Fully Tested: %d\n", + not_fully_run_cases); fprintf(logfile, "Total Failures: %d\n", failcnt); fprintf(logfile, "Kernel Version: %s\n", unamebuf.release); fprintf(logfile, "Machine Architecture: %s\n", @@ -679,7 +694,8 @@ propagate_signal(struct tag_pgrp *running, int keep_active, static int check_pids(struct tag_pgrp *running, int *num_active, int keep_active, FILE * logfile, FILE * failcmdfile, struct orphan_pgrp *orphans, - int fmt_print, int *failcnt, int quiet_mode) + int fmt_print, int *failcnt, int *not_fully_run_cases, + int quiet_mode) { int w; pid_t cpid; @@ -688,6 +704,7 @@ check_pids(struct tag_pgrp *running, int *num_active, int keep_active, int i; time_t t; char *status; + char *result_str; int signaled = 0; struct tms tms1, tms2; clock_t tck; @@ -735,7 +752,7 @@ check_pids(struct tag_pgrp *running, int *num_active, int keep_active, "child %d exited with status %d\n", cpid, w); --*num_active; - if (w != 0) + if (w != 0 && w != TCONF) ret++; } else if (WIFSTOPPED(stat_loc)) { /* should never happen */ w = WSTOPSIG(stat_loc); @@ -782,13 +799,20 @@ check_pids(struct tag_pgrp *running, int *num_active, int keep_active, (int)(tms2.tms_cstime - tms1.tms_cstime)); else { - if (w != 0) - ++ * failcnt; + if (strcmp(status, "exited") == 0 && TTYPE_RESULT(w) == TCONF) { + ++*not_fully_run_cases; + result_str = "Not Fully Tested"; + } else if (w != 0) { + ++*failcnt; + result_str = "FAIL"; + } else { + result_str = "PASS"; + } + fprintf(logfile, - "%-30.30s %-10.10s %-5d\n", + "%-30.30s %-20.20s %-5d\n", running[i].cmd->name, - ((w != - 0) ? "FAIL" : "PASS"), + result_str, w); } -- 1.8.2.1 ------------------------------------------------------------------------------ HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions Find What Matters Most in Your Big Data with HPCC Systems Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. Leverages Graph Analysis for Fast Processing & Easy Data Exploration http://p.sf.net/sfu/hpccsystems _______________________________________________ Ltp-list mailing list Ltp-list@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ltp-list ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [LTP] [PATCH 2/2] pan/ltp-pan.c: add statistics about not fully tested testcases 2014-06-23 8:59 ` [LTP] [PATCH 2/2] pan/ltp-pan.c: add statistics about not fully tested testcases Xiaoguang Wang @ 2014-06-23 14:50 ` chrubis 0 siblings, 0 replies; 4+ messages in thread From: chrubis @ 2014-06-23 14:50 UTC (permalink / raw) To: Xiaoguang Wang; +Cc: ltp-list Hi! > Currently numbers of testcases in LTP will return TCONF when they are not > appropriate to run. But when users execute './runltp' to run the default > test suite towards an linux distribution or upstream kernel, if testcases > return TCONF, ultimately they will print TPASS to users, then users will > not know the real LTP test coverage. > > Here we change this behaviour and show users testcases which return TCONF, > meaning that these testcases are not fully tested. > > Here is a sample: > 1. Before this patch, The output will be: > Test Start Time: Mon Jun 23 16:06:07 2014 > ----------------------------------------- > Testcase Result Exit Value > -------- ------ ---------- > access01 PASS 0 > access02 PASS 0 > access03 PASS 0 > getxattr01 PASS 0 > getxattr02 PASS 0 > access04 PASS 0 > access05 PASS 0 > access06 PASS 0 > > ----------------------------------------------- > Total Tests: 8 > Total Failures: 0 > Kernel Version: 3.16.0-rc1+ > Machine Architecture: x86_64 > Hostname: localhost.localdomain > > 2. After this patch, the output will be: > Test Start Time: Mon Jun 23 16:17:50 2014 > ----------------------------------------- > Testcase Result Exit Value > -------- ------ ---------- > access01 PASS 0 > access02 PASS 0 > access03 PASS 0 > getxattr01 Not Fully Tested 32 > getxattr02 Not Fully Tested 32 I would preffer TCONF instead of the "Not Fully Tested" string, because the "Not Fully Tested" sounds more like the test was finished (testcases not written), while TCONF means that the system lacks kernel or libc support or system resources or similar to execute the test. What should be improved here is the documentation which should explain what TCONF means. I guess that we need some introduction document on how to run LTP and how to interpret the results but that is outside of the scope for this patch. > access04 PASS 0 > access05 PASS 0 > access06 PASS 0 > > ----------------------------------------------- > Total Tests: 8 > Total Test Not Fully Tested: 2 > Total Failures: 0 > Kernel Version: 3.16.0-rc1+ > Machine Architecture: x86_64 > Hostname: localhost.localdomain > > Signed-off-by: Xiaoguang Wang <wangxg.fnst@cn.fujitsu.com> > --- > lib/tst_res.c | 2 +- > pan/ltp-pan.c | 46 +++++++++++++++++++++++++++++++++++----------- > 2 files changed, 36 insertions(+), 12 deletions(-) > > diff --git a/lib/tst_res.c b/lib/tst_res.c > index c179e31..31186e0 100644 > --- a/lib/tst_res.c > +++ b/lib/tst_res.c > @@ -563,7 +563,7 @@ void tst_exit(void) > tst_flush(); > > /* Mask out TRETR, TINFO, and TCONF results from the exit status. */ > - exit(T_exitval & ~(TRETR | TINFO | TCONF)); > + exit(T_exitval & ~(TRETR | TINFO)); > } That is OK, but we need to change the shell library as well. See tst_exit() in ltp/testcases/lib/test.sh > pid_t tst_fork(void) > diff --git a/pan/ltp-pan.c b/pan/ltp-pan.c > index c36a9ee..679822a 100644 > --- a/pan/ltp-pan.c > +++ b/pan/ltp-pan.c > @@ -68,6 +68,17 @@ > #include "splitstr.h" > #include "zoolib.h" > > +/* Use low 6 bits to encode test result type */ > +#define TTYPE_MASK 0x3f > +#define TPASS 0 /* Test passed flag */ > +#define TFAIL 1 /* Test failed flag */ > +#define TBROK 2 /* Test broken flag */ > +#define TWARN 4 /* Test warning flag */ > +#define TRETR 8 /* Test retire flag */ > +#define TINFO 16 /* Test information flag */ > +#define TCONF 32 /* Test not appropriate for configuration flag */ > +#define TTYPE_RESULT(ttype) ((ttype) & TTYPE_MASK) This is copy & paste from the test.h isn't it? We should put these into shared header and include it to both ltp-pan as well as test.h (assuming that you ommited including test.h because it contains quite a lot of fucntions that are not usable from ltp-pan). > /* One entry in the command line collection. */ > struct coll_entry { > char *name; /* tag name */ > @@ -104,7 +115,7 @@ static void pids_running(struct tag_pgrp *running, int keep_active); > static int check_pids(struct tag_pgrp *running, int *num_active, > int keep_active, FILE * logfile, FILE * failcmdfile, > struct orphan_pgrp *orphans, int fmt_print, > - int *failcnt, int quiet_mode); > + int *failcnt, int *not_fully_run_cases, int quiet_mode); Again, I would stick to conf for names, i.e. confcnt and the same in the rest of the code. > static void propagate_signal(struct tag_pgrp *running, int keep_active, > struct orphan_pgrp *orphans); > static void dump_coll(struct collection *coll); > @@ -160,6 +171,8 @@ int main(int argc, char **argv) > int keep_active = 1; > int num_active = 0; > int failcnt = 0; /* count of total testcases that failed. */ > + /* count of total testcases that are not fully run*/ > + int not_fully_run_cases = 0; > int err, i; > int starts = -1; > int timed = 0; > @@ -332,9 +345,9 @@ int main(int argc, char **argv) > fprintf(logfile, "Test Start Time: %s\n", s); > fprintf(logfile, > "-----------------------------------------\n"); > - fprintf(logfile, "%-30.20s %-10.10s %-10.10s\n", > + fprintf(logfile, "%-30.20s %-15.15s %-10.10s\n", > "Testcase", "Result", "Exit Value"); > - fprintf(logfile, "%-30.20s %-10.10s %-10.10s\n", > + fprintf(logfile, "%-30.20s %-15.15s %-10.10s\n", > "--------", "------", "------------"); > } > fflush(logfile); > @@ -568,7 +581,7 @@ int main(int argc, char **argv) > > err = check_pids(running, &num_active, keep_active, logfile, > failcmdfile, orphans, fmt_print, &failcnt, > - quiet_mode); > + ¬_fully_run_cases, quiet_mode); > if (Debug & Drunning) { > pids_running(running, keep_active); > orphans_running(orphans); > @@ -628,6 +641,8 @@ int main(int argc, char **argv) > fprintf(logfile, > "\n-----------------------------------------------\n"); > fprintf(logfile, "Total Tests: %d\n", coll->cnt); > + fprintf(logfile, "Total Test Not Fully Tested: %d\n", > + not_fully_run_cases); Hmm, the best message I can think of here is "Total Skipped Tests: " > fprintf(logfile, "Total Failures: %d\n", failcnt); > fprintf(logfile, "Kernel Version: %s\n", unamebuf.release); > fprintf(logfile, "Machine Architecture: %s\n", > @@ -679,7 +694,8 @@ propagate_signal(struct tag_pgrp *running, int keep_active, > static int > check_pids(struct tag_pgrp *running, int *num_active, int keep_active, > FILE * logfile, FILE * failcmdfile, struct orphan_pgrp *orphans, > - int fmt_print, int *failcnt, int quiet_mode) > + int fmt_print, int *failcnt, int *not_fully_run_cases, > + int quiet_mode) > { > int w; > pid_t cpid; > @@ -688,6 +704,7 @@ check_pids(struct tag_pgrp *running, int *num_active, int keep_active, > int i; > time_t t; > char *status; > + char *result_str; > int signaled = 0; > struct tms tms1, tms2; > clock_t tck; > @@ -735,7 +752,7 @@ check_pids(struct tag_pgrp *running, int *num_active, int keep_active, > "child %d exited with status %d\n", > cpid, w); > --*num_active; > - if (w != 0) > + if (w != 0 && w != TCONF) > ret++; > } else if (WIFSTOPPED(stat_loc)) { /* should never happen */ > w = WSTOPSIG(stat_loc); > @@ -782,13 +799,20 @@ check_pids(struct tag_pgrp *running, int *num_active, int keep_active, > (int)(tms2.tms_cstime - > tms1.tms_cstime)); > else { > - if (w != 0) > - ++ * failcnt; > + if (strcmp(status, "exited") == 0 && TTYPE_RESULT(w) == TCONF) { > + ++*not_fully_run_cases; > + result_str = "Not Fully Tested"; > + } else if (w != 0) { > + ++*failcnt; > + result_str = "FAIL"; > + } else { > + result_str = "PASS"; > + } The TTYPE_RESULT() IMHO is not needed here because if you look at tst_res() it's applied to filter out TERRNO and TTERRNO before the return value is stored (which is the value later masked for TINFO and TRETR and returned by tst_exit()). Or am I mistaken? Otherwise the logic seems fine to me. I.e. if only TCONF bit is set, return TCONF otherwise it's a failure. Othewise it looks good to me (I will have to look closer on ltp-pan.c source to make sure that these changes are 100% Ok, which I will do for the next iteration). -- Cyril Hrubis chrubis@suse.cz ------------------------------------------------------------------------------ HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions Find What Matters Most in Your Big Data with HPCC Systems Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. Leverages Graph Analysis for Fast Processing & Easy Data Exploration http://p.sf.net/sfu/hpccsystems _______________________________________________ Ltp-list mailing list Ltp-list@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ltp-list ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [LTP] [PATCH 1/2] pan/ltp-pan.c: fix compilation warnings 2014-06-23 8:59 [LTP] [PATCH 1/2] pan/ltp-pan.c: fix compilation warnings Xiaoguang Wang 2014-06-23 8:59 ` [LTP] [PATCH 2/2] pan/ltp-pan.c: add statistics about not fully tested testcases Xiaoguang Wang @ 2014-06-23 13:48 ` chrubis 1 sibling, 0 replies; 4+ messages in thread From: chrubis @ 2014-06-23 13:48 UTC (permalink / raw) To: Xiaoguang Wang; +Cc: ltp-list Hi! > static long cmdno = 0; > int errpipe[2]; /* way to communicate to parent that the tag */ > char errbuf[1024]; /* didn't actually start */ > + int ret; > > /* Try to open the file that will be stdout for the test */ > if (test_out_dir) { > @@ -1026,7 +1025,8 @@ run_child(struct coll_entry *colle, struct tag_pgrp *active, int quiet_mode, > char *termtype; > struct tms notime = { 0, 0, 0, 0 }; > > - read(errpipe[0], errbuf, errlen); > + ret = read(errpipe[0], errbuf, errlen); > + (void)ret; I don't like that we do this just to silence compiler warning. Can't we issue an error message in case that the read() failed instead? i.e. if (read(errpipe[0], errbuf, errlen)) fprintf(stderr, "Failed to read errbuf\n"); -- Cyril Hrubis chrubis@suse.cz ------------------------------------------------------------------------------ HPCC Systems Open Source Big Data Platform from LexisNexis Risk Solutions Find What Matters Most in Your Big Data with HPCC Systems Open Source. Fast. Scalable. Simple. Ideal for Dirty Data. Leverages Graph Analysis for Fast Processing & Easy Data Exploration http://p.sf.net/sfu/hpccsystems _______________________________________________ Ltp-list mailing list Ltp-list@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ltp-list ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2014-06-23 14:50 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2014-06-23 8:59 [LTP] [PATCH 1/2] pan/ltp-pan.c: fix compilation warnings Xiaoguang Wang 2014-06-23 8:59 ` [LTP] [PATCH 2/2] pan/ltp-pan.c: add statistics about not fully tested testcases Xiaoguang Wang 2014-06-23 14:50 ` chrubis 2014-06-23 13:48 ` [LTP] [PATCH 1/2] pan/ltp-pan.c: fix compilation warnings chrubis
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox