[LTP] RFC : refactor LTP execution model

[LTP] RFC : refactor LTP execution model

[Garrett Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 3562 bytes --]

Hi all,

   Currently LTP's execution scripts are split into 5 discrete blocks:

   1. execltp - python wrapper which invokes runltp and processes output to
expedite execution and reporting (provides summary and context for test
PASSes, and all non-PASSes).
   2. runalltests.sh - a semi-dynamic user-customizable execution wrapper
that executes runltp after performing needed setup for each section;
performs needed teardown after each section. Uses a number of scripts inside
testscripts/*.
   3. runltp - the primary end-user execution wrapper on top of ltp-pan.
   4. runltplite.sh - an abbreviated form of runltp (minus the email / html
output driver,
   5. testscripts/* - a number of ad hoc ltp-pan callers.

   If you take a look at the caller hierarchy, it's something similar to the
following:

| execltp |                     | runalltests.sh |
     |                                  |
| runltp  |  | runltplite.sh |  | testscripts/*  |
      \             |                  /
                | pan |

   Now, analyzing the layers in more detail:

   1. runltplite.sh contains the guts of runltp, but isn't maintained as
readily as runltp, thus needed bug fixes or feature enhancements aren't
making it into runltplite.sh as quickly, if at all, and as such there's some
unnecessary code divergence.
   2.  testscripts/* contains a lot of setup and teardown logic that could
be sorted out and properly handled inline before and after the test is
executed, unlike what's available currently.
   3. It would be nice if the commands and blocks of code were modularized,
s.t. outside groups could pre-process and/or post-process the tests to

   My proposal is as follows:

   1. A discrete set of reporting functions should be created (error, info,
warning, etc). This messages should adhere to the following basic paradigm:
       error - always displayed.
       info - only displayed when quiet isn't used.
       warning - only displayed when quiet isn't used.
       That way folks could
   2. runltp be split into discrete functional blocks, as follows:
       i.   Parse core options. Parse any additional options for
functionality like email, HTML output, etc.
       ii.  Prepare the tests based on the runtests files and options passed
in (run valgrind on the tests, etc).
       iii. Basic setup.
       iv.  Execute the tests.
       v.   Basic teardown.
       vi.  Output methods for email, HTML, etc.
   2. The testscripts/* setup and teardown logic be moved to a separate
script, which looks like the following:

<testname>_setup()
{
# Do setup here.
}

<testname>_test()
{
# Execute runtest file via pan.
}

<testname>_teardown()
{
# Do teardown here.
}

    So when runltp is executed, it does something like the following:
    1. source testscript [if it exists].
    2. Change the test ``execution plan'' so that they setup and teardown
inexplicably executed in the following manner (following Java's unit-test
and python unittest):

    Setup the test; if successful, continue on with the test(s) and teardown
at the end.

    The purpose for making this into a script as opposed to any other sort
of metadata, is that anyone executing the tests standalone will only need to
source the script to set everything up, then execute the test itself.
    The only thing that might be a problem is that runtest files are mashed
together if multiple runtest scenario files are specified. This shouldn't be
an issue though if everything is strung together properly via the calls
above (setup, test, teardown).
    Thoughts?
Thanks,
-Garrett

[-- Attachment #1.2: Type: text/html, Size: 4356 bytes --]

[-- Attachment #2: Type: text/plain, Size: 345 bytes --]

------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev

[-- Attachment #3: Type: text/plain, Size: 155 bytes --]

_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: [LTP] RFC : refactor LTP execution model

[Rishikesh K Rajak]

Hi Garret,

My point will still not be so much useful for you but i tried to understand your
proposal last weekend and thought to reply you. May be Subrata can give
you much input here as he is with this project from long time.

>   My proposal is as follows:
>
>   1. A discrete set of reporting functions should be created (error,
>info,
>warning, etc). This messages should adhere to the following basic
>paradigm:
>       error - always displayed.
>       info - only displayed when quiet isn't used.
>       warning - only displayed when quiet isn't used.
>       That way folks could
>

Hmm.... that's way we can give crisp info to user, but what about TBROK
tetscases ? Are we going to categorize that in error ?

>   2. runltp be split into discrete functional blocks, as follows:
>       i.   Parse core options. Parse any additional options for
>functionality like email, HTML output, etc.
>       ii.  Prepare the tests based on the runtests files and options
>passed
>in (run valgrind on the tests, etc).
>       iii. Basic setup.
>       iv.  Execute the tests.
>       v.   Basic teardown.
>       vi.  Output methods for email, HTML, etc.
>
So are you proposing for writing all new scripts ? Did not quite
understand what could be the motive behind it. We are already having
runalltest.sh , which does this segregation for user whether he wants an
email, HTML or not.

>    So when runltp is executed, it does something like the following:
>    1. source testscript [if it exists].
>    2. Change the test ``execution plan'' so that they setup and
>teardown
>inexplicably executed in the following manner (following Java's
>unit-test
>and python unittest):
>
>    Setup the test; if successful, continue on with the test(s) and
>teardown
>at the end.
>
    The purpose for making this into a script as opposed to any other
>sort
>of metadata, is that anyone executing the tests standalone will only
>need to
>source the script to set everything up, then execute the test itself.
>    The only thing that might be a problem is that runtest files are
>mashed
>together if multiple runtest scenario files are specified. This
>shouldn't be
>an issue though if everything is strung together properly via the calls
>above (setup, test, teardown).
>    Thoughts?

Overall nice proposal, i also could not understand this "teardown" concept.

-- 
Thanks & Regards
Rishi
LTP Maintainer
IBM, LTC, Bangalore
Please join IRC #ltp @ irc.freenode.net

------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list