[Lustre-devel] Language choice for Lustre tests

[Lustre-devel] Language choice for Lustre tests

[Nathan Rutman]

At LAD'12 we proposed a plan for improving the Lustre test framework as an important part of the Lustre quality story.  One of the discussion points there was that the bash language of the current tests was lacking in a variety of areas.  We're moving forward with this work but need community agreement on the best course.  

Given the requirements and language options below, the reasonable choices rapidly diminish to a showdown between perl and python.  I think we're leaning at this point toward perl, based on it's superior speed and inter-version compatibility.  The final piece of the puzzle is the knowledge of existing Lustre test writers, so please chime in.  (But note that "popularity" is the reason we chose bash the first time, and look where that got us...)

	requirements
		1. easy to use
		2. strict structure
		3. universally available
		4. widely maintained
		5. widely understood
		6. good filesystem interface: posix API
		7. fast - replace e.g. createmany with embedded function
		8. operate remote instances
		9. inter-version compatibility
	options
		bash - capable, but too flexible, easy to abuse
		perl - forward compatible, universal, more widely understood, xperior, compact; hard to read later
			posix::open, opendir, lseek, etc.
	parallel::MPI
			~2x faster than python
			more version compatible
		python - very clear structure, swig module for c inclusion; non-standard installations, support
			os.open: all c flags	
			MPI bindings
			tab/space requirements make remote editing more difficult
		cucumber - ruby based, difficult deployment
		java - easy deployment, dev environ, debugger, fast; must compile
		ruby - compact


http://silicainsilico.wordpress.com/2012/03/26/switching-from-perl-to-python-speed/
http://tenser.typepad.com/tenser_said_the_tensor/2006/08/python_vs_perl_.html
http://opennomad.com/content/performance-different-scripting-languages-shell-v-perl-v-python-v-ruby
http://hentenaar.com/serendipity/index.php?/archives/27-Benchmark-PHP-vs.-Python-vs.-Perl-vs.-Ruby.html


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20121023/3ba94dff/attachment.htm>

[Lustre-devel] Language choice for Lustre tests

[Jacques-Charles Lafoucriere]

hello all

I personally always had difficulties with perl ...
the code is generally very hard to read, and the language brings very 
quickly to tricky optimization only experts understood
python is a more generic language, very easy to learn and if we make the 
right basic object definitions, it will help a lot for future 
improvement and code structure
at CEA our admin tools like shine or cluster-shell are python based and 
this choice allows us in getting scallable/reliable tools
For example, the initial version of shine (~6 years ago) was perl based 
and we were pleased to move to python for the new design

The target is lustre developers (ie kernel developers), it will be much 
easier for them to learn python than perl.
it will also be easier for them to get the right "python" way of 
programming than the right "perl" way of programming
(in both cases the worst way is to use the new language as bash, which 
can arrive much quickly with perl than with python)

about python, what do you mean by non standard installation ?
if your python configuration is right, the local differences should be 
hidden to the test framework

Bye

JC

On 10/23/2012 08:36 PM, Nathan Rutman wrote:
> At LAD'12 we proposed a plan for improving the Lustre test framework 
> as an important part of the Lustre quality story.  One of the 
> discussion points there was that the bash language of the current 
> tests was lacking in a variety of areas.  We're moving forward with 
> this work but need community agreement on the best course.
>
> Given the requirements and language options below, the reasonable 
> choices rapidly diminish to a showdown between perl and python.  I 
> think we're leaning at this point toward perl, based on it's superior 
> speed and inter-version compatibility.  The final piece of the puzzle 
> is the knowledge of existing Lustre test writers, so please chime in. 
>  (But note that "popularity" is the reason we chose bash the first 
> time, and look where that got us...)
>
> requirements
> 1. easy to use
> 2. strict structure
> 3. universally available
> 4. widely maintained
> 5. widely understood
> 6. good filesystem interface: posix API
> 7. fast - replace e.g. createmany with embedded function
> 8. operate remote instances
> 9. inter-version compatibility
> options
> bash - capable, but too flexible, easy to abuse
> perl - forward compatible, universal, more widely understood, xperior, 
> compact; hard to read later
> posix::open, opendir, lseek, etc.
> parallel::MPI
> ~2x faster than python
> more version compatible
> python - very clear structure, swig module for c inclusion; 
> non-standard installations, support
> os.open: all c flags
> MPI bindings
> tab/space requirements make remote editing more difficult
> cucumber - ruby based, difficult deployment
> java - easy deployment, dev environ, debugger, fast; must compile
> ruby - compact
>
>
> http://silicainsilico.wordpress.com/2012/03/26/switching-from-perl-to-python-speed/
> http://tenser.typepad.com/tenser_said_the_tensor/2006/08/python_vs_perl_.html
> http://opennomad.com/content/performance-different-scripting-languages-shell-v-perl-v-python-v-ruby
> http://hentenaar.com/serendipity/index.php?/archives/27-Benchmark-PHP-vs.-Python-vs.-Perl-vs.-Ruby.html
>
>
>
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20121023/6616133b/attachment.htm>

[Lustre-devel] Language choice for Lustre tests

[Kilian Cavalotti]

Hi,

On Tue, Oct 23, 2012 at 8:36 PM, Nathan Rutman
<nathan_rutman@xyratex.com> wrote:
> Given the requirements and language options below, the reasonable choices
> rapidly diminish to a showdown between perl and python.  I think we're
> leaning at this point toward perl, based on it's superior speed and
> inter-version compatibility.

I myself tend towards Python, but that's perfectly knowing this
couldn't be very different from the recurring, almost religious "emacs
vs vim" or "kde vs gnome" flame wars. It's a matter of taste, and it's
almost pointless to try to convince somebody to adopt the other one
once their choice is made.

I would only recommend reading this :
http://www.strombergers.com/python/python_perl_slurp_file.html.

And the obligatory XKCD: http://xkcd.com/353/

Cheers,
-- 
Kilian

[Lustre-devel] Language choice for Lustre tests

[Roman Grigoryev]

Hi JC, all,

I should say more words about compatibility.

In comparing with server-only tools(which are often pretty good
maintained and controlled
on limited node set),  test tools often work on wider sets. Minimally,
it should work
on some clients and  also on clients with different versions then
servers, include clients in real clusters,
developers virtual clusters and so on.

As I know, Lustre users could have environment with Lustre latest
servers and  1.8 clients,
some companies use RedHat5.x clients and RedHat6.x(SL6.x) server, other
use Ubuntu. 

RH5 has only python2.4, SL61  has python2.4 and python2.6, and looks
like only
last Fedora will have python3. In same time, Ubuntu says that from next
release want to have only Python 3.
Which version should we use and how long backward compatibility will be
supported by Python and distros//for
selected version/?/

Precision Python version could be installed from non standard repos,
compiled from sources also as used
"non standard installation". Last item also mean testing own
installation on wide set of distros. Also we should
remember about external Python libraries which also could be touched by
breaking legacy compatibility.

I think, testing system should be friendly to developers as possible and
pushing to install precision version while
one or more pythons already in os could not be the simplest solution.

Now Lustre tests compatibility for wide set of system is solved by shell
and standard utilities. Perl also has great
compatibility history, many scripts could work on latest version as 10
years ago. It is reason why I see Perl as
good decision.

From compatibility(and my) point of view, also Java is preferable
solution then Python. It has few good described ways
of installation, proved compatibility, great library managers (maven,
ant+ivy) and could support scripting
languages(JPython, JRuby and more).
But it needs more memory and pretty big start time.  

Thanks,
    Roman
On 10/23/2012 11:39 PM, Jacques-Charles Lafoucriere wrote:
> hello all
.......
> about python, what do you mean by non standard installation ?
> if your python configuration is right, the local differences should be
> hidden to the test framework
>
> Bye
>
> JC
>
> On 10/23/2012 08:36 PM, Nathan Rutman wrote:
>> At LAD'12 we proposed a plan for improving the Lustre test framework
>> as an important part of the Lustre quality story.  One of the
>> discussion points there was that the bash language of the current
>> tests was lacking in a variety of areas.  We're moving forward with
>> this work but need community agreement on the best course.  
>>
>> Given the requirements and language options below, the reasonable
>> choices rapidly diminish to a showdown between perl and python.  I
>> think we're leaning at this point toward perl, based on it's superior
>> speed and inter-version compatibility.  The final piece of the puzzle
>> is the knowledge of existing Lustre test writers, so please chime in.
>>  (But note that "popularity" is the reason we chose bash the first
>> time, and look where that got us...)
>>
>> requirements
>> 1. easy to use
>> 2. strict structure
>> 3. universally available
>> 4. widely maintained
>> 5. widely understood
>> 6. good filesystem interface: posix API
>> 7. fast - replace e.g. createmany with embedded function
>> 8. operate remote instances
>> 9. inter-version compatibility
>> options
>> bash - capable, but too flexible, easy to abuse
>> perl - forward compatible, universal, more widely understood,
>> xperior, compact; hard to read later
>> posix::open, opendir, lseek, etc.
>> parallel::MPI
>> ~2x faster than python
>> more version compatible
>> python - very clear structure, swig module for c inclusion;
>> non-standard installations, support
>> os.open: all c flags
>> MPI bindings
>> tab/space requirements make remote editing more difficult
>> cucumber - ruby based, difficult deployment
>> java - easy deployment, dev environ, debugger, fast; must compile
>> ruby - compact
>>
>>
>> http://silicainsilico.wordpress.com/2012/03/26/switching-from-perl-to-python-speed/
>> http://tenser.typepad.com/tenser_said_the_tensor/2006/08/python_vs_perl_.html
>> http://opennomad.com/content/performance-different-scripting-languages-shell-v-perl-v-python-v-ruby
>> http://hentenaar.com/serendipity/index.php?/archives/27-Benchmark-PHP-vs.-Python-vs.-Perl-vs.-Ruby.html
>>
>>
>>
>>
>> _______________________________________________
>> Lustre-devel mailing list
>> Lustre-devel at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20121024/75f5193c/attachment.htm>

[Lustre-devel] Language choice for Lustre tests

[Kilian Cavalotti]

Hi Roman,

On Wed, Oct 24, 2012 at 8:31 AM, Roman Grigoryev
<roman_grigoryev@xyratex.com> wrote:
> RH5 has only python2.4, SL61  has python2.4 and python2.6, and looks like
> only last Fedora will have python3.

I agree that inter-version compatibility could be a problem with
Python. But to take this argument literally, the best way to avoid
compatibility issues is to use the widest-spread and most version
consistent language across supported distributions, ie. bash.

> In same time, Ubuntu says that from next
> release want to have only Python 3.

Since only RHEL, CentOS and SLES are supported, should we really care
about python versions in Ubuntu?
When installing Lustre in Ubuntu, users already have to do their own
packaging, so I'm not sure that having to install a specific version
of a scripting language would make much of a difference.

> Now Lustre tests compatibility for wide set of system is solved by shell and
> standard utilities. Perl also has great
> compatibility history, many scripts could work on latest version as 10 years
> ago. It is reason why I see Perl as
> good decision.

So what's wrong with bash again? If it's just "too flexible, easy to
abuse", I'm afraid there's little in Perl to prevent falling into the
same pitfalls. :)

Cheers,
-- 
Kilian

[Lustre-devel] Language choice for Lustre tests

[Roman Grigoryev]

Hi Kilian,

if you want I could try to explain why not bash from my point of view.

I agree that bash is language which could be used for test frameworks
and test-framework.sh prove it. But some
bash feature make current framework support pretty hard. For example, I
don't see good way to  use  bash unit tests for test-framework.sh

More structured requests:

1) Language features. Perl/Python/Java has many features which allows
simple write complex logic (OOP,AOP and so on)
2) Pretty wide set of libraries.
3) powerful and useful unit test frameworks.
4) support tools, e.g. inline documentation, schema generator based on
code, coverage collectiors, copyright checkers

Ubuntu is building by Intel
(http://build.whamcloud.com/job/lustre-b2_3/), so looks like it is
important platform.

Thanks,
    Roman
> Hi Roman,
>
> On Wed, Oct 24, 2012 at 8:31 AM, Roman Grigoryev
> <roman_grigoryev@xyratex.com> wrote:
>> RH5 has only python2.4, SL61  has python2.4 and python2.6, and looks like
>> only last Fedora will have python3.
> I agree that inter-version compatibility could be a problem with
> Python. But to take this argument literally, the best way to avoid
> compatibility issues is to use the widest-spread and most version
> consistent language across supported distributions, ie. bash.
>
>> In same time, Ubuntu says that from next
>> release want to have only Python 3.
> Since only RHEL, CentOS and SLES are supported, should we really care
> about python versions in Ubuntu?
> When installing Lustre in Ubuntu, users already have to do their own
> packaging, so I'm not sure that having to install a specific version
> of a scripting language would make much of a difference.
>
>> Now Lustre tests compatibility for wide set of system is solved by shell and
>> standard utilities. Perl also has great
>> compatibility history, many scripts could work on latest version as 10 years
>> ago. It is reason why I see Perl as
>> good decision.
> So what's wrong with bash again? If it's just "too flexible, easy to
> abuse", I'm afraid there's little in Perl to prevent falling into the
> same pitfalls. :)
>
> Cheers,

[Lustre-devel] Language choice for Lustre tests

[DEGREMONT Aurelien]

Hello

I've developed in Perl and Python for several years in both languages. I 
enjoyed both.
Regarding starting a new project, for testing framework, here is several 
points which are the key points for choosing the right language.

- Longevity

Perl as a long history behind it, is available in all distros, that's 
true. But Perl is there for compatibility for running projects which 
were starting a long time ago. Nobody is starting a really big and fancy 
project in Perl nowadays. All new hackers are only speaking of Python or 
Ruby. You will not attract contributors with Perl. Perl 5 is very 
compatible because no new changes is really added to this language.
And do not tell me that Perl 6 is coming. If Perl 6 is an option, then 
there is no issue with Python 3 and forget your compatibility.

- Compat

Developping in Python, for a large number of environment means coding 
for Python 2.4+. This will run nicely on Python up to 2.7
As long as only Python 2 only is concerned, compat for Python is fine.
But I agree that Python 3 is coming, and at long term, it should be 
taken in account even if no major distro is using it right now.
(No Python 3 before RHEL7 or next Ubuntu LTS (14.04))

-Lustre is for system people

Lustre is developped by system guys, which mainly use vim or emacs to 
develop in Lustre. Very few of them are using IDE or stuff like Maven. 
Population developing Java with Maven is the exact opposite of guys 
coding in C, for Kernel code. Moreover, JVM are awful to install 
regarding a standard Python, Perl or Ruby interpreter in all major distro.


My choice would go for Python 2 with a clear path to migrate to Python 3 
when needed (in 1 year)


Aur?lien



Le 24/10/2012 10:44, Roman Grigoryev a ?crit :
> Hi Kilian,
>
> if you want I could try to explain why not bash from my point of view.
>
> I agree that bash is language which could be used for test frameworks
> and test-framework.sh prove it. But some
> bash feature make current framework support pretty hard. For example, I
> don't see good way to  use  bash unit tests for test-framework.sh
>
> More structured requests:
>
> 1) Language features. Perl/Python/Java has many features which allows
> simple write complex logic (OOP,AOP and so on)
> 2) Pretty wide set of libraries.
> 3) powerful and useful unit test frameworks.
> 4) support tools, e.g. inline documentation, schema generator based on
> code, coverage collectiors, copyright checkers
>
> Ubuntu is building by Intel
> (http://build.whamcloud.com/job/lustre-b2_3/), so looks like it is
> important platform.
>
> Thanks,
>      Roman
>> Hi Roman,
>>
>> On Wed, Oct 24, 2012 at 8:31 AM, Roman Grigoryev
>> <roman_grigoryev@xyratex.com>  wrote:
>>> RH5 has only python2.4, SL61  has python2.4 and python2.6, and looks like
>>> only last Fedora will have python3.
>> I agree that inter-version compatibility could be a problem with
>> Python. But to take this argument literally, the best way to avoid
>> compatibility issues is to use the widest-spread and most version
>> consistent language across supported distributions, ie. bash.
>>
>>> In same time, Ubuntu says that from next
>>> release want to have only Python 3.
>> Since only RHEL, CentOS and SLES are supported, should we really care
>> about python versions in Ubuntu?
>> When installing Lustre in Ubuntu, users already have to do their own
>> packaging, so I'm not sure that having to install a specific version
>> of a scripting language would make much of a difference.
>>
>>> Now Lustre tests compatibility for wide set of system is solved by shell and
>>> standard utilities. Perl also has great
>>> compatibility history, many scripts could work on latest version as 10 years
>>> ago. It is reason why I see Perl as
>>> good decision.
>> So what's wrong with bash again? If it's just "too flexible, easy to
>> abuse", I'm afraid there's little in Perl to prevent falling into the
>> same pitfalls. :)
>>
>> Cheers,
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

[Lustre-devel] Language choice for Lustre tests

[Gearing, Chris]

Nathan,

I'm not 100% sure what you are proposing here, your LAD presentation suggested a 'tune-up' of the current test framework rather than a complete re-write. Which of the two are we discussing?

Which ever you are intending to undertake I think it is vital that before decisions are made on things such as languages a clear specification/definition of the activity is created and distributed. I am in fact working with Roman to bring such a document to the working group on the tune-up, making me somewhat confused about what is proposed here.

Going back to your LAD presentation for the tune-up your intention is to write some libraries to better address the requirements of the framework than is possible in bash, but those libraries would be called from the existing bash tests. This illustrates that the language used to develop the framework may not be the same language used for writing the tests, in fact I'm guessing that for both a tune-up and rewrite the requirements placed on the framework writer are different than those place on the test writer and so the languages required might well be different, and in fact if this is a tune-up then surely the tests must continue to be written in bash even if the libraries are in something more applicable, we do not want a single framework with mixed languages (do we?)

So to return to my original question could you please provide some greater depth of insight as to what you are embarking upon, and then we can make more objective input as the language required.

On the complete-rewrite topic, I will post the slides I sent you before LAD to the Whamcloud wiki and then provide a link to this list so that people can share our thoughts. (Network access prevents me doing this immediately)

Regards

Chris
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[Lustre-devel] Language choice for Lustre tests

[Nathan Rutman]

On Oct 24, 2012, at 1:02 PM, "Gearing, Chris" <chris.gearing@intel.com> wrote:

> Nathan,
> 
> I'm not 100% sure what you are proposing here, your LAD presentation suggested a 'tune-up' of the current test framework rather than a complete re-write. Which of the two are we discussing?
Both...
> 
> Which ever you are intending to undertake I think it is vital that before decisions are made on things such as languages a clear specification/definition of the activity is created and distributed. I am in fact working with Roman to bring such a document to the working group on the tune-up, making me somewhat confused about what is proposed here.
> 
> Going back to your LAD presentation for the tune-up your intention is to write some libraries to better address the requirements of the framework than is possible in bash, but those libraries would be called from the existing bash tests. This illustrates that the language used to develop the framework may not be the same language used for writing the tests, in fact I'm guessing that for both a tune-up and rewrite the requirements placed on the framework writer are different than those place on the test writer and so the languages required might well be different,
> and in fact if this is a tune-up then surely the tests must continue to be written in bash even if the libraries are in something more applicable, we do not want a single framework with mixed languages (do we?)

The ultimate goal is to produce higher-quality, more robust, well-controlled, and safer tests.  To that end, I think the eventual language of the tests must change to something meeting the requirements I stated before:
		1. easy to use
		2. strict structure
		3. universally available/portable
		4. widely maintained
		5. widely understood
		6. fully-featured filesystem interface: posix API
		7. fast - replace e.g. createmany with embedded function
		8. operate remote instances
		9. inter-version compatibility
The requirements on the framework language are more relaxed, but for ease of development and developer sanity, I assume that the framework language should match the test language.  So I'm using the test language as the requirements driver, and to gage community preference for that test language.

Based on the responses so far, it seems that there is a fairly clear preference for Python as a test language, and so I'll propose that Python should be used shorter-term to start replacing test-framework.


> So to return to my original question could you please provide some greater depth of insight as to what you are embarking upon, and then we can make more objective input as the language required.
> 
> On the complete-rewrite topic, I will post the slides I sent you before LAD to the Whamcloud wiki and then provide a link to this list so that people can share our thoughts. (Network access prevents me doing this immediately)
> 
> Regards
> 
> Chris
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ
> VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
> 
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20121024/65a3879f/attachment.htm>

[Lustre-devel] Language choice for Lustre tests

[Colin Faber]

My vote here is perl, there's a few [lustre] tools already written in 
perl, it also meets all of the stated requirements. Yes, perl can look 
like core dump. Yes perl can be easily obfuscated and impossibly hard to 
understand (even for expert class developers). However the enormous 
range of module support, as well as the maturity and cross platform 
portability make it ideal for this task.

-cf

On 10/24/2012 04:05 PM, Nathan Rutman wrote:
>
> On Oct 24, 2012, at 1:02 PM, "Gearing, Chris" <chris.gearing@intel.com 
> <mailto:chris.gearing@intel.com>> wrote:
>
>> Nathan,
>>
>> I'm not 100% sure what you are proposing here, your LAD presentation 
>> suggested a 'tune-up' of the current test framework rather than a 
>> complete re-write. Which of the two are we discussing?
> Both...
>>
>> Which ever you are intending to undertake I think it is vital that 
>> before decisions are made on things such as languages a clear 
>> specification/definition of the activity is created and distributed. 
>> I am in fact working with Roman to bring such a document to the 
>> working group on the tune-up, making me somewhat confused about what 
>> is proposed here.
>>
>> Going back to your LAD presentation for the tune-up your intention is 
>> to write some libraries to better address the requirements of the 
>> framework than is possible in bash, but those libraries would be 
>> called from the existing bash tests. This illustrates that the 
>> language used to develop the framework may not be the same language 
>> used for writing the tests, in fact I'm guessing that for both a 
>> tune-up and rewrite the requirements placed on the framework writer 
>> are different than those place on the test writer and so the 
>> languages required might well be different,
>> and in fact if this is a tune-up then surely the tests must continue 
>> to be written in bash even if the libraries are in something more 
>> applicable, we do not want a single framework with mixed languages 
>> (do we?)
>
> The ultimate goal is to produce higher-quality, more robust, 
> well-controlled, and safer tests.  To that end, I think the eventual 
> language of the tests must change to something meeting the 
> requirements I stated before:
> 1. easy to use
> 2. strict structure
> 3. universally available/portable
> 4. widely maintained
> 5. widely understood
> 6. fully-featured filesystem interface: posix API
> 7. fast - replace e.g. createmany with embedded function
> 8. operate remote instances
> 9. inter-version compatibility
> The requirements on the framework language are more relaxed, but for 
> ease of development and developer sanity, I assume that the framework 
> language should match the test language.  So I'm using the test 
> language as the requirements driver, and to gage community preference 
> for that test language.
>
> Based on the responses so far, it seems that there is a fairly clear 
> preference for Python as a test language, and so I'll propose that 
> Python should be used shorter-term to start replacing test-framework.
>
>
>> So to return to my original question could you please provide some 
>> greater depth of insight as to what you are embarking upon, and then 
>> we can make more objective input as the language required.
>>
>> On the complete-rewrite topic, I will post the slides I sent you 
>> before LAD to the Whamcloud wiki and then provide a link to this list 
>> so that people can share our thoughts. (Network access prevents me 
>> doing this immediately)
>>
>> Regards
>>
>> Chris
>> ---------------------------------------------------------------------
>> Intel Corporation (UK) Limited
>> Registered No. 1134945 (England)
>> Registered Office: Pipers Way, Swindon SN3 1RJ
>> VAT No: 860 2173 47
>>
>> This e-mail and any attachments may contain confidential material for
>> the sole use of the intended recipient(s). Any review or distribution
>> by others is strictly prohibited. If you are not the intended
>> recipient, please contact the sender and delete all copies.
>>
>> _______________________________________________
>> Lustre-devel mailing list
>> Lustre-devel at lists.lustre.org <mailto:Lustre-devel@lists.lustre.org>
>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>
>
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

[Lustre-devel] Language choice for Lustre tests

[Brian Behlendorf]

On Wed, 2012-10-24 at 15:05 -0700, Nathan Rutman wrote:
> On Oct 24, 2012, at 1:02 PM, "Gearing, Chris" <chris.gearing@intel.com> wrote:
> 
> > Nathan,
> > 
> > I'm not 100% sure what you are proposing here, your LAD presentation suggested a 'tune-up' of the current test framework rather than a complete re-write. Which of the two are we discussing?
> Both...
> > 
> The requirements on the framework language are more relaxed, but for ease of development and developer sanity, I assume that the framework language should match the test language.  So I'm using the test language as the requirements driver, and to gage community preference for that test language.

Before embarking on building yet another new and custom framework for
Lustre we should evaluate some existing frameworks.  For example, the
Autotest project was specifically designed to test the Linux kernel.
It's open source, looks active, is flexible, and there is detailed
documentation on how to write tests.  Plus it was designed specifically
for testing the kernel so there are likely existing file system tests.

  http://autotest.github.com/

  "Autotest is a framework for fully automated testing. It is
   designed primarily to test the Linux kernel, though it is useful
   for many other functions such as qualifying new hardware. It's an
   open-source project under the GPL and is used and developed by a
   number of organizations, including Google, IBM, Red Hat, and many
   others."

> Based on the responses so far, it seems that there is a fairly clear preference for Python as a test language, and so I'll propose that Python should be used shorter-term to start replacing test-framework.

If we decide the Autotest framework is a good fit then we'll want to
write the tests in python to be consistent with the framework language.
However, for a first cut it looks like you could use the existing bash
tests largely unmodified.

-- 
Thanks,
Brian

[Lustre-devel] Language choice for Lustre tests

[Nathan Rutman]

There are several levels of "test framework" that are involved with automated testing of Lustre:
unit tests themselves, written in bash, e.g. sanity 1a
test-framework.sh & friends, which provides some support/setup/communal library for unit tests
automated test systems, which automatically execute the unit tests, record and report the results.

I'm only speaking about the language for the first of these items at the moment.
For more context, including the motivation for changing things, see my LAD'12 presentation (more narrowly focused) on the OpenSFS Wiki.



On Oct 25, 2012, at 10:23 AM, Brian Behlendorf <behlendorf1@llnl.gov> wrote:

> On Wed, 2012-10-24 at 15:05 -0700, Nathan Rutman wrote:
>> On Oct 24, 2012, at 1:02 PM, "Gearing, Chris" <chris.gearing@intel.com> wrote:
>> 
>>> Nathan,
>>> 
>>> I'm not 100% sure what you are proposing here, your LAD presentation suggested a 'tune-up' of the current test framework rather than a complete re-write. Which of the two are we discussing?
>> Both...
>>> 
>> The requirements on the framework language are more relaxed, but for ease of development and developer sanity, I assume that the framework language should match the test language.  So I'm using the test language as the requirements driver, and to gage community preference for that test language.
> 
> Before embarking on building yet another new and custom framework for
> Lustre we should evaluate some existing frameworks.  For example, the
> Autotest project was specifically designed to test the Linux kernel.
> It's open source, looks active, is flexible, and there is detailed
> documentation on how to write tests.  Plus it was designed specifically
> for testing the kernel so there are likely existing file system tests.
> 
>  http://autotest.github.com/
> 
>  "Autotest is a framework for fully automated testing. It is
>   designed primarily to test the Linux kernel, though it is useful
>   for many other functions such as qualifying new hardware. It's an
>   open-source project under the GPL and is used and developed by a
>   number of organizations, including Google, IBM, Red Hat, and many
>   others."
> 
>> Based on the responses so far, it seems that there is a fairly clear preference for Python as a test language, and so I'll propose that Python should be used shorter-term to start replacing test-framework.
> 
> If we decide the Autotest framework is a good fit then we'll want to
> write the tests in python to be consistent with the framework language.
> However, for a first cut it looks like you could use the existing bash
> tests largely unmodified.
> 
> -- 
> Thanks,
> Brian
> 
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20121025/b8b4b1dd/attachment.htm>

[Lustre-devel] Language choice for Lustre tests

[Gearing, Chris]

Hi,

> Based on the responses so far, it seems that there is a fairly clear preference for Python as a test language, and so I'll propose that Python should be used shorter-term to start replacing test-framework.

I don't think that is the case Nathan, because unless I am mistaken

Jacques-Charles, Kilian Cavalotti and if it is a binary choice today my preference is Python (but please see below)
Bruce was neutral
Roman and Colin (after your email) expressed Perl as a choice.

So the preference would seem to be for Python, counting by people and organization although obviously it's a statistically suspect poll.

Chris

NOTES OF MY VOTE: Given the importance of the topic I believe a much broader piece of analysis needs to be done to select the right language, also if this is for an existing framework update the tests must all be in bash or all converted to the new language of choice.

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20121025/a567906a/attachment.htm>

[Lustre-devel] Language choice for Lustre tests

[Gearing, Chris]

Hi,

> On the complete-rewrite topic, I will post the slides I sent you before LAD to the Whamcloud wiki and then provide a link to this list so that people can share our thoughts. (Network access prevents me doing this immediately)

As promised:  http://wiki.whamcloud.com/download/attachments/7634975/new+test+framework.pdf

Regards

Chris

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[Lustre-devel] Language choice for Lustre tests

[Nathan Rutman]

On Oct 25, 2012, at 11:20 AM, "Gearing, Chris" <chris.gearing@intel.com> wrote:

> Hi,
> 
> > Based on the responses so far, it seems that there is a fairly clear preference for Python as a test language, and so I'll propose that Python should be used shorter-term to start replacing test-framework.
> 
> I don?t think that is the case Nathan, because unless I am mistaken
>  
> Jacques-Charles, Kilian Cavalotti and if it is a binary choice today my preference is Python (but please see below)
Aurelien - python 2

> Bruce was neutral
> Roman and Colin (after your email) expressed Perl as a choice.
>  
> So the preference would seem to be for Python, counting by people and organization although obviously it?s a statistically suspect poll.
Indeed.  As I stated before, a poll is not the best way to choose the language; it's just one data point.  But as a) we must choose something to start making any headway toward improvements, and b) bash doesn't meet the requirements, I'm trying to come up with some kind of baseline.  

>  
> Chris
> 
> NOTES OF MY VOTE: Given the importance of the topic I believe a much broader piece of analysis needs to be done to select the right language,
I'm just trying to keep things moving from theoretical optimum to practical conclusion.  We feel a need today for much better Lustre testing, and there's a huge pile of work waiting, and so I feel we must move forward.

> also if this is for an existing framework update the tests must all be in bash or all converted to the new language of choice.
Short term the idea would be to translate one of the test suites, for example conf-sanity, to the new language/framework.  The two systems (old and new) would live side-by-side for some period.  New tests should be implemented under the new system (e.g. sanity2).  Other old tests could be converted piecemeal subsequently.


>  
> ---------------------------------------------------------------------
> Intel Corporation (UK) Limited
> Registered No. 1134945 (England)
> Registered Office: Pipers Way, Swindon SN3 1RJ
> VAT No: 860 2173 47
> 
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20121025/e810c4b7/attachment.htm>

[Lustre-devel] Language choice for Lustre tests

[Christopher J. Morrone]

It really sounds like you are putting the cart before the horse. 
Picking a favorite language is probably a bad way to go.  First we 
should figure out why we're changing things, and what the scope and 
goals of the new design are.

Brian Behlendorf pointed out the kernel Autotest system as an option, 
which sounds worthy of investigation to me.  If we were to decide on 
Perl today, that would make working with that Autotest more difficult, 
because that is all done in Python.  But if choose Python and then try 
to use an entirely Visual Basic based framework (Ha! I kid!) we also 
probably made the wrong choice.

In other words, it is important to know something about where we are 
going before we pick a language.

On 10/25/2012 12:24 PM, Nathan Rutman wrote:
>
> On Oct 25, 2012, at 11:20 AM, "Gearing, Chris" <chris.gearing@intel.com
> <mailto:chris.gearing@intel.com>> wrote:
>
>> Hi,
>>
>> >Based on the responses so far, it seems that there is a fairly clear
>> preference for Python as a test language, and so I'll propose that
>> Python should be used shorter-term to start replacing test-framework.
>>
>> I don?t think that is the case Nathan, because unless I am mistaken
>> Jacques-Charles, Kilian Cavalotti and if it is a binary choice today
>> my preference is Python (but please see below)
> Aurelien - python 2
>
>> Bruce was neutral
>> Roman and Colin (after your email) expressed Perl as a choice.
>> So the preference would seem to be for Python, counting by people and
>> organization although obviously it?s a statistically suspect poll.
> Indeed.  As I stated before, a poll is not the best way to choose the
> language; it's just one data point.  But as a) we must choose something
> to start making any headway toward improvements, and b) bash doesn't
> meet the requirements, I'm trying to come up with some kind of baseline.
>
>> Chris
>>
>> NOTES OF MY VOTE: Given the importance of the topic I believe a much
>> broader piece of analysis needs to be done to select the right language,
> I'm just trying to keep things moving from theoretical optimum to
> practical conclusion.  We feel a need today for much better Lustre
> testing, and there's a huge pile of work waiting, and so I feel we must
> move forward.
>
>> also if this is for an existing framework update the tests must all be
>> in bash or all converted to the new language of choice.
> Short term the idea would be to translate one of the test suites, for
> example conf-sanity, to the new language/framework.  The two systems
> (old and new) would live side-by-side for some period.  New tests should
> be implemented under the new system (e.g. sanity2).  Other old tests
> could be converted piecemeal subsequently.
>
>
>> ---------------------------------------------------------------------
>> Intel Corporation (UK) Limited
>> Registered No. 1134945 (England)
>> Registered Office: Pipers Way, Swindon SN3 1RJ
>> VAT No: 860 2173 47
>>
>> This e-mail and any attachments may contain confidential material for
>> the sole use of the intended recipient(s). Any review or distribution
>> by others is strictly prohibited. If you are not the intended
>> recipient, please contact the sender and delete all copies.
>>
>
>
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel
>

[Lustre-devel] Language choice for Lustre tests

[Prakash Surya]

On Thu, Oct 25, 2012 at 11:04:55AM -0700, Nathan Rutman wrote:
>    There are several levels of "test framework" that are involved with
>    automated testing of Lustre:
> 
>      * unit tests themselves, written in bash, e.g. sanity 1a

What do you mean by "unit tests" here? Because as I understand it,
Lustre has no unit testing at all. It is all functional testing,
which is related but completely different.

>      * test-framework.sh & friends, which provides some
>        support/setup/communal library for unit tests

Again, unit tests? I really wish there _were_ unit tests, but aren't,
and I haven't seen any discussion about creating them.

>      * automated test systems, which automatically execute the unit tests,
>        record and report the results.

I don't quite understand why there is so much discussion regarding a new
testing framework specifically designed for testing Lustre. Is there
really nothing already out there we can make use of?

Brian linked to the Autotest framework, does that not do what we want?
If it doesn't work, why not? And how is our home brewed solution going
to differ? Are there any other alternatives out there? I don't see
benefit in spending development on a new framework if it's unnecessary.

There is a lot of discussion regarding a tool (the language) and little
involving the problem (lack of useful functional and unit tests to
properly stress the system). Pick something, move on, and start
improving the codes stability.

And if somebody *really* wants to build a new testing framework, have it
live out of tree. Create it as a standalone project which interfaces
with Lustre.

-- 
Cheers, Prakash


> 
>    I'm only speaking about the language for the first of these items at
>    the moment.
>    For more context, including the motivation for changing things, see my
>    LAD'12 presentation (more narrowly focused) on the [1]OpenSFS Wiki.
>    On Oct 25, 2012, at 10:23 AM, Brian Behlendorf
>    <[2]behlendorf1@llnl.gov> wrote:
> 
>      On Wed, 2012-10-24 at 15:05 -0700, Nathan Rutman wrote:
> 
>      On Oct 24, 2012, at 1:02 PM, "Gearing, Chris"
>      <[3]chris.gearing@intel.com> wrote:
> 
>      Nathan,
>      I'm not 100% sure what you are proposing here, your LAD presentation
>      suggested a 'tune-up' of the current test framework rather than a
>      complete re-write. Which of the two are we discussing?
> 
>      Both...
> 
>      The requirements on the framework language are more relaxed, but for
>      ease of development and developer sanity, I assume that the
>      framework language should match the test language.  So I'm using the
>      test language as the requirements driver, and to gage community
>      preference for that test language.
> 
>      Before embarking on building yet another new and custom framework
>      for
>      Lustre we should evaluate some existing frameworks.  For example,
>      the
>      Autotest project was specifically designed to test the Linux kernel.
>      It's open source, looks active, is flexible, and there is detailed
>      documentation on how to write tests.  Plus it was designed
>      specifically
>      for testing the kernel so there are likely existing file system
>      tests.
>       [4]http://autotest.github.com/
>       "Autotest is a framework for fully automated testing. It is
>        designed primarily to test the Linux kernel, though it is useful
>        for many other functions such as qualifying new hardware. It's an
>        open-source project under the GPL and is used and developed by a
>        number of organizations, including Google, IBM, Red Hat, and many
>        others."
> 
>      Based on the responses so far, it seems that there is a fairly clear
>      preference for Python as a test language, and so I'll propose that
>      Python should be used shorter-term to start replacing
>      test-framework.
> 
>      If we decide the Autotest framework is a good fit then we'll want to
>      write the tests in python to be consistent with the framework
>      language.
>      However, for a first cut it looks like you could use the existing
>      bash
>      tests largely unmodified.
>      --
>      Thanks,
>      Brian
>      _______________________________________________
>      Lustre-devel mailing list
>      [5]Lustre-devel at lists.lustre.org
>      http://lists.lustre.org/mailman/listinfo/lustre-devel
> 
> References
> 
>    1. http://www.opensfs.org/foswiki/bin/view/Lustre/TestingImprovements
>    2. mailto:behlendorf1 at llnl.gov
>    3. mailto:chris.gearing at intel.com
>    4. http://autotest.github.com/
>    5. mailto:Lustre-devel at lists.lustre.org

> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

[Lustre-devel] Language choice for Lustre tests

[Nathan Rutman]

On Oct 25, 2012, at 2:13 PM, Christopher J. Morrone <morrone2@llnl.gov> wrote:

> It really sounds like you are putting the cart before the horse. Picking a favorite language is probably a bad way to go.  First we should figure out why we're changing things, and what the scope and goals of the new design are.
> 
> Brian Behlendorf pointed out the kernel Autotest system as an option, which sounds worthy of investigation to me.  If we were to decide on Perl today, that would make working with that Autotest more difficult, because that is all done in Python.  But if choose Python and then try to use an entirely Visual Basic based framework (Ha! I kid!) we also probably made the wrong choice.
> 
> In other words, it is important to know something about where we are going before we pick a language.

As I said before:
> For more context, including the motivation for changing things, see my LAD'12 presentation (more narrowly focused) on the OpenSFS Wiki.

Also, see Chris' preso on a language:
> As promised:  http://wiki.whamcloud.com/download/attachments/7634975/new+test+framework.pdf


Picking a language is just one part of it, and definitely not the most important part.  I'll be posting more on some other proposed changes in the future.


On Oct 25, 2012, at 10:23 AM, Brian Behlendorf <behlendorf1@llnl.gov> wrote:

> On Wed, 2012-10-24 at 15:05 -0700, Nathan Rutman wrote:
>> On Oct 24, 2012, at 1:02 PM, "Gearing, Chris" <chris.gearing@intel.com> wrote:
>> 
>>> Nathan,
>>> 
>>> I'm not 100% sure what you are proposing here, your LAD presentation suggested a 'tune-up' of the current test framework rather than a complete re-write. Which of the two are we discussing?
>> Both...
>>> 
>> The requirements on the framework language are more relaxed, but for ease of development and developer sanity, I assume that the framework language should match the test language.  So I'm using the test language as the requirements driver, and to gage community preference for that test language.
> 
> Before embarking on building yet another new and custom framework for
> Lustre we should evaluate some existing frameworks.  For example, the
> Autotest project was specifically designed to test the Linux kernel.
> It's open source, looks active, is flexible, and there is detailed
> documentation on how to write tests.  Plus it was designed specifically
> for testing the kernel so there are likely existing file system tests.
> 
>  http://autotest.github.com/
> 
>  "Autotest is a framework for fully automated testing. It is
>   designed primarily to test the Linux kernel, though it is useful
>   for many other functions such as qualifying new hardware. It's an
>   open-source project under the GPL and is used and developed by a
>   number of organizations, including Google, IBM, Red Hat, and many
>   others."
> 
>> Based on the responses so far, it seems that there is a fairly clear preference for Python as a test language, and so I'll propose that Python should be used shorter-term to start replacing test-framework.
> 
> If we decide the Autotest framework is a good fit then we'll want to
> write the tests in python to be consistent with the framework language.
> However, for a first cut it looks like you could use the existing bash
> tests largely unmodified.
> 
> -- 
> Thanks,
> Brian
> 
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel



> 
> On 10/25/2012 12:24 PM, Nathan Rutman wrote:
>> 
>> On Oct 25, 2012, at 11:20 AM, "Gearing, Chris" <chris.gearing@intel.com
>> <mailto:chris.gearing@intel.com>> wrote:
>> 
>>> Hi,
>>> 
>>> >Based on the responses so far, it seems that there is a fairly clear
>>> preference for Python as a test language, and so I'll propose that
>>> Python should be used shorter-term to start replacing test-framework.
>>> 
>>> I don?t think that is the case Nathan, because unless I am mistaken
>>> Jacques-Charles, Kilian Cavalotti and if it is a binary choice today
>>> my preference is Python (but please see below)
>> Aurelien - python 2
>> 
>>> Bruce was neutral
>>> Roman and Colin (after your email) expressed Perl as a choice.
>>> So the preference would seem to be for Python, counting by people and
>>> organization although obviously it?s a statistically suspect poll.
>> Indeed.  As I stated before, a poll is not the best way to choose the
>> language; it's just one data point.  But as a) we must choose something
>> to start making any headway toward improvements, and b) bash doesn't
>> meet the requirements, I'm trying to come up with some kind of baseline.
>> 
>>> Chris
>>> 
>>> NOTES OF MY VOTE: Given the importance of the topic I believe a much
>>> broader piece of analysis needs to be done to select the right language,
>> I'm just trying to keep things moving from theoretical optimum to
>> practical conclusion.  We feel a need today for much better Lustre
>> testing, and there's a huge pile of work waiting, and so I feel we must
>> move forward.
>> 
>>> also if this is for an existing framework update the tests must all be
>>> in bash or all converted to the new language of choice.
>> Short term the idea would be to translate one of the test suites, for
>> example conf-sanity, to the new language/framework.  The two systems
>> (old and new) would live side-by-side for some period.  New tests should
>> be implemented under the new system (e.g. sanity2).  Other old tests
>> could be converted piecemeal subsequently.
>> 
>> 
>>> ---------------------------------------------------------------------
>>> Intel Corporation (UK) Limited
>>> Registered No. 1134945 (England)
>>> Registered Office: Pipers Way, Swindon SN3 1RJ
>>> VAT No: 860 2173 47
>>> 
>>> This e-mail and any attachments may contain confidential material for
>>> the sole use of the intended recipient(s). Any review or distribution
>>> by others is strictly prohibited. If you are not the intended
>>> recipient, please contact the sender and delete all copies.
>>> 
>> 
>> 
>> 
>> _______________________________________________
>> Lustre-devel mailing list
>> Lustre-devel at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-devel
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20121025/f5e37233/attachment.htm>

[Lustre-devel] Language choice for Lustre tests

[Nathan Rutman]

On Oct 25, 2012, at 2:17 PM, Prakash Surya <surya1@llnl.gov> wrote:

> On Thu, Oct 25, 2012 at 11:04:55AM -0700, Nathan Rutman wrote:
>>   There are several levels of "test framework" that are involved with
>>   automated testing of Lustre:
>> 
>>     * unit tests themselves, written in bash, e.g. sanity 1a
> 
> What do you mean by "unit tests" here? Because as I understand it,
> Lustre has no unit testing at all. It is all functional testing,
> which is related but completely different.
Let's please not get into this.  I just meant the individual tests.

> 
>>     * test-framework.sh & friends, which provides some
>>       support/setup/communal library for unit tests
>>     * automated test systems, which automatically execute the unit tests,
>>       record and report the results.
> 
> I don't quite understand why there is so much discussion regarding a new
> testing framework specifically designed for testing Lustre. Is there
> really nothing already out there we can make use of?
> 
> Brian linked to the Autotest framework, does that not do what we want?

That covers the third meaning, but not the first two.   I'm only concerned about the
first two at the moment. 

> If it doesn't work, why not?
It doesn't contain a Lustre-specific functional library, it doesn't safely limit the tests
that may be run on a specific system, it doesn't handle interoperability testing, etc.

> And how is our home brewed solution going
> to differ? Are there any other alternatives out there? I don't see
> benefit in spending development on a new framework if it's unnecessary.
> 
> There is a lot of discussion regarding a tool (the language) and little
> involving the problem (lack of useful functional and unit tests to
> properly stress the system). Pick something, move on, and start
> improving the codes stability.
That was what I was trying to do.

> 
> And if somebody *really* wants to build a new testing framework, have it
> live out of tree. Create it as a standalone project which interfaces
> with Lustre.
As stated in my presentation, that is part of the plan.

> 
> -- 
> Cheers, Prakash
> 
> 
>> 
>>   I'm only speaking about the language for the first of these items at
>>   the moment.
>>   For more context, including the motivation for changing things, see my
>>   LAD'12 presentation (more narrowly focused) on the [1]OpenSFS Wiki.
>>   On Oct 25, 2012, at 10:23 AM, Brian Behlendorf
>>   <[2]behlendorf1@llnl.gov> wrote:
>> 
>>     On Wed, 2012-10-24 at 15:05 -0700, Nathan Rutman wrote:
>> 
>>     On Oct 24, 2012, at 1:02 PM, "Gearing, Chris"
>>     <[3]chris.gearing@intel.com> wrote:
>> 
>>     Nathan,
>>     I'm not 100% sure what you are proposing here, your LAD presentation
>>     suggested a 'tune-up' of the current test framework rather than a
>>     complete re-write. Which of the two are we discussing?
>> 
>>     Both...
>> 
>>     The requirements on the framework language are more relaxed, but for
>>     ease of development and developer sanity, I assume that the
>>     framework language should match the test language.  So I'm using the
>>     test language as the requirements driver, and to gage community
>>     preference for that test language.
>> 
>>     Before embarking on building yet another new and custom framework
>>     for
>>     Lustre we should evaluate some existing frameworks.  For example,
>>     the
>>     Autotest project was specifically designed to test the Linux kernel.
>>     It's open source, looks active, is flexible, and there is detailed
>>     documentation on how to write tests.  Plus it was designed
>>     specifically
>>     for testing the kernel so there are likely existing file system
>>     tests.
>>      [4]http://autotest.github.com/
>>      "Autotest is a framework for fully automated testing. It is
>>       designed primarily to test the Linux kernel, though it is useful
>>       for many other functions such as qualifying new hardware. It's an
>>       open-source project under the GPL and is used and developed by a
>>       number of organizations, including Google, IBM, Red Hat, and many
>>       others."
>> 
>>     Based on the responses so far, it seems that there is a fairly clear
>>     preference for Python as a test language, and so I'll propose that
>>     Python should be used shorter-term to start replacing
>>     test-framework.
>> 
>>     If we decide the Autotest framework is a good fit then we'll want to
>>     write the tests in python to be consistent with the framework
>>     language.
>>     However, for a first cut it looks like you could use the existing
>>     bash
>>     tests largely unmodified.
>>     --
>>     Thanks,
>>     Brian
>>     _______________________________________________
>>     Lustre-devel mailing list
>>     [5]Lustre-devel at lists.lustre.org
>>     http://lists.lustre.org/mailman/listinfo/lustre-devel
>> 
>> References
>> 
>>   1. http://www.opensfs.org/foswiki/bin/view/Lustre/TestingImprovements
>>   2. mailto:behlendorf1 at llnl.gov
>>   3. mailto:chris.gearing at intel.com
>>   4. http://autotest.github.com/
>>   5. mailto:Lustre-devel at lists.lustre.org
> 
>> _______________________________________________
>> Lustre-devel mailing list
>> Lustre-devel at lists.lustre.org
>> http://lists.lustre.org/mailman/listinfo/lustre-devel
> 

[Lustre-devel] Language choice for Lustre tests

[Nathan Rutman]

Thank you all for your input; I think I've got what I need for now.
I'll continue to raise discussion points here as our plans solidify.

[Lustre-devel] Language choice for Lustre tests

[Prakash Surya]

On Thu, Oct 25, 2012 at 02:28:52PM -0700, Nathan Rutman wrote:
>    On Oct 25, 2012, at 2:13 PM, Christopher J. Morrone
>    <[1]morrone2@llnl.gov> wrote:
> 
>      It really sounds like you are putting the cart before the horse.
>      Picking a favorite language is probably a bad way to go.  First we
>      should figure out why we're changing things, and what the scope and
>      goals of the new design are.
>      Brian Behlendorf pointed out the kernel Autotest system as an
>      option, which sounds worthy of investigation to me.  If we were to
>      decide on Perl today, that would make working with that Autotest
>      more difficult, because that is all done in Python.  But if choose
>      Python and then try to use an entirely Visual Basic based framework
>      (Ha! I kid!) we also probably made the wrong choice.
>      In other words, it is important to know something about where we are
>      going before we pick a language.
> 
>    As I said before:
> 
>    For more context, including the motivation for changing things, see my
>    LAD'12 presentation (more narrowly focused) on the [2]OpenSFS Wiki.

Commenting on your presentation slides (slide 3 specifically) as I think
it relates to this discusstion:

 * "Bash languages is not modular or portable"

Why do you think this? Bash is extremely modular using the "source"
command. Write code in a file, source it in another file, and there you
go. The "sourced" file is a separate reusable module, is it not? Sure it
might be global in the namespace, but that's another issue.

Not portable? Really? Bash is available on just about every Linux
distribution that comes to mind. Definitely every one Lustre cares
about.

 * "Test scripts are not coherent"

That is not Bash's fault. Bad code can be written in any language. It's
up to the programmer to write "coherent" test scripts.

How would switching languages improve on either of these two fronts?

-- 
Cheers, Prakash

> 
>    Also, see Chris' preso on a language:
> 
>      As promised:
>      [3]http://wiki.whamcloud.com/download/attachments/7634975/new+test+f
>      ramework.pdf
> 
>    Picking a language is just one part of it, and definitely not the most
>    important part.  I'll be posting more on some other proposed changes in
>    the future.
>    On Oct 25, 2012, at 10:23 AM, Brian Behlendorf
>    <[4]behlendorf1@llnl.gov> wrote:
> 
>      On Wed, 2012-10-24 at 15:05 -0700, Nathan Rutman wrote:
> 
>      On Oct 24, 2012, at 1:02 PM, "Gearing, Chris"
>      <[5]chris.gearing@intel.com> wrote:
> 
>      Nathan,
>      I'm not 100% sure what you are proposing here, your LAD presentation
>      suggested a 'tune-up' of the current test framework rather than a
>      complete re-write. Which of the two are we discussing?
> 
>      Both...
> 
>      The requirements on the framework language are more relaxed, but for
>      ease of development and developer sanity, I assume that the
>      framework language should match the test language.  So I'm using the
>      test language as the requirements driver, and to gage community
>      preference for that test language.
> 
>      Before embarking on building yet another new and custom framework
>      for
>      Lustre we should evaluate some existing frameworks.  For example,
>      the
>      Autotest project was specifically designed to test the Linux kernel.
>      It's open source, looks active, is flexible, and there is detailed
>      documentation on how to write tests.  Plus it was designed
>      specifically
>      for testing the kernel so there are likely existing file system
>      tests.
>       [6]http://autotest.github.com/
>       "Autotest is a framework for fully automated testing. It is
>        designed primarily to test the Linux kernel, though it is useful
>        for many other functions such as qualifying new hardware. It's an
>        open-source project under the GPL and is used and developed by a
>        number of organizations, including Google, IBM, Red Hat, and many
>        others."
> 
>      Based on the responses so far, it seems that there is a fairly clear
>      preference for Python as a test language, and so I'll propose that
>      Python should be used shorter-term to start replacing
>      test-framework.
> 
>      If we decide the Autotest framework is a good fit then we'll want to
>      write the tests in python to be consistent with the framework
>      language.
>      However, for a first cut it looks like you could use the existing
>      bash
>      tests largely unmodified.
>      --
>      Thanks,
>      Brian
>      _______________________________________________
>      Lustre-devel mailing list
>      [7]Lustre-devel at lists.lustre.org
>      [8]http://lists.lustre.org/mailman/listinfo/lustre-devel
> 
>      On 10/25/2012 12:24 PM, Nathan Rutman wrote:
> 
>      On Oct 25, 2012, at 11:20 AM, "Gearing, Chris"
>      <[9]chris.gearing@intel.com
>      <[10]mailto:chris.gearing@intel.com>> wrote:
> 
>      Hi,
>      >Based on the responses so far, it seems that there is a fairly
>      clear
>      preference for Python as a test language, and so I'll propose that
>      Python should be used shorter-term to start replacing
>      test-framework.
>      I don?t think that is the case Nathan, because unless I am mistaken
>      Jacques-Charles, Kilian Cavalotti and if it is a binary choice today
>      my preference is Python (but please see below)
> 
>      Aurelien - python 2
> 
>      Bruce was neutral
>      Roman and Colin (after your email) expressed Perl as a choice.
>      So the preference would seem to be for Python, counting by people
>      and
>      organization although obviously it?s a statistically suspect poll.
> 
>      Indeed.  As I stated before, a poll is not the best way to choose
>      the
>      language; it's just one data point.  But as a) we must choose
>      something
>      to start making any headway toward improvements, and b) bash doesn't
>      meet the requirements, I'm trying to come up with some kind of
>      baseline.
> 
>      Chris
>      NOTES OF MY VOTE: Given the importance of the topic I believe a much
>      broader piece of analysis needs to be done to select the right
>      language,
> 
>      I'm just trying to keep things moving from theoretical optimum to
>      practical conclusion.  We feel a need today for much better Lustre
>      testing, and there's a huge pile of work waiting, and so I feel we
>      must
>      move forward.
> 
>      also if this is for an existing framework update the tests must all
>      be
>      in bash or all converted to the new language of choice.
> 
>      Short term the idea would be to translate one of the test suites,
>      for
>      example conf-sanity, to the new language/framework.  The two systems
>      (old and new) would live side-by-side for some period.  New tests
>      should
>      be implemented under the new system (e.g. sanity2).  Other old tests
>      could be converted piecemeal subsequently.
> 
>      --------------------------------------------------------------------
>      -
>      Intel Corporation (UK) Limited
>      Registered No. 1134945 (England)
>      Registered Office: Pipers Way, Swindon SN3 1RJ
>      VAT No: 860 2173 47
>      This e-mail and any attachments may contain confidential material
>      for
>      the sole use of the intended recipient(s). Any review or
>      distribution
>      by others is strictly prohibited. If you are not the intended
>      recipient, please contact the sender and delete all copies.
> 
>      _______________________________________________
>      Lustre-devel mailing list
>      [11]Lustre-devel at lists.lustre.org
>      http://lists.lustre.org/mailman/listinfo/lustre-devel
> 
> References
> 
>    1. mailto:morrone2 at llnl.gov
>    2. http://www.opensfs.org/foswiki/bin/view/Lustre/TestingImprovements
>    3. http://wiki.whamcloud.com/download/attachments/7634975/new+test+framework.pdf
>    4. mailto:behlendorf1 at llnl.gov
>    5. mailto:chris.gearing at intel.com
>    6. http://autotest.github.com/
>    7. mailto:Lustre-devel at lists.lustre.org
>    8. http://lists.lustre.org/mailman/listinfo/lustre-devel
>    9. mailto:chris.gearing at intel.com
>   10. mailto:chris.gearing at intel.com
>   11. mailto:Lustre-devel at lists.lustre.org

> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel

[Lustre-devel] Language choice for Lustre tests

[Prakash Surya]

On Thu, Oct 25, 2012 at 02:36:02PM -0700, Nathan Rutman wrote:
> 
> On Oct 25, 2012, at 2:17 PM, Prakash Surya <surya1@llnl.gov> wrote:
> 
> > On Thu, Oct 25, 2012 at 11:04:55AM -0700, Nathan Rutman wrote:
> >>   There are several levels of "test framework" that are involved with
> >>   automated testing of Lustre:
> >> 
> >>     * unit tests themselves, written in bash, e.g. sanity 1a
> > 
> > What do you mean by "unit tests" here? Because as I understand it,
> > Lustre has no unit testing at all. It is all functional testing,
> > which is related but completely different.
> Let's please not get into this.  I just meant the individual tests.

OK, I just wanted to make sure we're on the same page. I would *love*
to see unit testing incorporated into the test suite. I think that would
greatly benefit the stability of the project.

> 
> > 
> >>     * test-framework.sh & friends, which provides some
> >>       support/setup/communal library for unit tests
> >>     * automated test systems, which automatically execute the unit tests,
> >>       record and report the results.
> > 
> > I don't quite understand why there is so much discussion regarding a new
> > testing framework specifically designed for testing Lustre. Is there
> > really nothing already out there we can make use of?
> > 
> > Brian linked to the Autotest framework, does that not do what we want?
> 
> That covers the third meaning, but not the first two.   I'm only concerned about the
> first two at the moment. 

Well, I think it spills into the first and second as well.

1. Since Autotest is already implemented in python, it would make sense to use
   that as the language for the individual tests as well.

2. And it _may_ deal with set up of tests. But I haven't looked into
   Autotest enough to know its limitations. The shared library would, of
   course, be specific to Lustre.

> 
> > If it doesn't work, why not?
> It doesn't contain a Lustre-specific functional library, it doesn't safely limit the tests
> that may be run on a specific system, it doesn't handle interoperability testing, etc.

Right, the test library will be specific to Lustre.

Limiting the number of tests to run, which ones run, and interoperability
might be partially handled by whatever automated framework is used. Of
course it might not, as well.

> 
> > And how is our home brewed solution going
> > to differ? Are there any other alternatives out there? I don't see
> > benefit in spending development on a new framework if it's unnecessary.
> > 
> > There is a lot of discussion regarding a tool (the language) and little
> > involving the problem (lack of useful functional and unit tests to
> > properly stress the system). Pick something, move on, and start
> > improving the codes stability.
> That was what I was trying to do.

Great! :). I'm really looking forward to seeing a solid testing
infrastructure implemented.

> 
> > 
> > And if somebody *really* wants to build a new testing framework, have it
> > live out of tree. Create it as a standalone project which interfaces
> > with Lustre.
> As stated in my presentation, that is part of the plan.

Ah, I didn't catch that part. Good to know.

-- 
Cheers, Prakash

> 
> > 
> > -- 
> > Cheers, Prakash
> > 
> > 
> >> 
> >>   I'm only speaking about the language for the first of these items at
> >>   the moment.
> >>   For more context, including the motivation for changing things, see my
> >>   LAD'12 presentation (more narrowly focused) on the [1]OpenSFS Wiki.
> >>   On Oct 25, 2012, at 10:23 AM, Brian Behlendorf
> >>   <[2]behlendorf1@llnl.gov> wrote:
> >> 
> >>     On Wed, 2012-10-24 at 15:05 -0700, Nathan Rutman wrote:
> >> 
> >>     On Oct 24, 2012, at 1:02 PM, "Gearing, Chris"
> >>     <[3]chris.gearing@intel.com> wrote:
> >> 
> >>     Nathan,
> >>     I'm not 100% sure what you are proposing here, your LAD presentation
> >>     suggested a 'tune-up' of the current test framework rather than a
> >>     complete re-write. Which of the two are we discussing?
> >> 
> >>     Both...
> >> 
> >>     The requirements on the framework language are more relaxed, but for
> >>     ease of development and developer sanity, I assume that the
> >>     framework language should match the test language.  So I'm using the
> >>     test language as the requirements driver, and to gage community
> >>     preference for that test language.
> >> 
> >>     Before embarking on building yet another new and custom framework
> >>     for
> >>     Lustre we should evaluate some existing frameworks.  For example,
> >>     the
> >>     Autotest project was specifically designed to test the Linux kernel.
> >>     It's open source, looks active, is flexible, and there is detailed
> >>     documentation on how to write tests.  Plus it was designed
> >>     specifically
> >>     for testing the kernel so there are likely existing file system
> >>     tests.
> >>      [4]http://autotest.github.com/
> >>      "Autotest is a framework for fully automated testing. It is
> >>       designed primarily to test the Linux kernel, though it is useful
> >>       for many other functions such as qualifying new hardware. It's an
> >>       open-source project under the GPL and is used and developed by a
> >>       number of organizations, including Google, IBM, Red Hat, and many
> >>       others."
> >> 
> >>     Based on the responses so far, it seems that there is a fairly clear
> >>     preference for Python as a test language, and so I'll propose that
> >>     Python should be used shorter-term to start replacing
> >>     test-framework.
> >> 
> >>     If we decide the Autotest framework is a good fit then we'll want to
> >>     write the tests in python to be consistent with the framework
> >>     language.
> >>     However, for a first cut it looks like you could use the existing
> >>     bash
> >>     tests largely unmodified.
> >>     --
> >>     Thanks,
> >>     Brian
> >>     _______________________________________________
> >>     Lustre-devel mailing list
> >>     [5]Lustre-devel at lists.lustre.org
> >>     http://lists.lustre.org/mailman/listinfo/lustre-devel
> >> 
> >> References
> >> 
> >>   1. http://www.opensfs.org/foswiki/bin/view/Lustre/TestingImprovements
> >>   2. mailto:behlendorf1 at llnl.gov
> >>   3. mailto:chris.gearing at intel.com
> >>   4. http://autotest.github.com/
> >>   5. mailto:Lustre-devel at lists.lustre.org
> > 
> >> _______________________________________________
> >> Lustre-devel mailing list
> >> Lustre-devel at lists.lustre.org
> >> http://lists.lustre.org/mailman/listinfo/lustre-devel
> > 

[Lustre-devel] Language choice for Lustre tests

[Roman Grigoryev]

On 10/25/2012 09:23 PM, Brian Behlendorf wrote:
> On Wed, 2012-10-24 at 15:05 -0700, Nathan Rutman wrote:
>> On Oct 24, 2012, at 1:02 PM, "Gearing, Chris" <chris.gearing@intel.com> wrote:
>>
>>> Nathan,
>>>
>>> I'm not 100% sure what you are proposing here, your LAD presentation suggested a 'tune-up' of the current test framework rather than a complete re-write. Which of the two are we discussing?
>> Both...
>> The requirements on the framework language are more relaxed, but for ease of development and developer sanity, I assume that the framework language should match the test language.  So I'm using the test language as the requirements driver, and to gage community preference for that test language.
> Before embarking on building yet another new and custom framework for
> Lustre we should evaluate some existing frameworks.  For example, the
> Autotest project was specifically designed to test the Linux kernel.
> It's open source, looks active, is flexible, and there is detailed
> documentation on how to write tests.  Plus it was designed specifically
> for testing the kernel so there are likely existing file system tests.
>
>   http://autotest.github.com/
>
>   "Autotest is a framework for fully automated testing. It is
>    designed primarily to test the Linux kernel, though it is useful
>    for many other functions such as qualifying new hardware. It's an
>    open-source project under the GPL and is used and developed by a
>    number of organizations, including Google, IBM, Red Hat, and many
>    others."
Brian,
I agree that autotest is interesting tool and could be used for Lustre
tests,  but
it need pretty big improvement if we want to use it for Lustre.

I have made autotest evaluation for quick executing Lustre on 3 or 4
nodes and stopped
this activity after spending 3 days without  good result.
I tried  to find simple way to execute llmount.sh on mds, oss, clients
and run one sanity test.
I cannot found documentation and ask list (now documentation exists:
https://github.com/autotest/autotest/wiki/Synchronizationclientsinmultihoststest).

Sorry for long quote, it is from  Lucas Meneghel Rodrigues(RedHat) answer:
> So yes, autotest does have a mechanism to execute 'distributed' testing,
> as well as 'kvm autotest' does have tests that do use such a testing
> arrangement. In autotest, one of the mechanisms used to coordinate
> execution of tests among different machines is called barrier.
>
> A barrier is a class that blocks test execution until all 'members' have
> 'checked in' the barrier. So, consider this example from the client
> version of netperf:
>
>             if role == 'server':
>                 self.server_start(cpu_affinity)
>                 try:
>                     # Wait up to ten minutes for the client to reach this
>                     # point.
>                     self.job.barrier(server_tag, 'start_%d' % num_streams,
>                                      600).rendezvous(*all)
>                     # Wait up to test_time + 5 minutes for the test to
>                     # complete
>                     self.job.barrier(server_tag, 'stop_%d' % num_streams,
>                                      test_time+300).rendezvous(*all)
>                 finally:
>                     self.server_stop()
>
>             elif role == 'client':
>                 # Wait up to ten minutes for the server to start
>                 self.job.barrier(client_tag, 'start_%d' % num_streams,
>                                  600).rendezvous(*all)
>                 self.client(server_ip, test, test_time, num_streams,
>                             test_specific_args, cpu_affinity)
>                 # Wait up to 5 minutes for the server to also reach this 
> point
>                 self.job.barrier(client_tag, 'stop_%d' % num_streams,
>                                  300).rendezvous(*all)
>
> You can see above that the client only will start the client code if the
> server is active and did check in the barrier
As you could see, it is pretty limited functionality and current
test-framework.sh which just support
command series like below more answer on our needs.
-----------------------------------------------
do_facet $SINGLEMDS "$LCTL set_param mdd.${!var}.sync_permission=0"
do_facet $SINGLEMDS "$LCTL set_param mdt.${!var}.commit_on_sharing=0"
do_node $CLIENT1 mkdir -p -m 755 $MOUNT/$tdir
replay_barrier $SINGLEMDS
do_node $CLIENT2 chmod 777 $MOUNT2/$tdir
do_node $CLIENT1 openfile -f O_RDWR:O_CREAT $MOUNT/$tdir/$tfile
zconf_umount $CLIENT2 $MOUNT2
facet_failover $SINGLEMDS
-----------------------------------------------

There is also one more way to do it
(https://github.com/autotest/autotest/wiki/Autoserv) but in this case
this code
work out of test.

I don't like way of customizing test execution via control files. I
think it could be pretty unfriendly for Lustre developers.
Also there is not clean where and how implement permission control for
safe test execution on life clusters, random execution,
simple plug own code before/after tests steps.

It should be possible to implement this functionality (as usually in
software development) but could be not so simple.
>> Based on the responses so far, it seems that there is a fairly clear preference for Python as a test language, and so I'll propose that Python should be used shorter-term to start replacing test-framework.
> If we decide the Autotest framework is a good fit then we'll want to
> write the tests in python to be consistent with the framework language.
> However, for a first cut it looks like you could use the existing bash
> tests largely unmodified.
I think, it is to pretty complex schema  for developer to use autotest
for executing
bash lustre tests considering Lustre configuration translation to
control files.
Looks like benefits from this schema could get only people who
automate testing(and who often already could have own tools).

Thanks,
    Roman

[Lustre-devel] Language choice for Lustre tests

[Kilian Cavalotti]

Hi Prakash,

> Why do you think this? Bash is extremely modular using the "source"
> command.

> Not portable? Really? Bash is available on just about every Linux
> distribution that comes to mind. Definitely every one Lustre cares
> about.

> That is not Bash's fault. Bad code can be written in any language. It's
> up to the programmer to write "coherent" test scripts.
>
> How would switching languages improve on either of these two fronts?

I couldn't agree more. I'm still not sure why Bash has to be gotten
rid of, and what the improvement would be rewriting a whole new
testing framework from scratch in an other language, when a working
one currently exist. It can certainly be improved, but I'm pretty sure
Bash has been chosen for a reason at some point, and I'm not convinced
the reasons that lead to this choice are not valid anymore.

Cheers,
-- 
Kilian

[Lustre-devel] Language choice for Lustre tests

[Roman Grigoryev]

Hi Kilian,
On 10/26/2012 10:24 AM, Kilian Cavalotti wrote:
> 
> I couldn't agree more. I'm still not sure why Bash has to be gotten
> rid of, and what the improvement would be rewriting a whole new
> testing framework from scratch in an other language, when a working
> one currently exist. It can certainly be improved, but I'm pretty sure
> Bash has been chosen for a reason at some point, and I'm not convinced
> the reasons that lead to this choice are not valid anymore.

Few samples, which comparably hard to implement on bash than on other
languages from my point of view (especially via evolving tes-framework.sh):

 - how could be implemented for current framework random and repeatable
test execution?

 - how could be implemented observation of remote node status while
command executed on it and killed it by timeout?

 - how simple add machine-readable meta information for test, e.f. tags
and timeouts?


Thanks,
	Roman

[Lustre-devel] Language choice for Lustre tests

[Kilian Cavalotti]

Hi Roman,

> Few samples, which comparably hard to implement on bash than on other
> languages from my point of view (especially via evolving tes-framework.sh):

I don't really want to engage in a code battle I would definitely
loose, and I'm also not very familiar with the current test framework
inner workings, but here are a few points.

> - how could be implemented for current framework random and repeatable
> test execution?

Not sure what's required here, but how about storing the list of tests
in a Bash array, generating a list of random indexes, and using it to
loop over the test array?

> - how could be implemented observation of remote node status while
> command executed on it and killed it by timeout?

One can read a SSH output stream with the appropriate redirections and
act upon results, stdout or stderr contents, and/or return codes. The
read Bash builtin takes a timeout parameter, and "timeout" is also an
often overlooked coreutils command. :)

> - how simple add machine-readable meta information for test, e.f. tags
> and timeouts?

With an associative array, for instance:
$ declare -A TESTS
$ TESTS=( [test1]="tag1:timeout1"
        [test2]="tag2:timeout2"
        [test3]="tag3:timeout3" )
$ for test in ${!TESTS[@]}; do
        value=${TESTS[$test]}
        tag=${value%%:*}
        timeout=${value##*:}
        printf "%s is tagged with %s and has a timeout of %s.\n" $test
$tag $timeout
done
test1 is tagged with tag1 and has a timeout of timeout1.
test2 is tagged with tag2 and has a timeout of timeout2.
test3 is tagged with tag3 and has a timeout of timeout3.

Cheers,
--
Kilian