Re: [ANNOUNCE] Open POSIX Test Suite

Re: [ANNOUNCE] Open POSIX Test Suite

[Stephanie Glass]

Rusty,
We will take them.  We may set them up not to run with our runall portion
but as a separate run.  This is how we do several areas, such as
networking.   Just let us know when you are ready to start contributing.
It doesn't have to be all at once, we will take in stages also.  We put out
a new version almost every month so we can get them out there quickly.

Don't most of these test cases deal with things like POSIX timers,
semaphores, threads, etc.?  Or are these other types of tests?

Thanks

Stephanie

Linux Technology Center
 IBM, 11400 Burnet Road, Austin, TX  78758
 Phone: (512) 838-9284   T/L: 678-9284  Fax: (512) 838-3882
 E-Mail: sglass@us.ibm.com


                                                                                                                             
                      "Rusty Lynch"                                                                                          
                      <rusty@linux.co.i        To:       "Geoff Gustafson" <geoff@linux.co.intel.com>, Stephanie             
                      ntel.com>                 Glass/Austin/IBM@IBMUS                                                       
                                               cc:       "Dan Kegel" <dkegel@ixiacom.com>, "Linux Kernel Mailing List"       
                      11/05/2002 10:43          <linux-kernel@vger.kernel.org>                                               
                      AM                       Subject:  Re: [ANNOUNCE] Open POSIX Test Suite                                
                                                                                                                             
                                                                                                                             
                                                                                                                             



Stephanie,

All test are GPL, so anyone can do anything they want with them.  We would
be happy to donate test to any project.

The truth is that we modeled test cases after LTP, meaning that a test case
is
a simple executable that returns 0 for success and anything else to
indicate
failure, so copying a test from posixtest to LTP should be very easy.

I was under the impression that LTP did not want to accept a bunch of test
cases that did not currently have an associated implementation in Linux.
It sounds like this is not exactly correct.  How about test cases that will
probably
always be implemented in user space?  Isn't LTP specific to kernel testing?

    -rusty

----- Original Message -----
From: "Stephanie Glass" <sglass@us.ibm.com>
To: "Geoff Gustafson" <geoff@linux.co.intel.com>
Cc: "Dan Kegel" <dkegel@ixiacom.com>; "Linux Kernel Mailing List"
<linux-kernel@vger.kernel.org>
Sent: Tuesday, November 05, 2002 7:49 AM
Subject: Re: [ANNOUNCE] Open POSIX Test Suite


>
> Geoff,
> The LTP would be happy to have anyone in the Linux community donate test
> cases.  This includes any POSIX tests.
> The LTP would not be advertised as a POSIX compliance test, that would be
> up to LSB to handle.  These tests
> would only increase the overall LTP api coverages.
>
> Does your group own these tests?  Do you want to donate them to the LTP?
>
> Stephanie
>
> Linux Technology Center
>  IBM, 11400 Burnet Road, Austin, TX  78758
>  Phone: (512) 838-9284   T/L: 678-9284  Fax: (512) 838-3882
>  E-Mail: sglass@us.ibm.com
>
>
>
>                       "Geoff Gustafson"
>                       <geoff@linux.co.i        To:       "Dan Kegel"
<dkegel@ixiacom.com>, "Linux Kernel Mailing List"
>                       ntel.com>
<linux-kernel@vger.kernel.
org>
>                                                cc:       Stephanie
Glass/Austin/IBM@IBMUS
>                       11/04/2002 06:04         Subject:  Re: [ANNOUNCE]
Open POSIX Test Suite
>                       PM
>
>
>
>
>
> > You are about to duplicate http://ltp.sf.net
>
> My understanding is that LTP is focused on current mainline kernel
testing,
> while this project's initial concern is areas that are not currently in
> Linux
> like POSIX message queues, semaphores, and full support for POSIX
threads.
> I see
> this as being used to evaluate different implementations that are being
> considered for inclusion in the kernel, glibc, etc.
>
> This project is concerned with the POSIX APIs regardless of where they
are
> implemented (kernel, glibc, etc.). Thus it can focus on POSIX,
independent
> of
> implementation. This project will be more concerned with traceability
back
> to
> the POSIX specification, and completeness of coverage, than I would
expect
> from
> LTP.
>
> That said, there is some overlap, and an exchange of test cases between
the
> projects may be very useful.
>
> I've copied Stephanie from LTP to get her reaction.
>
> -- Geoff Gustafson
>
> These are my views and not necessarily those of my employer.
>
>
>
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel"
in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [ANNOUNCE] Open POSIX Test Suite

[Rusty Lynch]

We started with a few areas like message queues and signals, but it is just
a starting point.

    -rusty (hmm... guess rusty is already taken in this domain.  maybe I
should sign rustyl :->)

----- Original Message -----
From: "Stephanie Glass" <sglass@us.ibm.com>
To: "Rusty Lynch" <rusty@linux.co.intel.com>
Cc: "Dan Kegel" <dkegel@ixiacom.com>; "Geoff Gustafson"
<geoff@linux.co.intel.com>; "Linux Kernel Mailing List"
<linux-kernel@vger.kernel.org>
Sent: Tuesday, November 05, 2002 10:24 AM
Subject: Re: [ANNOUNCE] Open POSIX Test Suite


>
> Rusty,
> We will take them.  We may set them up not to run with our runall portion
> but as a separate run.  This is how we do several areas, such as
> networking.   Just let us know when you are ready to start contributing.
> It doesn't have to be all at once, we will take in stages also.  We put
out
> a new version almost every month so we can get them out there quickly.
>
> Don't most of these test cases deal with things like POSIX timers,
> semaphores, threads, etc.?  Or are these other types of tests?
>
> Thanks
>
> Stephanie
>
> Linux Technology Center
>  IBM, 11400 Burnet Road, Austin, TX  78758
>  Phone: (512) 838-9284   T/L: 678-9284  Fax: (512) 838-3882
>  E-Mail: sglass@us.ibm.com
>
>
>
>                       "Rusty Lynch"
>                       <rusty@linux.co.i        To:       "Geoff Gustafson"
<geoff@linux.co.intel.com>, Stephanie
>                       ntel.com>                 Glass/Austin/IBM@IBMUS
>                                                cc:       "Dan Kegel"
<dkegel@ixiacom.com>, "Linux Kernel Mailing List"
>                       11/05/2002 10:43
<linux-kernel@vger.kernel.org>
>                       AM                       Subject:  Re: [ANNOUNCE]
Open POSIX Test Suite
>
>
>
>
>
>
> Stephanie,
>
> All test are GPL, so anyone can do anything they want with them.  We would
> be happy to donate test to any project.
>
> The truth is that we modeled test cases after LTP, meaning that a test
case
> is
> a simple executable that returns 0 for success and anything else to
> indicate
> failure, so copying a test from posixtest to LTP should be very easy.
>
> I was under the impression that LTP did not want to accept a bunch of test
> cases that did not currently have an associated implementation in Linux.
> It sounds like this is not exactly correct.  How about test cases that
will
> probably
> always be implemented in user space?  Isn't LTP specific to kernel
testing?
>
>     -rusty
>
> ----- Original Message -----
> From: "Stephanie Glass" <sglass@us.ibm.com>
> To: "Geoff Gustafson" <geoff@linux.co.intel.com>
> Cc: "Dan Kegel" <dkegel@ixiacom.com>; "Linux Kernel Mailing List"
> <linux-kernel@vger.kernel.org>
> Sent: Tuesday, November 05, 2002 7:49 AM
> Subject: Re: [ANNOUNCE] Open POSIX Test Suite
>
>
> >
> > Geoff,
> > The LTP would be happy to have anyone in the Linux community donate test
> > cases.  This includes any POSIX tests.
> > The LTP would not be advertised as a POSIX compliance test, that would
be
> > up to LSB to handle.  These tests
> > would only increase the overall LTP api coverages.
> >
> > Does your group own these tests?  Do you want to donate them to the LTP?
> >
> > Stephanie
> >
> > Linux Technology Center
> >  IBM, 11400 Burnet Road, Austin, TX  78758
> >  Phone: (512) 838-9284   T/L: 678-9284  Fax: (512) 838-3882
> >  E-Mail: sglass@us.ibm.com
> >
> >
> >
> >                       "Geoff Gustafson"
> >                       <geoff@linux.co.i        To:       "Dan Kegel"
> <dkegel@ixiacom.com>, "Linux Kernel Mailing List"
> >                       ntel.com>
> <linux-kernel@vger.kernel.
> org>
> >                                                cc:       Stephanie
> Glass/Austin/IBM@IBMUS
> >                       11/04/2002 06:04         Subject:  Re: [ANNOUNCE]
> Open POSIX Test Suite
> >                       PM
> >
> >
> >
> >
> >
> > > You are about to duplicate http://ltp.sf.net
> >
> > My understanding is that LTP is focused on current mainline kernel
> testing,
> > while this project's initial concern is areas that are not currently in
> > Linux
> > like POSIX message queues, semaphores, and full support for POSIX
> threads.
> > I see
> > this as being used to evaluate different implementations that are being
> > considered for inclusion in the kernel, glibc, etc.
> >
> > This project is concerned with the POSIX APIs regardless of where they
> are
> > implemented (kernel, glibc, etc.). Thus it can focus on POSIX,
> independent
> > of
> > implementation. This project will be more concerned with traceability
> back
> > to
> > the POSIX specification, and completeness of coverage, than I would
> expect
> > from
> > LTP.
> >
> > That said, there is some overlap, and an exchange of test cases between
> the
> > projects may be very useful.
> >
> > I've copied Stephanie from LTP to get her reaction.
> >
> > -- Geoff Gustafson
> >
> > These are my views and not necessarily those of my employer.
> >
> >
> >
> >
> >
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel"
> in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
>
>
>
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [ANNOUNCE] Open POSIX Test Suite

[Stephanie Glass]

Geoff,
The LTP would be happy to have anyone in the Linux community donate test
cases.  This includes any POSIX tests.
The LTP would not be advertised as a POSIX compliance test, that would be
up to LSB to handle.  These tests
would only increase the overall LTP api coverages.

Does your group own these tests?  Do you want to donate them to the LTP?

Stephanie

Linux Technology Center
 IBM, 11400 Burnet Road, Austin, TX  78758
 Phone: (512) 838-9284   T/L: 678-9284  Fax: (512) 838-3882
 E-Mail: sglass@us.ibm.com


                                                                                                                             
                      "Geoff Gustafson"                                                                                      
                      <geoff@linux.co.i        To:       "Dan Kegel" <dkegel@ixiacom.com>, "Linux Kernel Mailing List"       
                      ntel.com>                 <linux-kernel@vger.kernel.org>                                               
                                               cc:       Stephanie Glass/Austin/IBM@IBMUS                                    
                      11/04/2002 06:04         Subject:  Re: [ANNOUNCE] Open POSIX Test Suite                                
                      PM                                                                                                     
                                                                                                                             
                                                                                                                             



> You are about to duplicate http://ltp.sf.net

My understanding is that LTP is focused on current mainline kernel testing,
while this project's initial concern is areas that are not currently in
Linux
like POSIX message queues, semaphores, and full support for POSIX threads.
I see
this as being used to evaluate different implementations that are being
considered for inclusion in the kernel, glibc, etc.

This project is concerned with the POSIX APIs regardless of where they are
implemented (kernel, glibc, etc.). Thus it can focus on POSIX, independent
of
implementation. This project will be more concerned with traceability back
to
the POSIX specification, and completeness of coverage, than I would expect
from
LTP.

That said, there is some overlap, and an exchange of test cases between the
projects may be very useful.

I've copied Stephanie from LTP to get her reaction.

-- Geoff Gustafson

These are my views and not necessarily those of my employer.

Re: [ANNOUNCE] Open POSIX Test Suite

[Rusty Lynch]

Stephanie,

All test are GPL, so anyone can do anything they want with them.  We would
be happy to donate test to any project.

The truth is that we modeled test cases after LTP, meaning that a test case
is
a simple executable that returns 0 for success and anything else to indicate
failure, so copying a test from posixtest to LTP should be very easy.

I was under the impression that LTP did not want to accept a bunch of test
cases that did not currently have an associated implementation in Linux.
It sounds like this is not exactly correct.  How about test cases that will
probably
always be implemented in user space?  Isn't LTP specific to kernel testing?

    -rusty

----- Original Message -----
From: "Stephanie Glass" <sglass@us.ibm.com>
To: "Geoff Gustafson" <geoff@linux.co.intel.com>
Cc: "Dan Kegel" <dkegel@ixiacom.com>; "Linux Kernel Mailing List"
<linux-kernel@vger.kernel.org>
Sent: Tuesday, November 05, 2002 7:49 AM
Subject: Re: [ANNOUNCE] Open POSIX Test Suite


>
> Geoff,
> The LTP would be happy to have anyone in the Linux community donate test
> cases.  This includes any POSIX tests.
> The LTP would not be advertised as a POSIX compliance test, that would be
> up to LSB to handle.  These tests
> would only increase the overall LTP api coverages.
>
> Does your group own these tests?  Do you want to donate them to the LTP?
>
> Stephanie
>
> Linux Technology Center
>  IBM, 11400 Burnet Road, Austin, TX  78758
>  Phone: (512) 838-9284   T/L: 678-9284  Fax: (512) 838-3882
>  E-Mail: sglass@us.ibm.com
>
>
>
>                       "Geoff Gustafson"
>                       <geoff@linux.co.i        To:       "Dan Kegel"
<dkegel@ixiacom.com>, "Linux Kernel Mailing List"
>                       ntel.com>                 <linux-kernel@vger.kernel.
org>
>                                                cc:       Stephanie
Glass/Austin/IBM@IBMUS
>                       11/04/2002 06:04         Subject:  Re: [ANNOUNCE]
Open POSIX Test Suite
>                       PM
>
>
>
>
>
> > You are about to duplicate http://ltp.sf.net
>
> My understanding is that LTP is focused on current mainline kernel
testing,
> while this project's initial concern is areas that are not currently in
> Linux
> like POSIX message queues, semaphores, and full support for POSIX threads.
> I see
> this as being used to evaluate different implementations that are being
> considered for inclusion in the kernel, glibc, etc.
>
> This project is concerned with the POSIX APIs regardless of where they are
> implemented (kernel, glibc, etc.). Thus it can focus on POSIX, independent
> of
> implementation. This project will be more concerned with traceability back
> to
> the POSIX specification, and completeness of coverage, than I would expect
> from
> LTP.
>
> That said, there is some overlap, and an exchange of test cases between
the
> projects may be very useful.
>
> I've copied Stephanie from LTP to get her reaction.
>
> -- Geoff Gustafson
>
> These are my views and not necessarily those of my employer.
>
>
>
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

re: [ANNOUNCE] Open POSIX Test Suite

[Dan Kegel]

"Geoff Gustafson" <geoff@linux.co.intel.com> wrote:
 > I would like to announce a new project to develop and/or
 > assemble a GPL test suite for POSIX APIs.

You are about to duplicate http://ltp.sf.net

I imagine the main difference is you're targeting Posix
compliance rather than LSB compliance.  That seems
like a fairly minor difference that could be
accommodated within the framework of the LTP.

Or do you feel the existing test framework
and tests assembled in LTP are so inadequate that you
need to start an entirely new project doing
essentially the same thing?

(Apologies if you've already answered this question.  I did
check your web page, but didn't see an answer.)
- Dan

Re: [ANNOUNCE] Open POSIX Test Suite

[Geoff Gustafson]

> You are about to duplicate http://ltp.sf.net

My understanding is that LTP is focused on current mainline kernel testing,
while this project's initial concern is areas that are not currently in Linux
like POSIX message queues, semaphores, and full support for POSIX threads. I see
this as being used to evaluate different implementations that are being
considered for inclusion in the kernel, glibc, etc.

This project is concerned with the POSIX APIs regardless of where they are
implemented (kernel, glibc, etc.). Thus it can focus on POSIX, independent of
implementation. This project will be more concerned with traceability back to
the POSIX specification, and completeness of coverage, than I would expect from
LTP.

That said, there is some overlap, and an exchange of test cases between the
projects may be very useful.

I've copied Stephanie from LTP to get her reaction.

-- Geoff Gustafson

These are my views and not necessarily those of my employer.

Re: [ANNOUNCE] Open POSIX Test Suite

[Dan Kegel]

Geoff Gustafson wrote:
>>You are about to duplicate http://ltp.sf.net
> 
> 
> My understanding is that LTP is focused on current mainline kernel testing,
> while this project's initial concern is areas that are not currently in Linux
> like POSIX message queues, semaphores, and full support for POSIX threads. I see
> this as being used to evaluate different implementations that are being
> considered for inclusion in the kernel, glibc, etc.
> 
> This project is concerned with the POSIX APIs regardless of where they are
> implemented (kernel, glibc, etc.). Thus it can focus on POSIX, independent of
> implementation. This project will be more concerned with traceability back to
> the POSIX specification, and completeness of coverage, than I would expect from
> LTP.
> 
> That said, there is some overlap, and an exchange of test cases between the
> projects may be very useful.
> 
> I've copied Stephanie from LTP to get her reaction.

Geoff,
thanks for the reply.  I have a feeling that LTP would be overjoyed
to have you contribute to the LTP and make it a more accurate Posix
compliance test.  The areas that Linux does not currently cover --
message queues, semaphores, etc -- *should* be in the LTP, regardless
of whether Linux currently implements them.  Linux's 'expected results'
on these tests would be 'fail' at the moment.  That's not a problem.

The LTP would also greatly benefit from better tracability and coverage.

I urge you to consider ways in which you could work within the
framework of the LTP to meet both your goals and the LTP's goals.
They may be more in synch than you originally thought!
- Dan

Re: [ANNOUNCE] Open POSIX Test Suite

[Dan Kegel]

Geoff Gustafson wrote:
 >Dan wrote:
>>You are about to duplicate http://ltp.sf.net
 >Chris wrote:
 >[You are about to duplicate http://www.linuxbase.org/test ]

Anyone know what the relationship between the LTP and LSB Posix compliance tests are?
Were they both originally derived from the OpenGroup Posix compliance tests?
- Dan

Re: [ANNOUNCE] Open POSIX Test Suite

[Christopher Yeoh]

At 2002/11/4 16:24-0800  Dan Kegel writes:
> 
> Anyone know what the relationship between the LTP and LSB Posix
> compliance tests are?

The emphasis on what the test suites are trying to achieve does differ
- eg the LSB doesn't have endurance type tests and concentrates more
on writing tests which can be used for certification purposes. And of
course the LSB test suites only cover the areas that the LSB spec
requires.

> Were they both originally derived from the OpenGroup Posix compliance tests?

I don't think the LTP were tests derived from the OpenGroup (though
I'm not that familiar with them).

Chris
-- 
cyeoh@au.ibm.com
IBM OzLabs Linux Development Group
Canberra, Australia

Re: [ANNOUNCE] Open POSIX Test Suite

[Nathan Straz]

On Mon, Nov 04, 2002 at 04:04:32PM -0800, Geoff Gustafson wrote:
> > You are about to duplicate http://ltp.sf.net
> 
> My understanding is that LTP is focused on current mainline kernel testing,
[ .. snip .. ]
> This project is concerned with the POSIX APIs regardless of where they are
> implemented (kernel, glibc, etc.). 

You are correct.  LTP isn't trying to be any kind of "compliance" test.
We're just trying to exercise and stress the Linux kernel to find bugs
and guard against regressions.

On Mon, Nov 04, 2002 at 04:08:44PM -0800, Dan Kegel wrote:
> I urge you to consider ways in which you could work within the
> framework of the LTP to meet both your goals and the LTP's goals.
> They may be more in synch than you originally thought!

I don't think LTP's "framework[1]" is going to help Geoff complete his
project.  Geoff already created a SourceForge site.  As part of the
LTP, I probably would have given him a separate CVS module and a mailing
list.  I think he has everything he needs to get started already.  

That being said, if Geoff does want his Open POSIX test suite to be part
of the LTP, we'll gladly accomodate him.  



[1] By framework I think you mean logistical, not libraries.  I think
our test libraries have some things he might like.  I just recommend
cleaning them up before using them. 
-- 
Nate Straz                                              nstraz@sgi.com
sgi, inc                                           http://www.sgi.com/
Linux Test Project                                  http://ltp.sf.net/

[ANNOUNCE] Open POSIX Test Suite

[Geoff Gustafson]

I would like to announce a new project to develop and/or assemble a GPL test
suite for POSIX APIs. The tests will focus on conformance to the IEEE Std
1003.1-2001, but will also include separate functional and stress tests.

The project's current approach to conformance testing is to record
assertions
from a close reading of the POSIX specifications, and write minimal test
cases
that prove or disprove these assertions. The test suite will be independent
of
specific API implementations, and will eventually be easily configurable to
work with different implementations. The project aims for OS independence,
using only POSIX APIs, the autoconf suite, and simple shell support.
However,
it is currently only being tested on Linux.

Ultimately, the plan is to use the test suite to evaluate current support in
Linux, as well as new implementations being considered in the open source
community, and then contribute patches or at least bug reports (with a
minimal
test case) to the appropriate places, like LKML.

Contributions of any test cases, review of the work, discussion of the
approach, etc. are very welcome. Join the development mailing list,
posixtest-discuss. The initial focus is on Signals, Message Queues, Threads,
Semaphores, and Clocks & Timers, based on current interests and resources.
You can help in these areas, or start work on another area of the spec.
There
will need to be some uniformity across the suite, but many details have yet
to
be worked out, so your involvement in those decisions help a lot.

For more information, see the project website at
http://posixtest.sourceforge.net

Thanks,

-- Geoff Gustafson

These are my views and not necessarily those of my employer.

Re: [ANNOUNCE] Open POSIX Test Suite

[Larry McVoy]

On Mon, Nov 04, 2002 at 02:48:47PM -0800, Geoff Gustafson wrote:
> I would like to announce a new project to develop and/or assemble a GPL test
> suite for POSIX APIs. 

Great idea.  We can help in the following way: BitKeeper has an extremely
simple test harness used for regressions.  It's well thought out in that
it is trivial to write simple tests and run them in isolation or to 
run the whole suite.  If you want the harness, we'll give it to you 
under whatever license you want, I assume GPL, but we don't care.

You can see what the tests look like in BK, if you have it installed, we
ship all the tests, they are in `bk bin`/t

A simple test might be

	#!/bin/sh

	# test that touch creates a file
	touch foo
	test -f foo || {
		echo failed to create foo
		exit 1
	}

The harness takes care of putting you in a clean isolated environment.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: [ANNOUNCE] Open POSIX Test Suite

[Geoff Gustafson]

Larry McVoy wrote:
> Great idea.  We can help in the following way: BitKeeper has an extremely
> simple test harness used for regressions.  It's well thought out in that
> it is trivial to write simple tests and run them in isolation or to 
> run the whole suite.  If you want the harness, we'll give it to you 
> under whatever license you want, I assume GPL, but we don't care.

Yeah, that would be great. GPL would be correct.

> You can see what the tests look like in BK, if you have it installed, we
> ship all the tests, they are in `bk bin`/t

I haven't used BK yet, but I'll go try to do that.

Thanks,

-- Geoff Gustafson

These are my views and not necessarily those of my employer.

Re: [ANNOUNCE] Open POSIX Test Suite

[Christopher Yeoh]

Hi Geoff,

At 2002/11/4 14:48-0800  Geoff Gustafson writes:
> 
> Contributions of any test cases, review of the work, discussion of the
> approach, etc. are very welcome. Join the development mailing list,
> posixtest-discuss. The initial focus is on Signals, Message Queues, Threads,
> Semaphores, and Clocks & Timers, based on current interests and resources.
> You can help in these areas, or start work on another area of the spec.
> There
> will need to be some uniformity across the suite, but many details have yet
> to
> be worked out, so your involvement in those decisions help a lot.

Have you looked at the LSB test suites yet? They already cover much of
what you plan on writing tests for, though we would welcome any
volunteers who would like to increase the coverage.  Most of the tests
suites are released under the Artistic License, with quite a bit of
the code donated by the Open Group (originally from the Unix
certification tests).

You can find more information about them here:

http://www.linuxbase.org/test

The CVS repository is on SourceForge in the LSB project.

Chris
-- 
cyeoh@au.ibm.com
IBM OzLabs Linux Development Group
Canberra, Australia

Re: [ANNOUNCE] Open POSIX Test Suite

[Geoff Gustafson]

Christopher Yeoh writes:
> Have you looked at the LSB test suites yet? They already cover much of
> what you plan on writing tests for, though we would welcome any
> volunteers who would like to increase the coverage.  Most of the tests
> suites are released under the Artistic License, with quite a bit of
> the code donated by the Open Group (originally from the Unix
> certification tests).

One issue is that this new project is primarily concerned with testing parts of
the spec that have not been fully supported in Linux so far. These are the kind
of things that are not included in LSB yet, so they wouldn't be appropriate in
LSB's test suite.

Another problem is the overhead of the TET framework. One of the goals of this
the new test suite is to have test cases which are utterly minimal. So far, each
test case has its own main() function and a bare minimum of surrounding code.
The idea is that when a bug is found, this one .c file can be sent to the
appropriate developer, and without any learning curve, they have the ability to
find their bug. I don't think LKML wants to see TET code posted here. :)

-- Geoff Gustafson

These are my views and not necessarily those of my employer.

Re: [ANNOUNCE] Open POSIX Test Suite

[Andreas Dilger]

On Nov 04, 2002  16:44 -0800, Geoff Gustafson wrote:
> Another problem is the overhead of the TET framework. One of the goals of
> this the new test suite is to have test cases which are utterly minimal.
> So far, each test case has its own main() function and a bare minimum of
> surrounding code. The idea is that when a bug is found, this one .c file
> can be sent to the appropriate developer, and without any learning curve,
> they have the ability to find their bug. I don't think LKML wants to see
> TET code posted here. :)

Having suffered through using the TET framework for the Open Group POSIX
test suite, I would agree that using TET sucks.  The code is so convoluted
as to be useless, and it is nearly impossible to see from the output what
it is actually doing.

I agree that having a simple C or shell or perl script which is the entire
test, and the rest of the framework is external to it is very desirable.

Cheers, Andreas
--
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [ANNOUNCE] Open POSIX Test Suite

[Christopher Yeoh]

At 2002/11/4 16:44-0800  Geoff Gustafson writes:
> One issue is that this new project is primarily concerned with
> testing parts of the spec that have not been fully supported in
> Linux so far. These are the kind of things that are not included in
> LSB yet, so they wouldn't be appropriate in LSB's test suite.

Actually we do include some areas that aren't yet in the LSB spec or
fully supported by Linux (eg aio) in the LSB test suites we release -
they just aren't run by default - but it is easy to enable them.

> utterly minimal. So far, each test case has its own main() function
> and a bare minimum of surrounding code.  The idea is that when a bug
> is found, this one .c file can be sent to the appropriate developer,
> and without any learning curve, they have the ability to find their
> bug. I don't think LKML wants to see TET code posted here. :)

Yes, I agree TET does have a significant learning curve, and I do end
up writing small test programs that don't include the TET stuff before
sending off bug reports.

I have however seen some advantages - It is nice when you get a test
failure the report tells you exactly which part of the specification
you're violating. Once you do understand the TET/vsxgen library calls
testcases look much simpler - and if you're aiming for complete
functionality coverage including all the tricky corner cases for
various interfaces which can require quite a bit of setup code to get
into the right situation I think you'll end up having to write helper
libraries anyway.

Chris
-- 
cyeoh@au.ibm.com
IBM OzLabs Linux Development Group
Canberra, Australia