[ANNOUNCE] Open POSIX Test Suite

[ANNOUNCE] Open POSIX Test Suite

[Geoff Gustafson]

I would like to announce a new project to develop and/or assemble a GPL test
suite for POSIX APIs. The tests will focus on conformance to the IEEE Std
1003.1-2001, but will also include separate functional and stress tests.

The project's current approach to conformance testing is to record
assertions
from a close reading of the POSIX specifications, and write minimal test
cases
that prove or disprove these assertions. The test suite will be independent
of
specific API implementations, and will eventually be easily configurable to
work with different implementations. The project aims for OS independence,
using only POSIX APIs, the autoconf suite, and simple shell support.
However,
it is currently only being tested on Linux.

Ultimately, the plan is to use the test suite to evaluate current support in
Linux, as well as new implementations being considered in the open source
community, and then contribute patches or at least bug reports (with a
minimal
test case) to the appropriate places, like LKML.

Contributions of any test cases, review of the work, discussion of the
approach, etc. are very welcome. Join the development mailing list,
posixtest-discuss. The initial focus is on Signals, Message Queues, Threads,
Semaphores, and Clocks & Timers, based on current interests and resources.
You can help in these areas, or start work on another area of the spec.
There
will need to be some uniformity across the suite, but many details have yet
to
be worked out, so your involvement in those decisions help a lot.

For more information, see the project website at
http://posixtest.sourceforge.net

Thanks,

-- Geoff Gustafson

These are my views and not necessarily those of my employer.

Re: [ANNOUNCE] Open POSIX Test Suite

[Larry McVoy]

On Mon, Nov 04, 2002 at 02:48:47PM -0800, Geoff Gustafson wrote:
> I would like to announce a new project to develop and/or assemble a GPL test
> suite for POSIX APIs. 

Great idea.  We can help in the following way: BitKeeper has an extremely
simple test harness used for regressions.  It's well thought out in that
it is trivial to write simple tests and run them in isolation or to 
run the whole suite.  If you want the harness, we'll give it to you 
under whatever license you want, I assume GPL, but we don't care.

You can see what the tests look like in BK, if you have it installed, we
ship all the tests, they are in `bk bin`/t

A simple test might be

	#!/bin/sh

	# test that touch creates a file
	touch foo
	test -f foo || {
		echo failed to create foo
		exit 1
	}

The harness takes care of putting you in a clean isolated environment.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

RFC: A POSIX Linux project?

[Jeff Garzik]

I wonder if any vendors, or independent groups, would be interested in 
maintaining a POSIX compliancy patchkit for the Linux kernel?

IMO such a "POSIX Linux" project would be useful for several reasons. 
 Overall, I think there is pressure from several directions to get all 
sorts of POSIX APIs into the kernel.  On occasion, kernel hackers are 
confronted with a situation where complete POSIX compliancy may mean a 
compromise in some area, be it performance, security, API issues, code 
cleanliness issues, etc.  Or simply that the POSIX-related code just 
isn't ready to be merged into the mainline kernel yet.

The vendors also benefit by this, because the barrier to entry in 
POSIX-related cases would be lowered, which would in turn satisfy the 
demands of customers.  Which would in turn give the mainline kernel all 
the software engineering benefits that come from a more reasoned and 
gradual review and merge of new features.

Does something like this already exist?  This would need to be an open, 
vendor-neutral project...

    Jeff

Re: [ANNOUNCE] Open POSIX Test Suite

[Geoff Gustafson]

Larry McVoy wrote:
> Great idea.  We can help in the following way: BitKeeper has an extremely
> simple test harness used for regressions.  It's well thought out in that
> it is trivial to write simple tests and run them in isolation or to 
> run the whole suite.  If you want the harness, we'll give it to you 
> under whatever license you want, I assume GPL, but we don't care.

Yeah, that would be great. GPL would be correct.

> You can see what the tests look like in BK, if you have it installed, we
> ship all the tests, they are in `bk bin`/t

I haven't used BK yet, but I'll go try to do that.

Thanks,

-- Geoff Gustafson

These are my views and not necessarily those of my employer.

Re: RFC: A POSIX Linux project?

[Andreas Dilger]

On Nov 04, 2002  18:14 -0500, Jeff Garzik wrote:
> I wonder if any vendors, or independent groups, would be interested in 
> maintaining a POSIX compliancy patchkit for the Linux kernel?
> 
> IMO such a "POSIX Linux" project would be useful for several reasons. 
>  Overall, I think there is pressure from several directions to get all 
> sorts of POSIX APIs into the kernel.  On occasion, kernel hackers are 
> confronted with a situation where complete POSIX compliancy may mean a 
> compromise in some area, be it performance, security, API issues, code 
> cleanliness issues, etc.  Or simply that the POSIX-related code just 
> isn't ready to be merged into the mainline kernel yet.
> 
> The vendors also benefit by this, because the barrier to entry in 
> POSIX-related cases would be lowered, which would in turn satisfy the 
> demands of customers.  Which would in turn give the mainline kernel all 
> the software engineering benefits that come from a more reasoned and 
> gradual review and merge of new features.
> 
> Does something like this already exist?  This would need to be an open, 
> vendor-neutral project...

What about the existing POSIX test suite from X/Open?  I don't know what
the current license is, but it is certainly freely downloadable from
their website.  However, it is a pain in the a** to set up and run, so
a new version would definitely be welcome.

I would give you a URL, but I don't have access to my mail archives now.

Cheers, Andreas
--
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: RFC: A POSIX Linux project?

[Rik van Riel]

On Mon, 4 Nov 2002, Jeff Garzik wrote:

> I wonder if any vendors, or independent groups, would be interested in
> maintaining a POSIX compliancy patchkit for the Linux kernel?

I'd be interested in helping.  There must be some POSIXisms
that are not in the kernel yet but still useful enough to keep
alive.

regards,

Rik
-- 
Bravely reimplemented by the knights who say "NIH".
http://www.surriel.com/		http://distro.conectiva.com/
Current spamtrap:  <a href=mailto:"october@surriel.com">october@surriel.com</a>

Re: RFC: A POSIX Linux project?

[Jim Freeman]

On Mon, Nov 04, 2002 at 04:31:13PM -0700, Andreas Dilger wrote:
...
> What about the existing POSIX test suite from X/Open?  I don't know what
> the current license is, but it is certainly freely downloadable from
> their website.  However, it is a pain in the a** to set up and run, so
> a new version would definitely be welcome.
> 
> I would give you a URL, but I don't have access to my mail archives now.

Howzabout the NIST one ?

	http://www.itl.nist.gov/div897/ctg/posix_form.htm

Re: [ANNOUNCE] Open POSIX Test Suite

[Christopher Yeoh]

Hi Geoff,

At 2002/11/4 14:48-0800  Geoff Gustafson writes:
> 
> Contributions of any test cases, review of the work, discussion of the
> approach, etc. are very welcome. Join the development mailing list,
> posixtest-discuss. The initial focus is on Signals, Message Queues, Threads,
> Semaphores, and Clocks & Timers, based on current interests and resources.
> You can help in these areas, or start work on another area of the spec.
> There
> will need to be some uniformity across the suite, but many details have yet
> to
> be worked out, so your involvement in those decisions help a lot.

Have you looked at the LSB test suites yet? They already cover much of
what you plan on writing tests for, though we would welcome any
volunteers who would like to increase the coverage.  Most of the tests
suites are released under the Artistic License, with quite a bit of
the code donated by the Open Group (originally from the Unix
certification tests).

You can find more information about them here:

http://www.linuxbase.org/test

The CVS repository is on SourceForge in the LSB project.

Chris
-- 
cyeoh@au.ibm.com
IBM OzLabs Linux Development Group
Canberra, Australia

Re: RFC: A POSIX Linux project?

[Geoff Gustafson]

Jim Freeman wrote:
> Howzabout the NIST one ?
>
> http://www.itl.nist.gov/div897/ctg/posix_form.htm

I think this is based on the 1990 POSIX spec. The things the new suite is
starting with have been added since then, up to and including the 2001 spec.

It is public domain I believe, so I would like to merge that in at some point,
but I would want to review the test cases individually to make sure they still
apply to the latest spec, etc. A lot of the work is just checking things in the
latest specification and adding/updating references in the code.

-- Geoff Gustafson

These are my views and not necessarily those of my employer.

Re: [ANNOUNCE] Open POSIX Test Suite

[Geoff Gustafson]

Christopher Yeoh writes:
> Have you looked at the LSB test suites yet? They already cover much of
> what you plan on writing tests for, though we would welcome any
> volunteers who would like to increase the coverage.  Most of the tests
> suites are released under the Artistic License, with quite a bit of
> the code donated by the Open Group (originally from the Unix
> certification tests).

One issue is that this new project is primarily concerned with testing parts of
the spec that have not been fully supported in Linux so far. These are the kind
of things that are not included in LSB yet, so they wouldn't be appropriate in
LSB's test suite.

Another problem is the overhead of the TET framework. One of the goals of this
the new test suite is to have test cases which are utterly minimal. So far, each
test case has its own main() function and a bare minimum of surrounding code.
The idea is that when a bug is found, this one .c file can be sent to the
appropriate developer, and without any learning curve, they have the ability to
find their bug. I don't think LKML wants to see TET code posted here. :)

-- Geoff Gustafson

These are my views and not necessarily those of my employer.

Re: RFC: A POSIX Linux project?

[Geoff Gustafson]

> I wonder if any vendors, or independent groups, would be interested in
> maintaining a POSIX compliancy patchkit for the Linux kernel?

I agree this sounds very useful. I could do something like this as part of the
test suite project; this would expand the scope to include testing and reporting
the status of the latest patches.

-- Geoff Gustafson

These are my views and not necessarily those of my employer.

Re: RFC: A POSIX Linux project?

[Geoff Gustafson]

Andreas Dilger wrote:
> What about the existing POSIX test suite from X/Open?  I don't know what
> the current license is, but it is certainly freely downloadable from
> their website.  However, it is a pain in the a** to set up and run, so
> a new version would definitely be welcome.

If I understand correctly, the test suites that cover POSIX extensions more
recent than 1990 are not free. Also, these use the TET framework, whereas this
project hopes to keep tests very simple and standalone, so the code can be
immediately sent to and warmly received by developers if bugs are found.

http://www.opengroup.org/testing/sales+support/prices.htm#VSTRC

-- Geoff Gustafson

These are my views and not necessarily those of my employer.

Re: [ANNOUNCE] Open POSIX Test Suite

[Andreas Dilger]

On Nov 04, 2002  16:44 -0800, Geoff Gustafson wrote:
> Another problem is the overhead of the TET framework. One of the goals of
> this the new test suite is to have test cases which are utterly minimal.
> So far, each test case has its own main() function and a bare minimum of
> surrounding code. The idea is that when a bug is found, this one .c file
> can be sent to the appropriate developer, and without any learning curve,
> they have the ability to find their bug. I don't think LKML wants to see
> TET code posted here. :)

Having suffered through using the TET framework for the Open Group POSIX
test suite, I would agree that using TET sucks.  The code is so convoluted
as to be useless, and it is nearly impossible to see from the output what
it is actually doing.

I agree that having a simple C or shell or perl script which is the entire
test, and the rest of the framework is external to it is very desirable.

Cheers, Andreas
--
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [ANNOUNCE] Open POSIX Test Suite

[Christopher Yeoh]

At 2002/11/4 16:44-0800  Geoff Gustafson writes:
> One issue is that this new project is primarily concerned with
> testing parts of the spec that have not been fully supported in
> Linux so far. These are the kind of things that are not included in
> LSB yet, so they wouldn't be appropriate in LSB's test suite.

Actually we do include some areas that aren't yet in the LSB spec or
fully supported by Linux (eg aio) in the LSB test suites we release -
they just aren't run by default - but it is easy to enable them.

> utterly minimal. So far, each test case has its own main() function
> and a bare minimum of surrounding code.  The idea is that when a bug
> is found, this one .c file can be sent to the appropriate developer,
> and without any learning curve, they have the ability to find their
> bug. I don't think LKML wants to see TET code posted here. :)

Yes, I agree TET does have a significant learning curve, and I do end
up writing small test programs that don't include the TET stuff before
sending off bug reports.

I have however seen some advantages - It is nice when you get a test
failure the report tells you exactly which part of the specification
you're violating. Once you do understand the TET/vsxgen library calls
testcases look much simpler - and if you're aiming for complete
functionality coverage including all the tricky corner cases for
various interfaces which can require quite a bit of setup code to get
into the right situation I think you'll end up having to write helper
libraries anyway.

Chris
-- 
cyeoh@au.ibm.com
IBM OzLabs Linux Development Group
Canberra, Australia