moving towards release criteria

[Mark Kampe <mark.kampe@dreamhost.com>]

At present we are running automated nightlies to catch
problems that slip past developers or only show up on
long runs, and filing bugs when they fail ... but the
decision of whether or not we are ready to push out a
new release is not yet criterion-based.  We have to
start moving towards official release criteria.

I suggest that our initial release criteria should fall
into four categories:

   (1) functional validation

	100% passage of designated validation suites,
	with a formal process for managing the functional
	assertions to be tested (or designating specific
	assertions to be compliance-optional)
	
   (2) regression tests

	100% passage of designated regression suites,
	with a formal process for designating which
	bugs do and do not require the creation of
	new regression test cases.

   (3) performance

	individual and aggregate throughput measurements,
	and key-event timings will be made with controlled
	loads on specified hardware configuration, and
	compared against performance targets, and a formal
	process for defining the target metrics and requirements.

   (4) reliability and robustness

	a specified number of hours of (client perceived) error
	free operation under continuous load (with specified
	levels and characteristics), in the face of specified
	error injections ... and a formal process for defining
	the times, load characteristics, error injections, and
	acceptable performance.

Does this seem like the right general form for our release criteria? 
What changes would you suggest?

Once we agree on the general form of our release criteria, the next
steps are:

   (a) put some stakes in the ground for the initial requirements
       (knowing that they will evolve in scope, specificity, and
       rigour)

   (b) propose some processes for the review, approval, and
       evolution of those standards, and the communication of
       the (current) requirements and results to the community.

   (c) set a date for the first release to be subject to these
       criteria

comments?