moving towards release criteria

moving towards release criteria

[Mark Kampe]

At present we are running automated nightlies to catch
problems that slip past developers or only show up on
long runs, and filing bugs when they fail ... but the
decision of whether or not we are ready to push out a
new release is not yet criterion-based.  We have to
start moving towards official release criteria.

I suggest that our initial release criteria should fall
into four categories:

   (1) functional validation

	100% passage of designated validation suites,
	with a formal process for managing the functional
	assertions to be tested (or designating specific
	assertions to be compliance-optional)
	
   (2) regression tests

	100% passage of designated regression suites,
	with a formal process for designating which
	bugs do and do not require the creation of
	new regression test cases.

   (3) performance

	individual and aggregate throughput measurements,
	and key-event timings will be made with controlled
	loads on specified hardware configuration, and
	compared against performance targets, and a formal
	process for defining the target metrics and requirements.

   (4) reliability and robustness

	a specified number of hours of (client perceived) error
	free operation under continuous load (with specified
	levels and characteristics), in the face of specified
	error injections ... and a formal process for defining
	the times, load characteristics, error injections, and
	acceptable performance.

Does this seem like the right general form for our release criteria? 
What changes would you suggest?

Once we agree on the general form of our release criteria, the next
steps are:

   (a) put some stakes in the ground for the initial requirements
       (knowing that they will evolve in scope, specificity, and
       rigour)

   (b) propose some processes for the review, approval, and
       evolution of those standards, and the communication of
       the (current) requirements and results to the community.

   (c) set a date for the first release to be subject to these
       criteria

comments?

Re: moving towards release criteria

[Tommi Virtanen]

On Mon, Nov 28, 2011 at 11:01, Mark Kampe <mark.kampe@dreamhost.com> wrote:
> Does this seem like the right general form for our release criteria? What
> changes would you suggest?

As long as we're blue-skying ;) -- at some point we should automate
tests for upgrades.

Simple first iteration: install previous release, add data, upgrade,
test functionality, test previous data is accessible
(multiply by access method: rados, rgw, rbd, cephfs fuse, cephfs kernel client)

Later:
To release x.y, upgrades from x.(y-1) and x.0 must work.
To release x.0, upgrades from (x-1).0 and (x-1).latest must work.
If these upgrades are not supported, release notes must say so.

Even later:
Test functionality *during* the upgrade.

Even more laterer:
Test extended cross-version compatibility, as in old-osd talks to
new-mon, or don't upgrade clients at all, etc..

Re: moving towards release criteria

[Wido den Hollander]

Hi,

On 11/29/2011 01:38 AM, Tommi Virtanen wrote:
> On Mon, Nov 28, 2011 at 11:01, Mark Kampe<mark.kampe@dreamhost.com>  wrote:
>> Does this seem like the right general form for our release criteria? What
>> changes would you suggest?
>
> As long as we're blue-skying ;) -- at some point we should automate
> tests for upgrades.
>

You could try to couple this to a specific version?

> Simple first iteration: install previous release, add data, upgrade,
> test functionality, test previous data is accessible
> (multiply by access method: rados, rgw, rbd, cephfs fuse, cephfs kernel client)
>

Starting with version 0.40

> Later:
> To release x.y, upgrades from x.(y-1) and x.0 must work.

Version 0.60 and later

> To release x.0, upgrades from (x-1).0 and (x-1).latest must work.

Version 0.80 and later

> If these upgrades are not supported, release notes must say so.
>
> Even later:
> Test functionality *during* the upgrade.
>

Version 0.90 and later

> Even more laterer:
> Test extended cross-version compatibility, as in old-osd talks to
> new-mon, or don't upgrade clients at all, etc..

Version 1.0

Since v1.0 would become the "stable" version you might expect users 
upgrading while maintaining their data.

Does this seem reasonable?

Wido

> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html