Re: [Notes for xen summit 2018 design session] Process changes: is the 6 monthly release Cadence too short, Security Process, ...

From: Juergen Gross <jgross@suse.com>
To: Ian Jackson <ian.jackson@citrix.com>
Cc: Lars Kurth <lars.kurth@citrix.com>,
	Andrew Cooper <andrew.cooper3@citrix.com>,
	"advisory-board@lists.xenproject.org"
	<advisory-board@lists.xenproject.org>,
	Doug Goldstein <cardoe@cardoe.com>,
	George Dunlap <George.Dunlap@citrix.com>,
	Rich Persaud <persaur@gmail.com>,
	"committers@xenproject.org" <committers@xenproject.org>,
	xen-devel <xen-devel@lists.xenproject.org>,
	Matt Spencer <Matt.Spencer@arm.com>
Subject: Re: [Notes for xen summit 2018 design session] Process changes: is the 6 monthly release Cadence too short, Security Process, ...
Date: Fri, 6 Jul 2018 10:58:30 +0200	[thread overview]
Message-ID: <4f90cd43-2781-a8e1-17f8-15b974f1aad4@suse.com> (raw)
In-Reply-To: <23358.13791.774484.444256@mariner.uk.xensource.com>

On 05/07/18 17:14, Ian Jackson wrote:
> Juergen Gross writes ("Re: [Xen-devel] [Notes for xen summit 2018 design session] Process changes: is the 6 monthly release Cadence too short, Security Process, ..."):
>> Same applies to the host: the base system (without the to be tested
>> component like qemu, xen, or whatever) could be installed just by
>> cloning a disk/partition/logical volume.
> 
> Certainly it would be a bad idea to use anything *on the test host
> itself* as a basis for a subsequent test.  The previous test might
> have corrupted it.

Right. Not sure, whether possible, but in an ideal environment we'd have
an external storage system reachable by the control nodes and the test
systems. The test systems should be able to access their test data only,
while the control nodes would initialize the test data while the related
test machine is still offline.

>> Each image would run through the stages new->staging->stable:
>>
>> - Each time a component is released an image is based on (e.g. a new
>>   mainline kernel) a new image is created by installing it. In case this
>>   succeeds, the image is moved to the staging area.
> 
> This would happen a lot more often than you seem to image.  "Releaed"
> here really means "is updated in its appropriate git branch".
> 
> Unless you think we should do our testing of Xen mainly with released
> versions of Linux stable branches (in which case, given how Linux
> stable branches are often broken, we might be long out of date), or
> our testing of Linux only with point releases of Xen, etc.

Yes, that's what I think.

The Xen (hypervisor, tools) tests should be done with released kernels
(either stable or the last one from upstream).

Tests of Linux Xen support should be done with released Xen versions.

> The current approach is mostly to take the most recent
> tested-and-working git commit from each of the inputs.  This aspect of
> osstest generally works well, I think.

We have a bandwidth problem already. If one unstable input product is
failing all related tests do so, too. I'd rather know which of the
input sources is the most probable one to be blamed for a test failure.

Another aspect of using stable versions is the possibility to find even
performance regressions automatically. Changing multiple versions
between tests makes that impossible, as you don't know who is to blame.

Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel