From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ian Campbell Subject: Re: [OSSTEST Nested PATCH v11 6/7] Compose the main recipe of nested test job Date: Fri, 12 Jun 2015 10:15:00 +0100 Message-ID: <1434100500.30003.198.camel@citrix.com> References: <1432631304-27347-1-git-send-email-longtaox.pang@intel.com> <1432631304-27347-7-git-send-email-longtaox.pang@intel.com> <21880.23264.813171.606123@mariner.uk.xensource.com> <86C3224E41A7434B904EC364302132D80E4BFEC4@SHSMSX101.ccr.corp.intel.com> <21881.42757.84361.654532@mariner.uk.xensource.com> <1434080545.32731.13.camel@localhost> <1434098672.30003.189.camel@citrix.com> <1434099619.29629.7.camel@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1434099619.29629.7.camel@localhost> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: robert.hu@intel.com Cc: "wei.liu2@citrix.com" , "Pang, LongtaoX" , Ian Jackson , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org On Fri, 2015-06-12 at 17:00 +0800, Robert Hu wrote: > On Fri, 2015-06-12 at 09:44 +0100, Ian Campbell wrote: > > On Fri, 2015-06-12 at 11:42 +0800, Robert Hu wrote: > > > Hi Ian J., because nested Xen isn't that matured at present; it is > > > currently 'tech preview' phase. We now just add some sanity test case to > > > defend current work fruits. We can add more complicated test cases later > > > as nested Xen development moves forward. > > > > I don't think it needs doing as part of this series, but I do think it > > would be worth adding the standard suite of tests steps to L2 guests > > soon after. > Agree. We can separately add those part in a later patch series. > > > > The way osstest deals with test failures i.e. classifying them into "has > > always failed" vs. "regression" means that it is fine to add tests for > > features which aren't mature yet, since they will fall into the "has > > always failed" set and not block pushes. In fact it is in some sense > > good to do this since it gives a concrete set of (necessary but not > > necessarily sufficient) things which need to be fixed for the feature to > > reach maturity. > We would like this test job's failure to be marked 'regression', which > shall block related breaking code's commitment. This is all done automatically, by comparison with previous flights. In essence if it passed last time then it will be considered a regression if it fails and for as long as it keeps failing (unless it is manually overridden via a forced push). So there is nothing to be "marked". > We can consider to contribute more code later, including proved/success > test cases (which will fall into regression as this case), and fail test > cases (which shall soon be fixed/developed). You should just add the cases, await the first results and investigate if it is not a pass as expected. Ian.