From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: References: From: Agustin Benito Bethencourt Message-ID: <5818A4D1.1070508@codethink.co.uk> Date: Tue, 1 Nov 2016 15:21:05 +0100 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Fuego] [Ksummit-discuss] Some ideas on open source,> testing List-Id: Mailing list for the Fuego test framework List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: fuego@lists.linuxfoundation.org Hi it is my first mail here. Apologies for answering based on a digest mail. I already disabled the digest mode for the future. > Message: 2 > Date: Mon, 24 Oct 2016 17:41:16 +0100 > From: Mark Brown > To: Theodore Ts'o > Cc: "ksummit-discuss@lists.linuxfoundation.org" > , > "fuego@lists.linuxfoundation.org" > Subject: Re: [Fuego] [Ksummit-discuss] Some ideas on open source > testing > Message-ID: <20161024164116.GB17252@sirena.org.uk> > Content-Type: text/plain; charset="us-ascii" > > On Fri, Oct 21, 2016 at 09:19:25PM -0400, Theodore Ts'o wrote: > >> And so you might have alarge number of workflows: > >> * Developers who want a fast smoke test after applying a patch or >> patch set. >> * Developers who want to dig into a specific test failure, and who >> want to be able to very quickly iterate over running a specific test >> against a quick succession of test kernels. >> * Maintainers who want to run a much more comprehensive test suite >> that might take hours to run. * Release engineers who want some >> kind of continuous integration test. > >> A test runner framework which is good for one is very likely not going >> to be good for another. > > That's true but there's also an awful lot that can be shared - for > example, the mechanics of booting on a given system are going to be the > same regardless of how the test was scheduled or how long it will run > for. We should be looking for ways to share things as much as we can, > especially around the bits where the skills needed to implement align > less well with kernel skills. > > [UI for reporting] >> I'm hoping to have an intern try to create something like that for >> gce-xfstests that would run on Google App Engine, over the next couple >> of months. So maybe by next year I'll have something that we'll be >> able to show off. We'll see.... > > This sort of UI thing seems like a prime example of an area where we > could really use some sharing - for example we've got some capacity to > run tests in kernelci but nobody to work on UI for doing anything useful > with the results. For UI testing there is an option for systems out there today, openQA, used in production by SUSE and more recently by Red Hat (Fedora). I would look into it before thinking about extending kernelci to UI testing, so the complexity of what today is successful, among other reasons, for its simplicity. When it comes to boot testing (acceptance testing) I wonder what is the percentage of cases in which testing on real hardware is required vs testing on emulated environments. My experience on this has showed me that the percentage I assumed in initially (in theory) was lower than in practice (in favour of emulation). Depending on what is that percentage for you, openQA might be a good enough answer vs extending kernelci/Fuego. openQA test visualization: https://openqa.opensuse.org/ Link to the project: http://open.qa/ I also believe Fuego can use openQA as a reference for some decisions that need to be taken in the future. There are not that many projects out there for system testing that are fully open, developed and used by people willing to help others. Best Regards -- Agustin Benito Bethencourt Principal Consultant - FOSS at Codethink agustin.benito@codethink.co.uk