From mboxrd@z Thu Jan 1 00:00:00 1970 From: Daniel Vetter Subject: Re: The whole round of i-g-t testing cost too long running time Date: Wed, 16 Apr 2014 17:50:20 +0200 Message-ID: <534EA6BC.4080906@intel.com> References: <28F4DB836A24D74D9070047733650BEF868F24@shsmsx102.ccr.corp.intel.com> <534D69C7.4070601@intel.com> <20140416084227.1356f68c@jbarnes-t420> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by gabe.freedesktop.org (Postfix) with ESMTP id 1F4966E3DB for ; Wed, 16 Apr 2014 08:51:23 -0700 (PDT) In-Reply-To: <20140416084227.1356f68c@jbarnes-t420> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" To: Jesse Barnes Cc: OTC GFX QA Extended , "Nikkanen, Kimmo" , "intel-gfx@lists.freedesktop.org" , "Widawsky, Benjamin" , "Jin, Gordon" , "Parenteau, Paul A" List-Id: intel-gfx@lists.freedesktop.org On 16/04/2014 17:42, Jesse Barnes wrote: > On Tue, 15 Apr 2014 19:17:59 +0200 > Daniel Vetter wrote: >> Ok there are a few cases where we can indeed make tests faster, but it >> will be work for us. And that won't really speed up much since we're >> adding piles more testcases at a pretty quick rate. And many of these >> new testcases are CRC based, so inheritely take some time to run. > But each test should run very quickly in general; I think we have too > many tests that take much longer than they need to. Adding some > hooks to the driver via debugfs may let us trigger specific cases > directly rather than trying to induce them through massive threading > and memory pressure for example. > > And can you elaborate on the CRC tests? It doesn't seem like those > should take more than a few frames to verify we're getting what we > expect... Well they don't take more than a few frames, but we have a _lot_ of them, and there's a lot of cominations to test. It adds up quickly. Iirc we have over 150 kms_flip testcases alone ... Like I've said I agree that we could speed tests up, but besides me doing the occasional tuning and improvement in that regard I have seen 0 patches from developers in this area. Which lets me conclude that apparently it's not reallly that bad an isssue ;-) If people _really_ care about this I have a list of things to knock down. But first someone needs to find some time and resources for this. > >> So I think longer-term we simply need to throw more machines at the >> problem and run testcases in parallel on identical machines. >> >> Wrt analyzing issues I think the right approach for moving forward is: >> a) switch to piglit to run tests, not just enumerate them. This will >> allow QA and developers to share testcase analysis. >> b) add automated analysis for time-consuming and error prone cases like >> dmesg warnings and backtraces. Thomas&I have just discussed a few ideas >> in this are in our 1:1 today. >> >> Reducing the set of igt tests we run is imo pointless: The goal of igt >> is to hit corner-cases, arbitrarily selecting which kinds of >> corner-cases we test just means that we have a nice illusion about our >> test coverage. > My goal is still to get full test coverage before patches get > committed, and that means having quick (<1hr) turnaround for testing > from the automated patch test system. It seems like we'll need to > approach that from all angles: speeding up tests, parallelizing > execution, adding hooks to the driver, etc. > > Currently <1h and "full test coverage" are rather mutually exclusive unfortunately :( I agree that having it would be extremely useful for developers, but if this happens with any cost/service reduction for nightly testing on my branches I'm really opposed. Atm we already have a _really_ hard time keeping track of all the various regressions and bugs. -Daniel Intel Semiconductor AG Registered No. 020.30.913.786-7 Registered Office: Badenerstrasse 549, 8048 Zurich, Switzerland