From mboxrd@z Thu Jan 1 00:00:00 1970 From: Daniel Vetter Subject: Re: [igt] Making the test-suite easier to run Date: Fri, 15 Nov 2013 18:23:18 +0100 Message-ID: <20131115172318.GY22741@phenom.ffwll.local> References: <1384533220-20780-1-git-send-email-damien.lespiau@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail-ee0-f53.google.com (mail-ee0-f53.google.com [74.125.83.53]) by gabe.freedesktop.org (Postfix) with ESMTP id DC113FB46A for ; Fri, 15 Nov 2013 09:22:44 -0800 (PST) Received: by mail-ee0-f53.google.com with SMTP id b57so1153556eek.26 for ; Fri, 15 Nov 2013 09:22:43 -0800 (PST) Content-Disposition: inline In-Reply-To: <1384533220-20780-1-git-send-email-damien.lespiau@intel.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: intel-gfx-bounces@lists.freedesktop.org Errors-To: intel-gfx-bounces@lists.freedesktop.org To: Damien Lespiau Cc: intel-gfx@lists.freedesktop.org List-Id: intel-gfx@lists.freedesktop.org On Fri, Nov 15, 2013 at 04:33:17PM +0000, Damien Lespiau wrote: > The objective of this series is to make the test-suite easier to run by > embedding a copy a piglit and providing porcelain on top of it in the form of a > makefile target. Beside python, there's no external dependency to run the test > suite after this series. > > The provided makefile target runs the full test suite. I'm still interested in > providing, as a follow-up of this work, shorcuts for subsets of the testsuite > that can be useful for developpers, subset what we would maintain in tree. For > instance: > - tests selected by topic: running gem_.* Vs kms_.* Vs pm_.* (or maybe when > running, say, gt tests, exlude ^kms_.* ^pm_.* so we still still a lot of > the other tests (core_, drm_, ...) > - quick subtests (where we disable long running stress, race, ... tests) > > patches 21-23 are something a bit different, try to pave the way for quick > runs (by really the first tiny step). I'm ok with doing the slow/quick thing if people like it.o On the real deal (i.e. patches 1-20) a few overall comments: - I think we need to make it very clear that piglit should still be developed upstream. So local changes shouldn't be allowed at all imo. Afaics that would only affect tests/igt.tests - can't we fix that by replicating the symlink, too? - The makefile target looks like a script. I think it'd be better to extract it as a real script. - I'm not too terribly sold on the convenience script. Imo it shouldn't be more than executable documentation, since I'm a bit afraid that we'll add neat features (like fancy resume with auto-blocking of crashing tests) to it that would better be done in upstream piglit to benefit everyone. But overall I agree that we need a local copy of piglit since apparently way too few people on our team us it to run igts. And running igts without piglit is just nuts. Cheers, Daniel > > The README has been updated and I copy/paste it here the documentation for what > would be the new way to run tests: > > After having compiled the tests, one can run the test-suite with: > > $ sudo make run-tests > > "make run-tests" create a $date-piglit-results.$n directory with the > results of the run. More specifically: > - $date-piglit-results.$n/main JSON file with the test results > - $date-piglit-results.$n/html/index.html HTML summary of the run > > Where $date is the date formated with `date +%Y%m%d` and $n the nth run > of the day. > > PIGLIT_FLAGS can be used to give options to the underlying piglit > runner. For instance, to exclude test matching '^kms_': > > $ sudo make run-tests PIGLIT_FLAGS="-x ^kms_" > > For the list of piglit options, run: > > $ ./piglit/piglit-run.py -h > > Another useful feature is to be able to resume an interrupted run. To > do that, make run-tests needs to know which run we are talking about: > > $ sudo make run-tests RESUME=$date-piglit-results.$n > > or, more succinctly: > > $ sudo make run-tests R=$date-piglit-results.$n > > It's possible to combine PIGLIT_FLAGS and RESUME. This is useful to > resume runs where a specific test deterministically hang the machine: > > $ sudo make run-tests PIGLIT_FLAGS="-x drv_module_reload" R=$date-piglit-results.$n > > HTH, > > -- > Damien > > _______________________________________________ > Intel-gfx mailing list > Intel-gfx@lists.freedesktop.org > http://lists.freedesktop.org/mailman/listinfo/intel-gfx -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch