From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([209.51.188.92]:51222) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1grQkv-0000EX-Af for qemu-devel@nongnu.org; Wed, 06 Feb 2019 12:13:02 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1grQbJ-0007tl-11 for qemu-devel@nongnu.org; Wed, 06 Feb 2019 12:03:07 -0500 References: <20190202005610.24048-1-crosa@redhat.com> <20190202005610.24048-3-crosa@redhat.com> <20190206153639.61a5411c.cohuck@redhat.com> From: Cleber Rosa Message-ID: <85b0f519-38d8-cf7d-a22a-537dab8470a7@redhat.com> Date: Wed, 6 Feb 2019 12:02:55 -0500 MIME-Version: 1.0 In-Reply-To: <20190206153639.61a5411c.cohuck@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [PATCH v2 02/20] Acceptance tests: show avocado test execution by default List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Cornelia Huck Cc: qemu-devel@nongnu.org, Aleksandar Markovic , Caio Carrara , Wainer dos Santos Moschetta , =?UTF-8?Q?Philippe_Mathieu-Daud=c3=a9?= , Eduardo Habkost , =?UTF-8?Q?Alex_Benn=c3=a9e?= , Aleksandar Rikalo , Aurelien Jarno , Fam Zheng , qemu-s390x@nongnu.org, Stefan Markovic On 2/6/19 9:36 AM, Cornelia Huck wrote: > On Fri, 1 Feb 2019 19:55:52 -0500 > Cleber Rosa wrote: >=20 >> The current version of the "check-acceptance" target will only show >> one line for execution of all tests. That's probably OK if the tests >> to be run are quick enough and they're always the same. >> >> But, there's already one test alone that takes on average ~5 seconds >> to run, we intend to adapt the list of tests to match the user's build >> environment (among other choices). >=20 > Btw: What are our expectations regarding execution time for tests? > Especially if we continue adding tests, and architecture-specific tests > are bound to be slower if run on a foreign architecture via tcg. >=20 > Would a make check-acceptance-quick command make sense? ("I only want > to verify quickly that I didn't break too much, so run the quicker > tests only, probably only for my host architecture") >=20 Yes, it definitely makes sense. Now, let me know if the following also makes sense to you: 1) Because these tests focus on functional testing, the default target/shortcut ("make check-acceptance") should run the complete set of test cases (including the slow ones). 2) Requirements vary greatly from user to user, to while adding a "check-acceptance-quick" is fine, you just mentioned one extra test execution variation ("for my host architecture"). For those, the idea is that: a) "make check-acceptance[-quick]" will adapt to the build environment (if you only built s390x targets, that's all it's going to use) b) "avocado" command line interface *should* be easy enough to fulfill other requirements, and not necessarily require a "make" target. For instance, if you're only interested in your host arch and one specific machine type, a command line such as the following should do the trick: $ make check-venv $ ./tests/venv/bin/avocado run -t arch:`uname -m` -t machine:WHAT_I_CARE_ABOUT tests/acceptance/ How does that sound? Regards, - Cleber. >> >> Because of that, let's present the default Avocado UI by default. >> Users can always choose a different output by setting the AVOCADO_SHOW >> variable. >> >> Reviewed-by: Caio Carrara >> Reviewed-by: Philippe Mathieu-Daud=C3=A9 >> Reviewed-by: Alex Benn=C3=A9e >> Signed-off-by: Cleber Rosa >> --- >> .travis.yml | 2 +- >> tests/Makefile.include | 2 +- >> 2 files changed, 2 insertions(+), 2 deletions(-)