From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05D52C4CEC9 for ; Thu, 19 Sep 2019 01:24:59 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CF5DF21925 for ; Thu, 19 Sep 2019 01:24:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CF5DF21925 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:36790 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iAlBp-0007jq-VY for qemu-devel@archiver.kernel.org; Wed, 18 Sep 2019 21:24:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45104) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iAlAw-0007IL-6e for qemu-devel@nongnu.org; Wed, 18 Sep 2019 21:24:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iAlAv-0005uE-1j for qemu-devel@nongnu.org; Wed, 18 Sep 2019 21:24:02 -0400 Received: from mx1.redhat.com ([209.132.183.28]:34782) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1iAlAu-0005ty-PL for qemu-devel@nongnu.org; Wed, 18 Sep 2019 21:24:00 -0400 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id B7111308FC4A; Thu, 19 Sep 2019 01:23:59 +0000 (UTC) Received: from localhost.localdomain (ovpn-123-8.rdu2.redhat.com [10.10.123.8]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 955105C22C; Thu, 19 Sep 2019 01:23:52 +0000 (UTC) Date: Wed, 18 Sep 2019 21:23:51 -0400 From: Cleber Rosa To: Philippe =?iso-8859-1?Q?Mathieu-Daud=E9?= Message-ID: <20190919012351.GB23168@localhost.localdomain> References: <20190918122748.2144-1-philmd@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <20190918122748.2144-1-philmd@redhat.com> User-Agent: Mutt/1.12.1 (2019-06-15) X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.43]); Thu, 19 Sep 2019 01:24:00 +0000 (UTC) Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: Re: [Qemu-devel] [PATCH] BootLinuxSshTest: Only run the tests when explicitly requested X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Aleksandar Rikalo , qemu-devel@nongnu.org, Gerd Hoffmann , Aurelien Jarno , David Gibson Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On Wed, Sep 18, 2019 at 02:27:48PM +0200, Philippe Mathieu-Daud=E9 wrote: > Currently the Avocado framework does not distinct the time spent > downloading assets vs. the time spent running a test. With big > assets (like a full VM image) the tests likely fail. >=20 > This is a limitation known by the Avocado team. > Until this issue get fixed, do not run this tests automatically. >=20 > Tests can still be run setting the AVOCADO_TIMEOUT_EXPECTED > environment variable. >=20 > Reported-by: Gerd Hoffmann > Reported-by: David Gibson > Signed-off-by: Philippe Mathieu-Daud=E9 > --- > tests/acceptance/linux_ssh_mips_malta.py | 10 +++++----- > 1 file changed, 5 insertions(+), 5 deletions(-) >=20 > diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptanc= e/linux_ssh_mips_malta.py > index 134f10cac3..7200507a3a 100644 > --- a/tests/acceptance/linux_ssh_mips_malta.py > +++ b/tests/acceptance/linux_ssh_mips_malta.py > @@ -12,7 +12,7 @@ import logging > import paramiko > import time > =20 > -from avocado import skipIf > +from avocado import skipUnless > from avocado_qemu import Test > from avocado.utils import process > from avocado.utils import archive > @@ -171,7 +171,7 @@ class LinuxSSH(Test): > self.run_common_commands() > self.shutdown_via_ssh() > =20 > - @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI= ') > + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might tim= eout') This is definitely an improvement, specially because IMO a "make check-acceptance" command should be more similar across executions, wether on Travis or not. The problem is (which is not new) is that users have to resort to the test to learn how to run those tests. FIY, I'm working on a RFC to have a handful of "make check-acceptance-$(TYPE)" targets that hopefully can represent most users' needs. The avocado command line would still be available for more advanced users. > def test_mips_malta32eb_kernel3_2_0(self): > """ > :avocado: tags=3Darch:mips > @@ -186,7 +186,7 @@ class LinuxSSH(Test): > =20 > self.check_mips_malta('be', kernel_path, 'mips') > =20 > - @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI= ') > + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might tim= eout') > def test_mips_malta32el_kernel3_2_0(self): > """ > :avocado: tags=3Darch:mipsel > @@ -201,7 +201,7 @@ class LinuxSSH(Test): > =20 > self.check_mips_malta('le', kernel_path, 'mips') > =20 > - @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI= ') > + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might tim= eout') > def test_mips_malta64eb_kernel3_2_0(self): > """ > :avocado: tags=3Darch:mips64 > @@ -215,7 +215,7 @@ class LinuxSSH(Test): > kernel_path =3D self.fetch_asset(kernel_url, asset_hash=3Dkern= el_hash) > self.check_mips_malta('be', kernel_path, 'mips64') > =20 > - @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI= ') > + @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might tim= eout') > def test_mips_malta64el_kernel3_2_0(self): > """ > :avocado: tags=3Darch:mips64el > --=20 > 2.20.1 >=20 Reviewed-by: Cleber Rosa