Re: [Qemu-devel] [PATCH] BootLinuxSshTest: Only run the tests when explicitly requested

[Cleber Rosa <crosa@redhat.com>]

On Wed, Sep 18, 2019 at 09:23:59PM -0400, Cleber Rosa wrote:
> On Wed, Sep 18, 2019 at 02:27:48PM +0200, Philippe Mathieu-Daudé wrote:
> > Currently the Avocado framework does not distinct the time spent
> > downloading assets vs. the time spent running a test. With big
> > assets (like a full VM image) the tests likely fail.
> > 
> > This is a limitation known by the Avocado team.
> > Until this issue get fixed, do not run this tests automatically.
> > 
> > Tests can still be run setting the AVOCADO_TIMEOUT_EXPECTED
> > environment variable.
> > 
> > Reported-by: Gerd Hoffmann <kraxel@redhat.com>
> > Reported-by: David Gibson <david@gibson.dropbear.id.au>
> > Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> > ---
> >  tests/acceptance/linux_ssh_mips_malta.py | 10 +++++-----
> >  1 file changed, 5 insertions(+), 5 deletions(-)
> > 
> > diff --git a/tests/acceptance/linux_ssh_mips_malta.py b/tests/acceptance/linux_ssh_mips_malta.py
> > index 134f10cac3..7200507a3a 100644
> > --- a/tests/acceptance/linux_ssh_mips_malta.py
> > +++ b/tests/acceptance/linux_ssh_mips_malta.py
> > @@ -12,7 +12,7 @@ import logging
> >  import paramiko
> >  import time
> >  
> > -from avocado import skipIf
> > +from avocado import skipUnless
> >  from avocado_qemu import Test
> >  from avocado.utils import process
> >  from avocado.utils import archive
> > @@ -171,7 +171,7 @@ class LinuxSSH(Test):
> >          self.run_common_commands()
> >          self.shutdown_via_ssh()
> >  
> > -    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
> > +    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
> 
> This is definitely an improvement, specially because IMO a "make
> check-acceptance" command should be more similar across executions,
> wether on Travis or not.  The problem is (which is not new) is that
> users have to resort to the test to learn how to run those tests.
> 
> FIY, I'm working on a RFC to have a handful of "make
> check-acceptance-$(TYPE)" targets that hopefully can represent most
> users' needs.  The avocado command line would still be available for
> more advanced users.
> 
> >      def test_mips_malta32eb_kernel3_2_0(self):
> >          """
> >          :avocado: tags=arch:mips
> > @@ -186,7 +186,7 @@ class LinuxSSH(Test):
> >  
> >          self.check_mips_malta('be', kernel_path, 'mips')
> >  
> > -    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
> > +    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
> >      def test_mips_malta32el_kernel3_2_0(self):
> >          """
> >          :avocado: tags=arch:mipsel
> > @@ -201,7 +201,7 @@ class LinuxSSH(Test):
> >  
> >          self.check_mips_malta('le', kernel_path, 'mips')
> >  
> > -    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
> > +    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
> >      def test_mips_malta64eb_kernel3_2_0(self):
> >          """
> >          :avocado: tags=arch:mips64
> > @@ -215,7 +215,7 @@ class LinuxSSH(Test):
> >          kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> >          self.check_mips_malta('be', kernel_path, 'mips64')
> >  
> > -    @skipIf(os.getenv('CONTINUOUS_INTEGRATION'), 'Running on Travis-CI')
> > +    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
> >      def test_mips_malta64el_kernel3_2_0(self):
> >          """
> >          :avocado: tags=arch:mips64el
> > -- 
> > 2.20.1
> > 
> 
> Reviewed-by: Cleber Rosa <crosa@redhat.com>

BTW, also queueing this one on my python-next branch.