[PATCH 0/2] tests: Add 'fetch-acceptance' rule

[PATCH 0/2] tests: Add 'fetch-acceptance' rule

[Philippe Mathieu-Daudé]

Add a rule to fetch acceptance test assets.

This is particularly useful in a CI context, when a single job
can fetch and save the cache so other jobs reuse it directly.

It is also useful to measure the time spent downloading the
assets versus the time spent running the tests.

For now we have to exclude the 'boot_linux.py' which triggers
an exception.

Philippe Mathieu-Daudé (2):
  tests: Add 'fetch-acceptance' rule
  tests: Exclude 'boot_linux.py' from fetch-acceptance rule

 tests/Makefile.include | 9 +++++++++
 1 file changed, 9 insertions(+)

-- 
2.21.3

[PATCH 1/2] tests: Add 'fetch-acceptance' rule

[Philippe Mathieu-Daudé]

Add a rule to fetch acceptance test assets.

This is particularly useful in a CI context, when a single job
can fetch and save the cache so other jobs reuse it directly.

It is also useful to measure the time spent downloading the
assets versus the time spent running the tests.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/Makefile.include | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index c7e4646ded..238974d8da 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -944,6 +944,15 @@ get-vm-image-fedora-31-%: check-venv
 # download all vm images, according to defined targets
 get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
 
+# fetch acceptance test assets
+fetch-acceptance: check-venv
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m avocado \
+            $(if $(V),--show=$(AVOCADO_SHOW)) \
+            assets fetch \
+            $(wildcard tests/acceptance/*.py), \
+            "AVOCADO", "tests/acceptance")
+
 check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
 	$(call quiet-command, \
             $(TESTS_VENV_DIR)/bin/python -m avocado \
-- 
2.21.3

[PATCH 2/2] tests: Exclude 'boot_linux.py' from fetch-acceptance rule

[Philippe Mathieu-Daudé]

The boot_linux.py file triggers an exception:

  $ make fetch-acceptance
  AVOCADO tests/acceptance
  Fetching assets from tests/acceptance/empty_cpu_model.py.
  Fetching assets from tests/acceptance/vnc.py.
  Fetching assets from tests/acceptance/boot_linux_console.py.
  Fetching assets from tests/acceptance/boot_linux.py.
  Traceback (most recent call last):
    File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/__main__.py", line 11, in <module>
      sys.exit(main.run())
    File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/core/app.py", line 91, in run
      return method(self.parser.config)
    File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 291, in run
      success, fail = fetch_assets(test_file)
    File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 200, in fetch_assets
      handler = FetchAssetHandler(test_file, klass, method)
    File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 65, in __init__
      self.visit(self.tree)
    File "/usr/lib64/python3.7/ast.py", line 271, in visit
      return visitor(node)
    File "/usr/lib64/python3.7/ast.py", line 279, in generic_visit
      self.visit(item)
    File "/usr/lib64/python3.7/ast.py", line 271, in visit
      return visitor(node)
    File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 139, in visit_ClassDef
      self.generic_visit(node)
    File "/usr/lib64/python3.7/ast.py", line 279, in generic_visit
      self.visit(item)
    File "/usr/lib64/python3.7/ast.py", line 271, in visit
      return visitor(node)
    File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 171, in visit_Assign
      self.asgmts[cur_klass][cur_method][name] = node.value.s
  KeyError: 'launch_and_wait'
  make: *** [tests/Makefile.include:949: fetch-acceptance] Error 1

Exclude it for now. We will revert this commit once the script is
fixed.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 238974d8da..7c9cf7a818 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -950,7 +950,7 @@ fetch-acceptance: check-venv
             $(TESTS_VENV_DIR)/bin/python -m avocado \
             $(if $(V),--show=$(AVOCADO_SHOW)) \
             assets fetch \
-            $(wildcard tests/acceptance/*.py), \
+            $(filter-out tests/acceptance/boot_linux.py,$(wildcard tests/acceptance/*.py)), \
             "AVOCADO", "tests/acceptance")
 
 check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
-- 
2.21.3

Re: [PATCH 2/2] tests: Exclude 'boot_linux.py' from fetch-acceptance rule

[Wainer dos Santos Moschetta]

Hi Philippe,

On 7/24/20 4:35 AM, Philippe Mathieu-Daudé wrote:
> The boot_linux.py file triggers an exception:
>
>    $ make fetch-acceptance
>    AVOCADO tests/acceptance
>    Fetching assets from tests/acceptance/empty_cpu_model.py.
>    Fetching assets from tests/acceptance/vnc.py.
>    Fetching assets from tests/acceptance/boot_linux_console.py.
>    Fetching assets from tests/acceptance/boot_linux.py.
>    Traceback (most recent call last):
>      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/__main__.py", line 11, in <module>
>        sys.exit(main.run())
>      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/core/app.py", line 91, in run
>        return method(self.parser.config)
>      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 291, in run
>        success, fail = fetch_assets(test_file)
>      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 200, in fetch_assets
>        handler = FetchAssetHandler(test_file, klass, method)
>      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 65, in __init__
>        self.visit(self.tree)
>      File "/usr/lib64/python3.7/ast.py", line 271, in visit
>        return visitor(node)
>      File "/usr/lib64/python3.7/ast.py", line 279, in generic_visit
>        self.visit(item)
>      File "/usr/lib64/python3.7/ast.py", line 271, in visit
>        return visitor(node)
>      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 139, in visit_ClassDef
>        self.generic_visit(node)
>      File "/usr/lib64/python3.7/ast.py", line 279, in generic_visit
>        self.visit(item)
>      File "/usr/lib64/python3.7/ast.py", line 271, in visit
>        return visitor(node)
>      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 171, in visit_Assign
>        self.asgmts[cur_klass][cur_method][name] = node.value.s
>    KeyError: 'launch_and_wait'
>    make: *** [tests/Makefile.include:949: fetch-acceptance] Error 1

Currently the acceptance tests use Avocado 7.6. I bumped to 80.0 (latest 
released) here and that error is gone. Could you double check?

Regards,

Wainer

>
> Exclude it for now. We will revert this commit once the script is
> fixed.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   tests/Makefile.include | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 238974d8da..7c9cf7a818 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -950,7 +950,7 @@ fetch-acceptance: check-venv
>               $(TESTS_VENV_DIR)/bin/python -m avocado \
>               $(if $(V),--show=$(AVOCADO_SHOW)) \
>               assets fetch \
> -            $(wildcard tests/acceptance/*.py), \
> +            $(filter-out tests/acceptance/boot_linux.py,$(wildcard tests/acceptance/*.py)), \
>               "AVOCADO", "tests/acceptance")
>   
>   check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images

Re: [PATCH 2/2] tests: Exclude 'boot_linux.py' from fetch-acceptance rule

[Willian Rampazzo]

On Fri, Jul 24, 2020 at 3:26 PM Wainer dos Santos Moschetta
<wainersm@redhat.com> wrote:
>
> Hi Philippe,
>
> On 7/24/20 4:35 AM, Philippe Mathieu-Daudé wrote:
> > The boot_linux.py file triggers an exception:
> >
> >    $ make fetch-acceptance
> >    AVOCADO tests/acceptance
> >    Fetching assets from tests/acceptance/empty_cpu_model.py.
> >    Fetching assets from tests/acceptance/vnc.py.
> >    Fetching assets from tests/acceptance/boot_linux_console.py.
> >    Fetching assets from tests/acceptance/boot_linux.py.
> >    Traceback (most recent call last):
> >      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/__main__.py", line 11, in <module>
> >        sys.exit(main.run())
> >      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/core/app.py", line 91, in run
> >        return method(self.parser.config)
> >      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 291, in run
> >        success, fail = fetch_assets(test_file)
> >      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 200, in fetch_assets
> >        handler = FetchAssetHandler(test_file, klass, method)
> >      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 65, in __init__
> >        self.visit(self.tree)
> >      File "/usr/lib64/python3.7/ast.py", line 271, in visit
> >        return visitor(node)
> >      File "/usr/lib64/python3.7/ast.py", line 279, in generic_visit
> >        self.visit(item)
> >      File "/usr/lib64/python3.7/ast.py", line 271, in visit
> >        return visitor(node)
> >      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 139, in visit_ClassDef
> >        self.generic_visit(node)
> >      File "/usr/lib64/python3.7/ast.py", line 279, in generic_visit
> >        self.visit(item)
> >      File "/usr/lib64/python3.7/ast.py", line 271, in visit
> >        return visitor(node)
> >      File "/var/tmp/qemu-builddir/tests/venv/lib64/python3.7/site-packages/avocado/plugins/assets.py", line 171, in visit_Assign
> >        self.asgmts[cur_klass][cur_method][name] = node.value.s
> >    KeyError: 'launch_and_wait'
> >    make: *** [tests/Makefile.include:949: fetch-acceptance] Error 1
>
> Currently the acceptance tests use Avocado 7.6. I bumped to 80.0 (latest
> released) here and that error is gone. Could you double check?
>
> Regards,
>
> Wainer

Hi Wainer, thanks for looking at this problem.

This bug was fixed here
https://github.com/avocado-framework/avocado/pull/3665, on release 78
of Avocado. It was reported by Philippe at that time. I think we
forgot to bump the Avocado version here.

>
> >
> > Exclude it for now. We will revert this commit once the script is
> > fixed.
> >
> > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > ---
> >   tests/Makefile.include | 2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > index 238974d8da..7c9cf7a818 100644
> > --- a/tests/Makefile.include
> > +++ b/tests/Makefile.include
> > @@ -950,7 +950,7 @@ fetch-acceptance: check-venv
> >               $(TESTS_VENV_DIR)/bin/python -m avocado \
> >               $(if $(V),--show=$(AVOCADO_SHOW)) \
> >               assets fetch \
> > -            $(wildcard tests/acceptance/*.py), \
> > +            $(filter-out tests/acceptance/boot_linux.py,$(wildcard tests/acceptance/*.py)), \
> >               "AVOCADO", "tests/acceptance")
> >
> >   check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>

Re: [PATCH 1/2] tests: Add 'fetch-acceptance' rule

[Wainer dos Santos Moschetta]

Hello,

On 7/24/20 4:35 AM, Philippe Mathieu-Daudé wrote:
> Add a rule to fetch acceptance test assets.
>
> This is particularly useful in a CI context, when a single job
> can fetch and save the cache so other jobs reuse it directly.
>
> It is also useful to measure the time spent downloading the
> assets versus the time spent running the tests.
>
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>   tests/Makefile.include | 9 +++++++++
>   1 file changed, 9 insertions(+)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index c7e4646ded..238974d8da 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -944,6 +944,15 @@ get-vm-image-fedora-31-%: check-venv
>   # download all vm images, according to defined targets
>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>   
> +# fetch acceptance test assets
> +fetch-acceptance: check-venv

This new target misses an entry on check-help.

> +	$(call quiet-command, \
> +            $(TESTS_VENV_DIR)/bin/python -m avocado \
> +            $(if $(V),--show=$(AVOCADO_SHOW)) \
> +            assets fetch \

Perhaps pass '--ignore-errors' so that intermittent network failurse 
won't disturb the execution (the test will have a second chance to 
download the asset later when it executes).

> +            $(wildcard tests/acceptance/*.py), \
> +            "AVOCADO", "tests/acceptance")

nit: print "Downloading acceptance tests assets" (similar to 
get-vm-image-fedora-32-% target).

Talking about get-vm-images...that target is pre-req of 
check-acceptance, which makes me think that fetch-acceptance should be 
either (for the sake of consistency.) The downside is that - as a 
developer running it on my machine - `avocado assets fetch` will attempt 
to download artifacts even for those tests which I'm not going to run 
anyway. Any opinion?

Regards,

Wainer

> +
>   check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>   	$(call quiet-command, \
>               $(TESTS_VENV_DIR)/bin/python -m avocado \