Re: [PATCH-for-5.0 1/7] tests/acceptance/machine_sparc_leon3: Disable HelenOS test

["Philippe Mathieu-Daudé" <philmd@redhat.com>]

On 4/1/20 7:43 PM, Willian Rampazzo wrote:
> On Tue, Mar 31, 2020 at 5:07 PM Philippe Mathieu-Daudé
> <philmd@redhat.com> wrote:
> <snip>
>>
>> First job failed by timeout, 2nd succeeded:
>> https://travis-ci.org/github/philmd/qemu/jobs/669265466
>>
>> However "Ran for 46 min 48 sec"
>>
>>   From the log:
>>
>> Fetching asset from
>> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips64el_malta_5KEc_cpio
>> Fetching asset from
>> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips64el_malta_5KEc_cpio
>> Fetching asset from
>> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi
>> Fetching asset from
>> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_initrd
>> Fetching asset from
>> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_initrd
>> Fetching asset from
>> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_sd
>> Fetching asset from
>> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_sd
>> Fetching asset from
>> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_bionic
>> Fetching asset from
>> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9
>> Fetching asset from
>> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9
>> Fetching asset from
>> tests/acceptance/ppc_prep_40p.py:IbmPrep40pMachine.test_openbios_and_netbsd
>> ...
>>    (13/82)
>> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_mips64el_malta_5KEc_cpio:
>>    SKIP: untrusted code
>>    (24/82)
>> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_bionic:
>>    SKIP: storage limited
>> ...
>>    (25/82)
>> tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_orangepi_uboot_netbsd9:
>>    SKIP: storage limited
>> ...
>>    (63/82)
>> tests/acceptance/ppc_prep_40p.py:IbmPrep40pMachine.test_openbios_and_netbsd:
>>    SKIP: Running on Travis-CI
>>
>> Is it possible that we are now fetching assets for tests we are not
>> running? In particular the one marked @skip because the downloading time
>> was too long on Travis?
> 
> Yes, your assumption is correct, this execution of Avocado downloaded
> assets for tests that were skipped. Let me try to explain how the
> asset feature works today on Avocado.
> 
> Avocado has two basic ways to work with assets:
> 
> 1. Parse limited use cases of `fetch_asset` call in the test file and
> execute them. This operation can happen in two different scenarios.
> First, when using the command line `avocado assets fetch <test_file>`.

Odd, with avocado-framework==76.0 I get:

https://travis-ci.org/github/philmd/qemu/jobs/669851870#L4908

Traceback (most recent call last):
   File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
     "__main__", mod_spec)
   File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
     exec(code, run_globals)
   File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6zsite-packages/avocado/__main__.py", 
line 11, in <module>
     sys.exit(main.run())
   File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/core/app.py", 
line 91, in run
     return method(self.parser.config)
   File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/plugins/assets.py", 
line 291, in run
     success, fail = fetch_assets(test_file)
   File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/plugins/assets.py", 
line 200, in fetch_assets
     handler = FetchAssetHandler(test_file, klass, method)
   File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/plugins/assets.py", 
line 65, in __init__
     self.visit(self.tree)
   File "/usr/lib/python3.6/ast.py", line 253, in visit
     return visitor(node)
   File "/usr/lib/python3.6/ast.py", line 261, in generic_visit
     self.visit(item)
   File "/usr/lib/python3.6/ast.py", line 253, in visit
     return visitor(node)
   File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/plugins/assets.py", 
line 139, in visit_ClassDef
     self.generic_visit(node)
   File "/usr/lib/python3.6/ast.py", line 261, in generic_visit
     self.visit(item)
   File "/usr/lib/python3.6/ast.py", line 253, in visit
     return visitor(node)
   File 
"/home/travis/build/philmd/qemu/build/tests/venv/lib/python3.6/site-packages/avocado/plugins/assets.py", 
line 171, in visit_Assign
     self.asgmts[cur_klass][cur_method][name] = node.value.s
KeyError: 'launch_and_wait'
/home/travis/build/philmd/qemu/tests/Makefile.include:910: recipe for 
target 'fetch-acceptance-assets' failed

This launch_and_wait comes from:

tests/acceptance/boot_linux.py:88:    def launch_and_wait(self):

> In this case, it is a standalone execution of each fetch call and the
> test is not executed at all. Second, by running the test. The, enabled
> by default, plugin FetchAssetJob will do the same operation of parsing
> the test file and executing occurrences of `fetch_asset` call before
> the tests start to run. Again, the fetch time is not computed in the
> job time.
> 
> 2. Execute the `fetch_asset` call from each test during the test
> execution. In this case, the FetchAssetJob plugin should be disabled.
> The fetch time is added to the job time as the asset download occurs
> during the test execution.
> 
> The acceptance tests which make use of `fetch_asset` are all using the
> first method with FetchAssetJob plugin enabled. As Avocado is parsing
> the test file before it starts to run the tests, it is not aware of
> possible skips that may occur during a test execution due to possible
> dynamic dependency.
> 
> This is not the desired behavior, as you mentioned, Avocado is
> downloading an asset that will not be used because its test will be
> skipped. To minimize the damage on the download side, the Travis job
> is holding the avocado cache. It means the download should happen just
> once. This does not minimize the damage to space usage.
> 
> One possible workaround here is to temporarily disable the
> FetchAssetJob plugin, now that the needed assets are on Travis Avocado
> cache. The downside is that when an asset is not available in the
> cache, it will be downloaded during the test execution and the
> download time will be added to the job time. I don't know if it is
> possible to manually remove an asset from Travis Avocado cache. If so,
> this can be done for the tests that should be skipped because of space
> usage.
> 
> We have been trying to make the Asset feature as flexible as possible
> to accommodate the use cases we have been identifying. Thanks for
> reporting this!
> 
>>
>> RESULTS    : PASS 65 | ERROR 0 | FAIL 0 | SKIP 14 | WARN 0 | INTERRUPT 0
>> | CANCEL 3
>> JOB TIME   : 1480.72 s
>>
>> Does this "JOB TIME" sums the 'Fetching asset' part?
> 
> Answered in the comments above.

Thanks for the big clear explanation :)

Phil.