Re: [PULL 15/42] tests/functional: enable pre-emptive caching of assets

[Thomas Huth <thuth@redhat.com>]

On 26/11/2024 18.46, Peter Maydell wrote:
> On Tue, 26 Nov 2024 at 17:31, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>
>> On Tue, Nov 26, 2024 at 05:44:29PM +0100, Philippe Mathieu-Daudé wrote:
>>> Hi,
>>>
>>> On 4/9/24 12:38, Thomas Huth wrote:
>>> fetch() can fail [*] (see previous patch, various Exceptions returned).
>>>
>>> What should we do in this case? If we ignore a missing artifact,
>>> the tests will eventually fail. Better bail out early and save
>>> credit minutes?
>>
>> We already do what you describe - 'fetch' will raise an exception
>> which causes the precache task to fail, and the CI job gets marked
>> as failed. We don't attempt to run tests if assets are missing.
>>
>>
>>>> @@ -58,6 +59,12 @@ def tearDown(self):
>>>>        def main():
>>>>            path = os.path.basename(sys.argv[0])[:-3]
>>>> +
>>>> +        cache = os.environ.get("QEMU_TEST_PRECACHE", None)
>>>> +        if cache is not None:
>>>> +            Asset.precache_suites(path, cache)
>>>> +            return
>>>> +
>>>>            tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
>>>>                                       test_output_log = pycotap.LogMode.LogToError)
>>>>            unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])
>>>
>>> [*] Peter reported the following CI failure:
>>>
>>>    https://gitlab.com/qemu-project/qemu/-/jobs/8474928266
>>>
>>> 2024-11-26 14:58:53,170 - qemu-test - ERROR - Unable to download https://apt.armbian.com/pool/main/l/linux-6.6.16/linux-image-current-sunxi_24.2.1_armhf__6.6.16-Seb3e-D6b4a-P2359-Ce96bHfe66-HK01ba-V014b-B067e-R448a.deb:
>>
>> This looks to be working as intended. We failed to cache
>> the asset, and so we stopped the job, without trying to
>> run the tests.
> 
> The job ended up in state "failed", with a red X mark in
> the gitlab UI. If we intend that not being able to fetch
> the assets doesn't count as a test failure, that didn't
> work here. If we do intend that fetch failures should be
> CI failures, we need to make our process of fetching and
> caching the images more robust, because otherwise the result
> is flaky CI jobs.

I think we want to continue to maek failing downloads as test failures, 
otherwise we'll never notice when an asset is not available from the 
internet anymore (since SKIPs just get ignored).

What we really need is a working cache for the private CI runners to ease 
the pain when the host just has a networking hiccup.

  Thomas