Re: [RFC PATCH 5/8] tests_pytest: Implement fetch_asset() method for downloading assets

[Thomas Huth <thuth@redhat.com>]

On 11/07/2024 23.35, Richard Henderson wrote:
> On 7/11/24 12:23, Alex Bennée wrote:
>> Richard Henderson <richard.henderson@linaro.org> writes:
>>
>>> On 7/11/24 09:45, Richard Henderson wrote:
>>>> On 7/11/24 04:55, Thomas Huth wrote:
>>>>> +    def fetch_asset(self, url, asset_hash):
>>>>> +        cache_dir = os.path.expanduser("~/.cache/qemu/download")
>>>>> +        if not os.path.exists(cache_dir):
>>>>> +            os.makedirs(cache_dir)
>>>>> +        fname = os.path.join(cache_dir,
>>>>> +                             
>>>>> hashlib.sha1(url.encode("utf-8")).hexdigest())
>>>>> +        if os.path.exists(fname) and self.check_hash(fname, asset_hash):
>>>>> +            return fname
>>>>> +        logging.debug("Downloading %s to %s...", url, fname)
>>>>> +        subprocess.check_call(["wget", "-c", url, "-O", fname + 
>>>>> ".download"])
>>>>> +        os.rename(fname + ".download", fname)
>>>>> +        return fname
>>>> Download failure via exception?
>>>> Check hash on downloaded asset?
>>>
>>> I would prefer to see assets, particularly downloading, handled in a
>>> separate pass from tests.
>>
>> And I assume cachable?
> 
> The cache is already handled here.  But downloading after cache miss is 
> non-optional, may not fail, and is accounted against the meson test timeout.

Unless someone has a really good idea how to implement that download before 
running the tests, I think we can start by simply giving enough headroom in 
the initial timeout settings. We can then hopefully easily fine-tune later - 
since this time the framework is under our control, so we don't have to sync 
in a cumbersome way with the avocado test runner any more.

  Thomas