Did Avocado assets hashing system changed?

Did Avocado assets hashing system changed?

[Philippe Mathieu-Daudé]

Hi,

We are trying to figure out why old assets saved in our Avocado
cache (which we populate in our various CI jobs) isn't used
anymore.

Trying to reproduce locally, I see for example:

2023-09-13 14:39:18,834 test             L0520 INFO | START 
14-tests/avocado/machine_mips_malta.py:MaltaMachine.test_mipsel_malta_yamon
2023-09-13 14:39:18,834 test             L0205 DEBUG| DATA 
(filename=output.expected) => NOT FOUND (data sources: variant, test, file)
2023-09-13 14:39:18,861 asset            L0339 INFO | Asset not in 
cache, fetching it.
2023-09-13 14:39:18,862 download         L0067 INFO | Fetching 
http://www.imgtec.com/tools/mips-tools/downloads/yamon/yamon-bin-02.22.zip 
-> 
/home/philippe.mathieu-daude/avocado/data/cache/by_location/70c32932b7794b6b37a5040a8808ac943699b944/yamon-bin-02.22.zip.csi3gk2v
2023-09-13 14:39:18,940 asset            L0144 INFO | Temporary asset 
file unavailable due to failed download attempt.
2023-09-13 14:39:18,940 asset            L0378 ERROR| HTTPError: HTTP 
Error 403: Forbidden

I remember this test used to pass last year, so checking the file I
can see:

$ find ~/avocado/data/ -name yamon-bin-02.22.zip\* -ls
   1489611   6629 -rw-r--r--   1 philmd primary   6776580 Dec 30  2022 
/home/philmd/avocado/data/cache/by_location/c1961386ba52f06c6005b48cd32195a371152769/yamon-bin-02.22.zip
   1489613      1 -rw-r--r--   1 philmd primary        46 Dec 30  2022 
/home/philmd/avocado/data/cache/by_location/c1961386ba52f06c6005b48cd32195a371152769/yamon-bin-02.22.zip-CHECKSUM

But the hash is different... Looking at QEMU git history we haven't
changed the URL in tests/avocado/machine_mips_malta.py, so I wonder,
did Avocado changed its asset hashing during the last year?

Thanks,

Phil.

Re: Did Avocado assets hashing system changed?

[Daniel P. Berrangé]

On Wed, Sep 13, 2023 at 04:17:24PM +0200, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> We are trying to figure out why old assets saved in our Avocado
> cache (which we populate in our various CI jobs) isn't used
> anymore.
> 
> Trying to reproduce locally, I see for example:
> 
> 2023-09-13 14:39:18,834 test             L0520 INFO | START
> 14-tests/avocado/machine_mips_malta.py:MaltaMachine.test_mipsel_malta_yamon
> 2023-09-13 14:39:18,834 test             L0205 DEBUG| DATA
> (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
> 2023-09-13 14:39:18,861 asset            L0339 INFO | Asset not in cache,
> fetching it.
> 2023-09-13 14:39:18,862 download         L0067 INFO | Fetching
> http://www.imgtec.com/tools/mips-tools/downloads/yamon/yamon-bin-02.22.zip
> -> /home/philippe.mathieu-daude/avocado/data/cache/by_location/70c32932b7794b6b37a5040a8808ac943699b944/yamon-bin-02.22.zip.csi3gk2v
> 2023-09-13 14:39:18,940 asset            L0144 INFO | Temporary asset file
> unavailable due to failed download attempt.
> 2023-09-13 14:39:18,940 asset            L0378 ERROR| HTTPError: HTTP Error
> 403: Forbidden
> 
> I remember this test used to pass last year, so checking the file I
> can see:
> 
> $ find ~/avocado/data/ -name yamon-bin-02.22.zip\* -ls
>   1489611   6629 -rw-r--r--   1 philmd primary   6776580 Dec 30  2022 /home/philmd/avocado/data/cache/by_location/c1961386ba52f06c6005b48cd32195a371152769/yamon-bin-02.22.zip
>   1489613      1 -rw-r--r--   1 philmd primary        46 Dec 30  2022 /home/philmd/avocado/data/cache/by_location/c1961386ba52f06c6005b48cd32195a371152769/yamon-bin-02.22.zip-CHECKSUM
> 
> But the hash is different... Looking at QEMU git history we haven't
> changed the URL in tests/avocado/machine_mips_malta.py, so I wonder,
> did Avocado changed its asset hashing during the last year?

How can it have changed when we've been pinned to avocado 88.1 for
ages now, not consuming new versions ?

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Did Avocado assets hashing system changed?

[Thomas Huth]

On 13/09/2023 17.20, Daniel P. Berrangé wrote:
> On Wed, Sep 13, 2023 at 04:17:24PM +0200, Philippe Mathieu-Daudé wrote:
>> Hi,
>>
>> We are trying to figure out why old assets saved in our Avocado
>> cache (which we populate in our various CI jobs) isn't used
>> anymore.
>>
>> Trying to reproduce locally, I see for example:
>>
>> 2023-09-13 14:39:18,834 test             L0520 INFO | START
>> 14-tests/avocado/machine_mips_malta.py:MaltaMachine.test_mipsel_malta_yamon
>> 2023-09-13 14:39:18,834 test             L0205 DEBUG| DATA
>> (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
>> 2023-09-13 14:39:18,861 asset            L0339 INFO | Asset not in cache,
>> fetching it.
>> 2023-09-13 14:39:18,862 download         L0067 INFO | Fetching
>> http://www.imgtec.com/tools/mips-tools/downloads/yamon/yamon-bin-02.22.zip
>> -> /home/philippe.mathieu-daude/avocado/data/cache/by_location/70c32932b7794b6b37a5040a8808ac943699b944/yamon-bin-02.22.zip.csi3gk2v
>> 2023-09-13 14:39:18,940 asset            L0144 INFO | Temporary asset file
>> unavailable due to failed download attempt.
>> 2023-09-13 14:39:18,940 asset            L0378 ERROR| HTTPError: HTTP Error
>> 403: Forbidden
>>
>> I remember this test used to pass last year, so checking the file I
>> can see:
>>
>> $ find ~/avocado/data/ -name yamon-bin-02.22.zip\* -ls
>>    1489611   6629 -rw-r--r--   1 philmd primary   6776580 Dec 30  2022 /home/philmd/avocado/data/cache/by_location/c1961386ba52f06c6005b48cd32195a371152769/yamon-bin-02.22.zip
>>    1489613      1 -rw-r--r--   1 philmd primary        46 Dec 30  2022 /home/philmd/avocado/data/cache/by_location/c1961386ba52f06c6005b48cd32195a371152769/yamon-bin-02.22.zip-CHECKSUM
>>
>> But the hash is different... Looking at QEMU git history we haven't
>> changed the URL in tests/avocado/machine_mips_malta.py, so I wonder,
>> did Avocado changed its asset hashing during the last year?
> 
> How can it have changed when we've been pinned to avocado 88.1 for
> ages now, not consuming new versions ?

We've temporarily switched to version 101 some months ago, see commit 
ec5ffa0056389c3c.

  Thomas

Re: Did Avocado assets hashing system changed?

[Daniel P. Berrangé]

On Wed, Sep 13, 2023 at 05:40:25PM +0200, Thomas Huth wrote:
> On 13/09/2023 17.20, Daniel P. Berrangé wrote:
> > On Wed, Sep 13, 2023 at 04:17:24PM +0200, Philippe Mathieu-Daudé wrote:
> > > Hi,
> > > 
> > > We are trying to figure out why old assets saved in our Avocado
> > > cache (which we populate in our various CI jobs) isn't used
> > > anymore.
> > > 
> > > Trying to reproduce locally, I see for example:
> > > 
> > > 2023-09-13 14:39:18,834 test             L0520 INFO | START
> > > 14-tests/avocado/machine_mips_malta.py:MaltaMachine.test_mipsel_malta_yamon
> > > 2023-09-13 14:39:18,834 test             L0205 DEBUG| DATA
> > > (filename=output.expected) => NOT FOUND (data sources: variant, test, file)
> > > 2023-09-13 14:39:18,861 asset            L0339 INFO | Asset not in cache,
> > > fetching it.
> > > 2023-09-13 14:39:18,862 download         L0067 INFO | Fetching
> > > http://www.imgtec.com/tools/mips-tools/downloads/yamon/yamon-bin-02.22.zip
> > > -> /home/philippe.mathieu-daude/avocado/data/cache/by_location/70c32932b7794b6b37a5040a8808ac943699b944/yamon-bin-02.22.zip.csi3gk2v
> > > 2023-09-13 14:39:18,940 asset            L0144 INFO | Temporary asset file
> > > unavailable due to failed download attempt.
> > > 2023-09-13 14:39:18,940 asset            L0378 ERROR| HTTPError: HTTP Error
> > > 403: Forbidden
> > > 
> > > I remember this test used to pass last year, so checking the file I
> > > can see:
> > > 
> > > $ find ~/avocado/data/ -name yamon-bin-02.22.zip\* -ls
> > >    1489611   6629 -rw-r--r--   1 philmd primary   6776580 Dec 30  2022 /home/philmd/avocado/data/cache/by_location/c1961386ba52f06c6005b48cd32195a371152769/yamon-bin-02.22.zip
> > >    1489613      1 -rw-r--r--   1 philmd primary        46 Dec 30  2022 /home/philmd/avocado/data/cache/by_location/c1961386ba52f06c6005b48cd32195a371152769/yamon-bin-02.22.zip-CHECKSUM
> > > 
> > > But the hash is different... Looking at QEMU git history we haven't
> > > changed the URL in tests/avocado/machine_mips_malta.py, so I wonder,
> > > did Avocado changed its asset hashing during the last year?
> > 
> > How can it have changed when we've been pinned to avocado 88.1 for
> > ages now, not consuming new versions ?
> 
> We've temporarily switched to version 101 some months ago, see commit
> ec5ffa0056389c3c.

Which was quickly reverted in eaf245becd9ece615a4831c0cf4d27b165d8675f,
but I guess perhaps the cache has some files from that window of the
upgrade.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|