Problem running functional tests from China

Problem running functional tests from China

[Philippe Mathieu-Daudé]

Just to inform the files hosted on github don't work when
testing from China:

 (02/23)
tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_emcraft_sf2:
CANCEL: Missing asset
https://raw.githubusercontent.com/Subbaraya-Sundeep/qemu-test-binaries/fe371d32e50ca682391e1e70ab98c2942aeffb01/u-boot
(0.02 s)
 (03/23)
tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_raspi2_uart0:
INTERRUPTED: Missing asset
http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/raspberrypi-kernel_1.20190215-1_armhf.deb\nRunner
error occurred: Timeout reached\nOriginal status: CANCEL\n{'name':
'03-tests/acceptance/boot_linux_console.py:BootLinuxCons... (90.61 s)
 (04/23)
tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_exynos4210_initrd:
CANCEL: Missing asset
https://github.com/groeck/linux-build-test/raw/2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/arm/rootfs-armv5.cpio.gz
(35.63 s)
 (05/23)
tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_cubieboard_initrd:
CANCEL: Missing asset
https://github.com/groeck/linux-build-test/raw/2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/arm/rootfs-armv5.cpio.gz
(8.75 s)
 (06/23)
tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_arm_cubieboard_sata:
CANCEL: Missing asset
https://github.com/groeck/linux-build-test/raw/2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/arm/rootfs-armv5.ext2.gz
(8.63 s)
 (17/23)
tests/acceptance/machine_arm_integratorcp.py:IntegratorMachine.test_integratorcp_console:
CANCEL: Missing asset
https://github.com/zayac/qemu-arm/raw/master/arm-test/kernel/zImage.integrator
(0.76 s)
 (22/23)
tests/acceptance/replay_kernel.py:ReplayKernelNormal.test_arm_cubieboard_initrd:
CANCEL: Missing asset
https://github.com/groeck/linux-build-test/raw/2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/arm/rootfs-armv5.cpio.gz
(8.11 s)

raw.githubusercontent.com resolves to 127.0.0.1

Regards,

Phil.

Re: Problem running functional tests from China

[Michal Suchánek]

On Wed, Feb 24, 2021 at 10:29:15AM +0100, Philippe Mathieu-Daudé wrote:
> Just to inform the files hosted on github don't work when
> testing from China:

> 
> raw.githubusercontent.com resolves to 127.0.0.1

And from where does this come from?

Your local system, your ISP, ... ?

Thanks

Michal

Re: Problem running functional tests from China

[Philippe Mathieu-Daudé]

On 2/24/21 12:06 PM, Michal Suchánek wrote:
> On Wed, Feb 24, 2021 at 10:29:15AM +0100, Philippe Mathieu-Daudé wrote:
>> Just to inform the files hosted on github don't work when
>> testing from China:
> 
>>
>> raw.githubusercontent.com resolves to 127.0.0.1

Actually not even localhost but 0.0.0.0 :)

> And from where does this come from?
> 
> Your local system, your ISP, ... ?

Probably the ISP:

$ host -v raw.githubusercontent.com
Trying "raw.githubusercontent.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21011
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;raw.githubusercontent.com.     IN      A

;; ANSWER SECTION:
raw.githubusercontent.com. 299  IN      A       0.0.0.0

Received 59 bytes from 210.22.70.3#53 in 3 ms
Trying "raw.githubusercontent.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60262
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;raw.githubusercontent.com.     IN      AAAA

;; ANSWER SECTION:
raw.githubusercontent.com. 297  IN      AAAA    ::

Received 71 bytes from 210.22.70.3#53 in 3 ms
Trying "raw.githubusercontent.com"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30016
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;raw.githubusercontent.com.     IN      MX

;; AUTHORITY SECTION:
badlist.                350     IN      SOA     localhost.
root.locahost. 1 3600 900 2592000 7200

Received 108 bytes from 210.22.70.3#53 in 3 ms

I raised that problem not to find a way to bypass an ISP
firewall, but to see if there is a way to use another
storage for test artifacts so all the community can run
the tests.

Regards,

Phil.

Re: Problem running functional tests from China

[Michal Suchánek]

On Wed, Feb 24, 2021 at 12:37:53PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/24/21 12:06 PM, Michal Suchánek wrote:
> > On Wed, Feb 24, 2021 at 10:29:15AM +0100, Philippe Mathieu-Daudé wrote:
> >> Just to inform the files hosted on github don't work when
> >> testing from China:
> > 
> >>
> >> raw.githubusercontent.com resolves to 127.0.0.1
> 
> Actually not even localhost but 0.0.0.0 :)
> 
> > And from where does this come from?
> > 
> > Your local system, your ISP, ... ?
> 
> Probably the ISP:
> 
...
> 
> I raised that problem not to find a way to bypass an ISP
> firewall, but to see if there is a way to use another
> storage for test artifacts so all the community can run
> the tests.

I don't think protecting from random network malfunction is something
the qemu project can do.

That said, downloading the test data during test run does indeed look
fragile.

We have the concept of git submodules which is used in qemu extensively
so the test data could possibly be included directly in the git tree.

The obvious downside is that the current and past test binaries will take
up disk space for all users that check out the repository, even those
that don't run the tests.

Thanks

Michal

Re: Problem running functional tests from China

[Daniel P. Berrangé]

On Wed, Feb 24, 2021 at 01:01:40PM +0100, Michal Suchánek wrote:
> On Wed, Feb 24, 2021 at 12:37:53PM +0100, Philippe Mathieu-Daudé wrote:
> > On 2/24/21 12:06 PM, Michal Suchánek wrote:
> > > On Wed, Feb 24, 2021 at 10:29:15AM +0100, Philippe Mathieu-Daudé wrote:
> > >> Just to inform the files hosted on github don't work when
> > >> testing from China:
> > > 
> > >>
> > >> raw.githubusercontent.com resolves to 127.0.0.1
> > 
> > Actually not even localhost but 0.0.0.0 :)
> > 
> > > And from where does this come from?
> > > 
> > > Your local system, your ISP, ... ?
> > 
> > Probably the ISP:
> > 
> ...
> > 
> > I raised that problem not to find a way to bypass an ISP
> > firewall, but to see if there is a way to use another
> > storage for test artifacts so all the community can run
> > the tests.
> 
> I don't think protecting from random network malfunction is something
> the qemu project can do.
> 
> That said, downloading the test data during test run does indeed look
> fragile.
> 
> We have the concept of git submodules which is used in qemu extensively
> so the test data could possibly be included directly in the git tree.
> 
> The obvious downside is that the current and past test binaries will take
> up disk space for all users that check out the repository, even those
> that don't run the tests.

We explicitly do not want to be distributing / including the
images ourselves, as that makes the QEMU project responsible for
license compliance and thus provision of full & corresponding source.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Problem running functional tests from China

[Philippe Mathieu-Daudé]

On 2/24/21 1:08 PM, Daniel P. Berrangé wrote:
> On Wed, Feb 24, 2021 at 01:01:40PM +0100, Michal Suchánek wrote:
>> On Wed, Feb 24, 2021 at 12:37:53PM +0100, Philippe Mathieu-Daudé wrote:
>>> On 2/24/21 12:06 PM, Michal Suchánek wrote:
>>>> On Wed, Feb 24, 2021 at 10:29:15AM +0100, Philippe Mathieu-Daudé wrote:
>>>>> Just to inform the files hosted on github don't work when
>>>>> testing from China:
>>>>
>>>>>
>>>>> raw.githubusercontent.com resolves to 127.0.0.1
>>>
>>> Actually not even localhost but 0.0.0.0 :)
>>>
>>>> And from where does this come from?
>>>>
>>>> Your local system, your ISP, ... ?
>>>
>>> Probably the ISP:
>>>
>> ...
>>>
>>> I raised that problem not to find a way to bypass an ISP
>>> firewall, but to see if there is a way to use another
>>> storage for test artifacts so all the community can run
>>> the tests.
>>
>> I don't think protecting from random network malfunction is something
>> the qemu project can do.
>>
>> That said, downloading the test data during test run does indeed look
>> fragile.
>>
>> We have the concept of git submodules which is used in qemu extensively
>> so the test data could possibly be included directly in the git tree.
>>
>> The obvious downside is that the current and past test binaries will take
>> up disk space for all users that check out the repository, even those
>> that don't run the tests.
> 
> We explicitly do not want to be distributing / including the
> images ourselves, as that makes the QEMU project responsible for
> license compliance and thus provision of full & corresponding source.

I am not asking that QEMU redistribute these files, I'm asking to the
community if someone know an alternative to store files so we can run
QEMU tests in Chinese cloud farms.

Back to your comment, should we remove these tests?

Thanks,

Phil.

Re: Problem running functional tests from China

[Daniel P. Berrangé]

On Wed, Feb 24, 2021 at 01:48:25PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/24/21 1:08 PM, Daniel P. Berrangé wrote:
> > On Wed, Feb 24, 2021 at 01:01:40PM +0100, Michal Suchánek wrote:
> >> On Wed, Feb 24, 2021 at 12:37:53PM +0100, Philippe Mathieu-Daudé wrote:
> >>> On 2/24/21 12:06 PM, Michal Suchánek wrote:
> >>>> On Wed, Feb 24, 2021 at 10:29:15AM +0100, Philippe Mathieu-Daudé wrote:
> >>>>> Just to inform the files hosted on github don't work when
> >>>>> testing from China:
> >>>>
> >>>>>
> >>>>> raw.githubusercontent.com resolves to 127.0.0.1
> >>>
> >>> Actually not even localhost but 0.0.0.0 :)
> >>>
> >>>> And from where does this come from?
> >>>>
> >>>> Your local system, your ISP, ... ?
> >>>
> >>> Probably the ISP:
> >>>
> >> ...
> >>>
> >>> I raised that problem not to find a way to bypass an ISP
> >>> firewall, but to see if there is a way to use another
> >>> storage for test artifacts so all the community can run
> >>> the tests.
> >>
> >> I don't think protecting from random network malfunction is something
> >> the qemu project can do.
> >>
> >> That said, downloading the test data during test run does indeed look
> >> fragile.
> >>
> >> We have the concept of git submodules which is used in qemu extensively
> >> so the test data could possibly be included directly in the git tree.
> >>
> >> The obvious downside is that the current and past test binaries will take
> >> up disk space for all users that check out the repository, even those
> >> that don't run the tests.
> > 
> > We explicitly do not want to be distributing / including the
> > images ourselves, as that makes the QEMU project responsible for
> > license compliance and thus provision of full & corresponding source.
> 
> I am not asking that QEMU redistribute these files, I'm asking to the
> community if someone know an alternative to store files so we can run
> QEMU tests in Chinese cloud farms.
> 
> Back to your comment, should we remove these tests?

No, I don't think that we should do that.

If someone has broken ISP and they need to run all tests, then they'll
need to use a VPN.

Regardless of that though, we should make sure that tests automatically
mark themselves as "skipped" if any downloading of assets fails. ie a
download failure shouldn't cause tests to report failure.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Problem running functional tests from China

[Michal Suchánek]

On Wed, Feb 24, 2021 at 12:57:13PM +0000, Daniel P. Berrangé wrote:
> On Wed, Feb 24, 2021 at 01:48:25PM +0100, Philippe Mathieu-Daudé wrote:
> > On 2/24/21 1:08 PM, Daniel P. Berrangé wrote:
> > > On Wed, Feb 24, 2021 at 01:01:40PM +0100, Michal Suchánek wrote:
> > >> On Wed, Feb 24, 2021 at 12:37:53PM +0100, Philippe Mathieu-Daudé wrote:
> > >>> On 2/24/21 12:06 PM, Michal Suchánek wrote:
> > >>>> On Wed, Feb 24, 2021 at 10:29:15AM +0100, Philippe Mathieu-Daudé wrote:
> > >>>>> Just to inform the files hosted on github don't work when
> > >>>>> testing from China:
> > >>>>
> > >>>>>
> > >>>>> raw.githubusercontent.com resolves to 127.0.0.1
> > >>>
> > >>> Actually not even localhost but 0.0.0.0 :)
> > >>>
> > >>>> And from where does this come from?
> > >>>>
> > >>>> Your local system, your ISP, ... ?
> > >>>
> > >>> Probably the ISP:
> > >>>
> > >> ...
> > >>>
> > >>> I raised that problem not to find a way to bypass an ISP
> > >>> firewall, but to see if there is a way to use another
> > >>> storage for test artifacts so all the community can run
> > >>> the tests.
> > >>
> > >> I don't think protecting from random network malfunction is something
> > >> the qemu project can do.
> > >>
> > >> That said, downloading the test data during test run does indeed look
> > >> fragile.
> > >>
> > >> We have the concept of git submodules which is used in qemu extensively
> > >> so the test data could possibly be included directly in the git tree.
> > >>
> > >> The obvious downside is that the current and past test binaries will take
> > >> up disk space for all users that check out the repository, even those
> > >> that don't run the tests.
> > > 
> > > We explicitly do not want to be distributing / including the
> > > images ourselves, as that makes the QEMU project responsible for
> > > license compliance and thus provision of full & corresponding source.
> > 
> > I am not asking that QEMU redistribute these files, I'm asking to the
> > community if someone know an alternative to store files so we can run
> > QEMU tests in Chinese cloud farms.
> > 
> > Back to your comment, should we remove these tests?
> 
> No, I don't think that we should do that.
> 
> If someone has broken ISP and they need to run all tests, then they'll
> need to use a VPN.

I would say running tests offline is generally desirable feature - runs
in secure environments, networking errors not affecting test results,
etc.

Also pulling test data from a number of random repositories does not
sound great for test reproducibility. Is there some plan for addressing
these random artifacts going away some day other than ignoring the tests
for which test data is missing?

Thanks

Michal

Re: Problem running functional tests from China

[Wainer dos Santos Moschetta]

Hi,

On 2/24/21 9:57 AM, Daniel P. Berrangé wrote:
> On Wed, Feb 24, 2021 at 01:48:25PM +0100, Philippe Mathieu-Daudé wrote:
>> On 2/24/21 1:08 PM, Daniel P. Berrangé wrote:
>>> On Wed, Feb 24, 2021 at 01:01:40PM +0100, Michal Suchánek wrote:
>>>> On Wed, Feb 24, 2021 at 12:37:53PM +0100, Philippe Mathieu-Daudé wrote:
>>>>> On 2/24/21 12:06 PM, Michal Suchánek wrote:
>>>>>> On Wed, Feb 24, 2021 at 10:29:15AM +0100, Philippe Mathieu-Daudé wrote:
>>>>>>> Just to inform the files hosted on github don't work when
>>>>>>> testing from China:
>>>>>>> raw.githubusercontent.com resolves to 127.0.0.1
>>>>> Actually not even localhost but 0.0.0.0 :)
>>>>>
>>>>>> And from where does this come from?
>>>>>>
>>>>>> Your local system, your ISP, ... ?
>>>>> Probably the ISP:
>>>>>
>>>> ...
>>>>> I raised that problem not to find a way to bypass an ISP
>>>>> firewall, but to see if there is a way to use another
>>>>> storage for test artifacts so all the community can run
>>>>> the tests.
>>>> I don't think protecting from random network malfunction is something
>>>> the qemu project can do.
>>>>
>>>> That said, downloading the test data during test run does indeed look
>>>> fragile.
>>>>
>>>> We have the concept of git submodules which is used in qemu extensively
>>>> so the test data could possibly be included directly in the git tree.
>>>>
>>>> The obvious downside is that the current and past test binaries will take
>>>> up disk space for all users that check out the repository, even those
>>>> that don't run the tests.
>>> We explicitly do not want to be distributing / including the
>>> images ourselves, as that makes the QEMU project responsible for
>>> license compliance and thus provision of full & corresponding source.
>> I am not asking that QEMU redistribute these files, I'm asking to the
>> community if someone know an alternative to store files so we can run
>> QEMU tests in Chinese cloud farms.
>>
>> Back to your comment, should we remove these tests?
> No, I don't think that we should do that.
>
> If someone has broken ISP and they need to run all tests, then they'll
> need to use a VPN.
>
> Regardless of that though, we should make sure that tests automatically
> mark themselves as "skipped" if any downloading of assets fails. ie a
> download failure shouldn't cause tests to report failure.

That is the current behavior, tests are marked "CANCEL" in case the 
download failed.

PS: canceled tests don't change the overall status of the job.

- Wainer

>
> Regards,
> Daniel

Re: Problem running functional tests from China

[Wainer dos Santos Moschetta]

Hi,

On 2/24/21 9:01 AM, Michal Suchánek wrote:
> On Wed, Feb 24, 2021 at 12:37:53PM +0100, Philippe Mathieu-Daudé wrote:
>> On 2/24/21 12:06 PM, Michal Suchánek wrote:
>>> On Wed, Feb 24, 2021 at 10:29:15AM +0100, Philippe Mathieu-Daudé wrote:
>>>> Just to inform the files hosted on github don't work when
>>>> testing from China:
>>>> raw.githubusercontent.com resolves to 127.0.0.1
>> Actually not even localhost but 0.0.0.0 :)
>>
>>> And from where does this come from?
>>>
>>> Your local system, your ISP, ... ?
>> Probably the ISP:
>>
> ...
>> I raised that problem not to find a way to bypass an ISP
>> firewall, but to see if there is a way to use another
>> storage for test artifacts so all the community can run
>> the tests.
> I don't think protecting from random network malfunction is something
> the qemu project can do.
>
> That said, downloading the test data during test run does indeed look
> fragile.

For those functional tests, the test data is fetched once and cached 
locally.

The user can also inject files on Avocado's cache directory, by using 
the `assets` sub-command:

---

$ ./build/tests/venv/bin/avocado assets register --help
usage: avocado assets register [-h] [--hash SHA1] name url

positional arguments:
   name         Unique name to associate with this asset.
   url          Path to asset that you would like to register manually.

optional arguments:
   -h, --help   show this help message and exit
   --hash SHA1  SHA1 hash of this asset.

----

More features for managing the cache has been implemented, e.g.:

https://github.com/avocado-framework/avocado/issues/4311

Thanks!

- Wainer

>
> We have the concept of git submodules which is used in qemu extensively
> so the test data could possibly be included directly in the git tree.
>
> The obvious downside is that the current and past test binaries will take
> up disk space for all users that check out the repository, even those
> that don't run the tests.
>
> Thanks
>
> Michal
>