[RFC PATCH] tests/avocado: use new rootfs for orangepi test

[RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Alex Bennée]

The old URL wasn't stable. I suspect the current URL will only be
stable for a few months so maybe we need another strategy for hosting
rootfs snapshots?

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 tests/avocado/boot_linux_console.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
index 4c9d551f47..5a2923c423 100644
--- a/tests/avocado/boot_linux_console.py
+++ b/tests/avocado/boot_linux_console.py
@@ -793,8 +793,8 @@ def test_arm_orangepi_sd(self):
         dtb_path = '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
         dtb_path = self.extract_from_deb(deb_path, dtb_path)
         rootfs_url = ('http://storage.kernelci.org/images/rootfs/buildroot/'
-                      'kci-2019.02/armel/base/rootfs.ext2.xz')
-        rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
+                      'buildroot-baseline/20221116.0/armel/rootfs.ext2.xz')
+        rootfs_hash = 'fae32f337c7b87547b10f42599acf109da8b6d9a'
         rootfs_path_xz = self.fetch_asset(rootfs_url, asset_hash=rootfs_hash)
         rootfs_path = os.path.join(self.workdir, 'rootfs.cpio')
         archive.lzma_uncompress(rootfs_path_xz, rootfs_path)
-- 
2.34.1

Re: [RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Thomas Huth]

On 18/11/2022 12.33, Alex Bennée wrote:
> The old URL wasn't stable. I suspect the current URL will only be
> stable for a few months so maybe we need another strategy for hosting
> rootfs snapshots?

Looking at 
http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/ 
there are only images for the very last month available - so the new URL 
will likely be invalid in one month already. That's not usable. I think 
you've got to find another solution or remove the test.

  Thomas

Re: [RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Thomas Huth]

On 18/11/2022 15.18, Thomas Huth wrote:
> On 18/11/2022 12.33, Alex Bennée wrote:
>> The old URL wasn't stable. I suspect the current URL will only be
>> stable for a few months so maybe we need another strategy for hosting
>> rootfs snapshots?
> 
> Looking at 
> http://storage.kernelci.org/images/rootfs/buildroot/buildroot-baseline/ 
> there are only images for the very last month available - so the new URL 
> will likely be invalid in one month already. That's not usable. I think 
> you've got to find another solution or remove the test.

Looking at that folder again, it seems there wasn't any more change within 
the last week ... so maybe it's not rotating so fast as I was thinking 
initially ... thus I guess we could give the patch a try, and if the current 
image finally disappears again, we still can think of a different solution.

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Philippe Mathieu-Daudé]

On 18/11/22 12:33, Alex Bennée wrote:
> The old URL wasn't stable. I suspect the current URL will only be
> stable for a few months so maybe we need another strategy for hosting
> rootfs snapshots?
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>   tests/avocado/boot_linux_console.py | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
> index 4c9d551f47..5a2923c423 100644
> --- a/tests/avocado/boot_linux_console.py
> +++ b/tests/avocado/boot_linux_console.py
> @@ -793,8 +793,8 @@ def test_arm_orangepi_sd(self):
>           dtb_path = '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
>           dtb_path = self.extract_from_deb(deb_path, dtb_path)
>           rootfs_url = ('http://storage.kernelci.org/images/rootfs/buildroot/'
> -                      'kci-2019.02/armel/base/rootfs.ext2.xz')
> -        rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
> +                      'buildroot-baseline/20221116.0/armel/rootfs.ext2.xz')
> +        rootfs_hash = 'fae32f337c7b87547b10f42599acf109da8b6d9a'
If Avocado doesn't find an artifact in its local cache, it will fetch it
from the URL.
The cache might be populated with artifacts previously downloaded, but
their URL is not valid anymore (my case for many tests).
We can also add artifacts manually, see [1].

I'd rather keep pre-existing tests if possible, to test older (kernel / 
user-space) images. We don't need to run all the tests all the time:
tests can be filtered by tags (see [2]).

My preference here is to refactor this test, adding the "kci-2019.02"
and "baseline-20221116.0" releases. I can prepare the patch if you /
Thomas don't object.

Regards,

Phil.

[1] 
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/assets.html#registering-assets
[2] 
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html#filtering-tests-by-tags

Re: [RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Thomas Huth]

On 23/11/2022 12.15, Philippe Mathieu-Daudé wrote:
> On 18/11/22 12:33, Alex Bennée wrote:
>> The old URL wasn't stable. I suspect the current URL will only be
>> stable for a few months so maybe we need another strategy for hosting
>> rootfs snapshots?
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> ---
>>   tests/avocado/boot_linux_console.py | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/avocado/boot_linux_console.py 
>> b/tests/avocado/boot_linux_console.py
>> index 4c9d551f47..5a2923c423 100644
>> --- a/tests/avocado/boot_linux_console.py
>> +++ b/tests/avocado/boot_linux_console.py
>> @@ -793,8 +793,8 @@ def test_arm_orangepi_sd(self):
>>           dtb_path = 
>> '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
>>           dtb_path = self.extract_from_deb(deb_path, dtb_path)
>>           rootfs_url = 
>> ('http://storage.kernelci.org/images/rootfs/buildroot/'
>> -                      'kci-2019.02/armel/base/rootfs.ext2.xz')
>> -        rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
>> +                      'buildroot-baseline/20221116.0/armel/rootfs.ext2.xz')
>> +        rootfs_hash = 'fae32f337c7b87547b10f42599acf109da8b6d9a'
> If Avocado doesn't find an artifact in its local cache, it will fetch it
> from the URL.
> The cache might be populated with artifacts previously downloaded, but
> their URL is not valid anymore (my case for many tests).
> We can also add artifacts manually, see [1].
> 
> I'd rather keep pre-existing tests if possible, to test older (kernel / 
> user-space) images. We don't need to run all the tests all the time:
> tests can be filtered by tags (see [2]).
> 
> My preference here is to refactor this test, adding the "kci-2019.02"
> and "baseline-20221116.0" releases. I can prepare the patch if you /
> Thomas don't object.

IMHO we shouldn't keep tests in the upstream git repository where the 
binaries are not available in public anymore. They won't get run by new 
contributors anymore, and also could vanish from the disks of the people who 
previously downloaded it, once they wipe their cache or upgrade to a new 
installation, so the test code will sooner or later be bitrotting. But if 
you want to keep the tests around on your hard disk, you could also stick 
the test in a local branch on your hard disk instead.

The other possibility is to upload the binaries to a new public location in 
the web ... but for software that contains GPLed software, you should then 
also make sure to provide the source code to comply with the license.

  Thomas

Re: [RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> On 23/11/2022 12.15, Philippe Mathieu-Daudé wrote:
>> On 18/11/22 12:33, Alex Bennée wrote:
>>> The old URL wasn't stable. I suspect the current URL will only be
>>> stable for a few months so maybe we need another strategy for hosting
>>> rootfs snapshots?
>>>
>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>> ---
>>>   tests/avocado/boot_linux_console.py | 4 ++--
>>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/tests/avocado/boot_linux_console.py
>>> b/tests/avocado/boot_linux_console.py
>>> index 4c9d551f47..5a2923c423 100644
>>> --- a/tests/avocado/boot_linux_console.py
>>> +++ b/tests/avocado/boot_linux_console.py
>>> @@ -793,8 +793,8 @@ def test_arm_orangepi_sd(self):
>>>           dtb_path =
>>> '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
>>>           dtb_path = self.extract_from_deb(deb_path, dtb_path)
>>>           rootfs_url =
>>> ('http://storage.kernelci.org/images/rootfs/buildroot/'
>>> -                      'kci-2019.02/armel/base/rootfs.ext2.xz')
>>> -        rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
>>> +                      'buildroot-baseline/20221116.0/armel/rootfs.ext2.xz')
>>> +        rootfs_hash = 'fae32f337c7b87547b10f42599acf109da8b6d9a'
>> If Avocado doesn't find an artifact in its local cache, it will fetch it
>> from the URL.
>> The cache might be populated with artifacts previously downloaded, but
>> their URL is not valid anymore (my case for many tests).
>> We can also add artifacts manually, see [1].
>> I'd rather keep pre-existing tests if possible, to test older
>> (kernel / user-space) images. We don't need to run all the tests all
>> the time:
>> tests can be filtered by tags (see [2]).
>> My preference here is to refactor this test, adding the
>> "kci-2019.02"
>> and "baseline-20221116.0" releases. I can prepare the patch if you /
>> Thomas don't object.
>
> IMHO we shouldn't keep tests in the upstream git repository where the
> binaries are not available in public anymore. They won't get run by
> new contributors anymore, and also could vanish from the disks of the
> people who previously downloaded it, once they wipe their cache or
> upgrade to a new installation, so the test code will sooner or later
> be bitrotting. But if you want to keep the tests around on your hard
> disk, you could also stick the test in a local branch on your hard
> disk instead.

CI/Workstation splits aside I tend to agree with Thomas here that having
tests no one else can run will lead to an accretion of broken tests.
Given the tests themselves are standalone couldn't the prospective test
hoarder keep their own personal repository to be run with the rest of the
in-tree code, something like:

  cd my/test/zoo/repo
  $(QEMU_BUILD)/tests/venv/bin/avocado run my_test_zoo.py

for convenience we could maybe support an env variable so the existing
test selection tags would work:

  set -x QEMU_AVOCADO_EXTRA_TESTS /my/test/zoo/repo
  ./tests/venv/bin/avocado list
  ...
  <list all tests in qemu src tree and extra>
  ...

?

> The other possibility is to upload the binaries to a new public
> location in the web ... but for software that contains GPLed software,
> you should then also make sure to provide the source code to comply
> with the license.

This is the traditional reason we've lent so hard on external hosting
for binaries because the upstream doesn't want the hassle of maintaining
that sort of zoo of binaries. That said we have tests where binaries are
served from fileserver.linaro.org but its then only my problem to deal
with GPL requirements and not the upstream.

-- 
Alex Bennée

Re: [RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Philippe Mathieu-Daudé]

On 23/11/22 15:12, Alex Bennée wrote:
> Thomas Huth <thuth@redhat.com> writes:
>> On 23/11/2022 12.15, Philippe Mathieu-Daudé wrote:
>>> On 18/11/22 12:33, Alex Bennée wrote:
>>>> The old URL wasn't stable. I suspect the current URL will only be
>>>> stable for a few months so maybe we need another strategy for hosting
>>>> rootfs snapshots?
>>>>
>>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>> ---
>>>>    tests/avocado/boot_linux_console.py | 4 ++--
>>>>    1 file changed, 2 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/tests/avocado/boot_linux_console.py
>>>> b/tests/avocado/boot_linux_console.py
>>>> index 4c9d551f47..5a2923c423 100644
>>>> --- a/tests/avocado/boot_linux_console.py
>>>> +++ b/tests/avocado/boot_linux_console.py
>>>> @@ -793,8 +793,8 @@ def test_arm_orangepi_sd(self):
>>>>            dtb_path =
>>>> '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
>>>>            dtb_path = self.extract_from_deb(deb_path, dtb_path)
>>>>            rootfs_url =
>>>> ('http://storage.kernelci.org/images/rootfs/buildroot/'
>>>> -                      'kci-2019.02/armel/base/rootfs.ext2.xz')
>>>> -        rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
>>>> +                      'buildroot-baseline/20221116.0/armel/rootfs.ext2.xz')
>>>> +        rootfs_hash = 'fae32f337c7b87547b10f42599acf109da8b6d9a'
>>> If Avocado doesn't find an artifact in its local cache, it will fetch it
>>> from the URL.
>>> The cache might be populated with artifacts previously downloaded, but
>>> their URL is not valid anymore (my case for many tests).
>>> We can also add artifacts manually, see [1].
>>> I'd rather keep pre-existing tests if possible, to test older
>>> (kernel / user-space) images. We don't need to run all the tests all
>>> the time:
>>> tests can be filtered by tags (see [2]).
>>> My preference here is to refactor this test, adding the
>>> "kci-2019.02"
>>> and "baseline-20221116.0" releases. I can prepare the patch if you /
>>> Thomas don't object.
>>
>> IMHO we shouldn't keep tests in the upstream git repository where the
>> binaries are not available in public anymore. They won't get run by
>> new contributors anymore, and also could vanish from the disks of the
>> people who previously downloaded it, once they wipe their cache or
>> upgrade to a new installation, so the test code will sooner or later
>> be bitrotting. But if you want to keep the tests around on your hard
>> disk, you could also stick the test in a local branch on your hard
>> disk instead.
> 
> CI/Workstation splits aside I tend to agree with Thomas here that having
> tests no one else can run will lead to an accretion of broken tests.

Following this idea, should we remove all boards for which no open
source & GPL software is available? I.e:

40p                  IBM RS/6000 7020 (40p)
akita                Sharp SL-C1000 (Akita) PDA (PXA270)
midway               Calxeda Midway (ECX-2000)
terrier              Sharp SL-C3200 (Terrier) PDA (PXA270)
tosa                 Sharp SL-6000 (Tosa) PDA (PXA255)

> Given the tests themselves are standalone couldn't the prospective test
> hoarder keep their own personal repository to be run with the rest of the
> in-tree code, something like:
> 
>    cd my/test/zoo/repo
>    $(QEMU_BUILD)/tests/venv/bin/avocado run my_test_zoo.py
> 
> for convenience we could maybe support an env variable so the existing
> test selection tags would work:
> 
>    set -x QEMU_AVOCADO_EXTRA_TESTS /my/test/zoo/repo
>    ./tests/venv/bin/avocado list
>    ...
>    <list all tests in qemu src tree and extra>
>    ...
> 
> ?

Yes, this is what we use to test the Fuloong2E:

$ git grep RESCUE_YL_PATH tests/avocado/
tests/avocado/machine_mips_fuloong2e.py:21: 
@skipUnless(os.getenv('RESCUE_YL_PATH'), 'RESCUE_YL_PATH not available')
tests/avocado/machine_mips_fuloong2e.py:34:        kernel_path = 
self.fetch_asset('file://' + os.getenv('RESCUE_YL_PATH'),

The firmware is not open source / GPL but if you have a Fuloong2E board
you can dump it from the flash, then use it to test QEMU from hard reset
up to userland. Otherwise you are forced to use the -kernel argument.

>> The other possibility is to upload the binaries to a new public
>> location in the web ... but for software that contains GPLed software,
>> you should then also make sure to provide the source code to comply
>> with the license.
> 
> This is the traditional reason we've lent so hard on external hosting
> for binaries because the upstream doesn't want the hassle of maintaining
> that sort of zoo of binaries. That said we have tests where binaries are
> served from fileserver.linaro.org but its then only my problem to deal
> with GPL requirements and not the upstream.

Maybe we are discussing 2 different topics. I am in possession of
old Solaris installation CDROMs and could boot some of them with
qemu-system-sparc. I want to automatize my testing, and wrote Avocado
scripts doing that. I suppose other QEMU users have similar CDROMs.
If I contribute my tests, they can run them. Isn't it in the interest
of the community to have such examples and tests available?

Regards,

Phil.

Re: [RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Cédric Le Goater]

On 11/23/22 19:13, Philippe Mathieu-Daudé wrote:
> On 23/11/22 15:12, Alex Bennée wrote:
>> Thomas Huth <thuth@redhat.com> writes:
>>> On 23/11/2022 12.15, Philippe Mathieu-Daudé wrote:
>>>> On 18/11/22 12:33, Alex Bennée wrote:
>>>>> The old URL wasn't stable. I suspect the current URL will only be
>>>>> stable for a few months so maybe we need another strategy for hosting
>>>>> rootfs snapshots?
>>>>>
>>>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>>> ---
>>>>>    tests/avocado/boot_linux_console.py | 4 ++--
>>>>>    1 file changed, 2 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/tests/avocado/boot_linux_console.py
>>>>> b/tests/avocado/boot_linux_console.py
>>>>> index 4c9d551f47..5a2923c423 100644
>>>>> --- a/tests/avocado/boot_linux_console.py
>>>>> +++ b/tests/avocado/boot_linux_console.py
>>>>> @@ -793,8 +793,8 @@ def test_arm_orangepi_sd(self):
>>>>>            dtb_path =
>>>>> '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
>>>>>            dtb_path = self.extract_from_deb(deb_path, dtb_path)
>>>>>            rootfs_url =
>>>>> ('http://storage.kernelci.org/images/rootfs/buildroot/'
>>>>> -                      'kci-2019.02/armel/base/rootfs.ext2.xz')
>>>>> -        rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
>>>>> +                      'buildroot-baseline/20221116.0/armel/rootfs.ext2.xz')
>>>>> +        rootfs_hash = 'fae32f337c7b87547b10f42599acf109da8b6d9a'
>>>> If Avocado doesn't find an artifact in its local cache, it will fetch it
>>>> from the URL.
>>>> The cache might be populated with artifacts previously downloaded, but
>>>> their URL is not valid anymore (my case for many tests).
>>>> We can also add artifacts manually, see [1].
>>>> I'd rather keep pre-existing tests if possible, to test older
>>>> (kernel / user-space) images. We don't need to run all the tests all
>>>> the time:
>>>> tests can be filtered by tags (see [2]).
>>>> My preference here is to refactor this test, adding the
>>>> "kci-2019.02"
>>>> and "baseline-20221116.0" releases. I can prepare the patch if you /
>>>> Thomas don't object.
>>>
>>> IMHO we shouldn't keep tests in the upstream git repository where the
>>> binaries are not available in public anymore. They won't get run by
>>> new contributors anymore, and also could vanish from the disks of the
>>> people who previously downloaded it, once they wipe their cache or
>>> upgrade to a new installation, so the test code will sooner or later
>>> be bitrotting. But if you want to keep the tests around on your hard
>>> disk, you could also stick the test in a local branch on your hard
>>> disk instead.
>>
>> CI/Workstation splits aside I tend to agree with Thomas here that having
>> tests no one else can run will lead to an accretion of broken tests.
> 
> Following this idea, should we remove all boards for which no open
> source & GPL software is available? I.e:
> 
> 40p                  IBM RS/6000 7020 (40p)

This machine can run debian :

   qemu-system-ppc -M 40p -cpu 604 -nic user -hda ./prep.qcow2 -cdrom ./zImage.hdd -serial mon:stdio -nographic
   
   >> =============================================================
   >> OpenBIOS 1.1 [Mar 7 2022 23:07]
   >> Configuration device id QEMU version 1 machine id 0
   >> CPUs: 0
   >> Memory: 128M
   >> UUID: 00000000-0000-0000-0000-000000000000
   >> CPU type PowerPC,604
   milliseconds isn't unique.
   Welcome to OpenBIOS v1.1 built on Mar 7 2022 23:07
   Trying hd:,\\:tbxi...
   >> Not a bootable ELF image
   >> switching to new context:
   loaded at:     04000400 04015218
   relocated to:  00800000 00814E18
   board data at: 07C9E870 07CA527C
   relocated to:  0080B130 00811B3C
   zimage at:     0400B400 0411DC98
   avail ram:     00400000 00800000
   
   Linux/PPC load: console=/dev/ttyS0,9600 console=tty0 ether=5,0x210,eth0 ether=11,0x300,eth1 ramdisk_size=8192 root=/dev/sda3
   Uncompressing Linux................................................done.
   Now booting the kernel
   
   Debian GNU/Linux 3.0 6015 ttyS0
   
   6015 login:

Please keep it ! :)

and it also boots AIX 4.4/5.1 (with 2 small patches) but that's clearly
not open source. It is downloadable from the net though, like many macos
PPC images.

That said, we might have been putting too much in avocado and it takes
ages to run (when it does not hit some random python issue).


> akita                Sharp SL-C1000 (Akita) PDA (PXA270)
> midway               Calxeda Midway (ECX-2000)
> terrier              Sharp SL-C3200 (Terrier) PDA (PXA270)
> tosa                 Sharp SL-6000 (Tosa) PDA (PXA255)
> 
>> Given the tests themselves are standalone couldn't the prospective test
>> hoarder keep their own personal repository to be run with the rest of the
>> in-tree code, something like:
>>
>>    cd my/test/zoo/repo
>>    $(QEMU_BUILD)/tests/venv/bin/avocado run my_test_zoo.py
>>
>> for convenience we could maybe support an env variable so the existing
>> test selection tags would work:
>>
>>    set -x QEMU_AVOCADO_EXTRA_TESTS /my/test/zoo/repo
>>    ./tests/venv/bin/avocado list
>>    ...
>>    <list all tests in qemu src tree and extra>
>>    ...
>>
>> ?
> 
> Yes, this is what we use to test the Fuloong2E:
> 
> $ git grep RESCUE_YL_PATH tests/avocado/
> tests/avocado/machine_mips_fuloong2e.py:21: @skipUnless(os.getenv('RESCUE_YL_PATH'), 'RESCUE_YL_PATH not available')
> tests/avocado/machine_mips_fuloong2e.py:34:        kernel_path = self.fetch_asset('file://' + os.getenv('RESCUE_YL_PATH'),
> 
> The firmware is not open source / GPL but if you have a Fuloong2E board
> you can dump it from the flash, then use it to test QEMU from hard reset
> up to userland. Otherwise you are forced to use the -kernel argument.
> 
>>> The other possibility is to upload the binaries to a new public
>>> location in the web ... but for software that contains GPLed software,
>>> you should then also make sure to provide the source code to comply
>>> with the license.
>>
>> This is the traditional reason we've lent so hard on external hosting
>> for binaries because the upstream doesn't want the hassle of maintaining
>> that sort of zoo of binaries. That said we have tests where binaries are
>> served from fileserver.linaro.org but its then only my problem to deal
>> with GPL requirements and not the upstream.
> 
> Maybe we are discussing 2 different topics. I am in possession of
> old Solaris installation CDROMs and could boot some of them with
> qemu-system-sparc. I want to automatize my testing, and wrote Avocado
> scripts doing that. I suppose other QEMU users have similar CDROMs.
> If I contribute my tests, they can run them. Isn't it in the interest
> of the community to have such examples and tests available?

I think so.

Is it time to move some of the tests (and images) in an external tree ?
That would be one way to keep them available for all. I like qemu-zoo.

It would certainly require some legal advice.
  

C.


> 
> Regards,
> 
> Phil.
> 

Re: [RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Philippe Mathieu-Daudé]

On 23/11/22 19:49, Cédric Le Goater wrote:
> On 11/23/22 19:13, Philippe Mathieu-Daudé wrote:
>> On 23/11/22 15:12, Alex Bennée wrote:
>>> Thomas Huth <thuth@redhat.com> writes:
>>>> On 23/11/2022 12.15, Philippe Mathieu-Daudé wrote:
>>>>> On 18/11/22 12:33, Alex Bennée wrote:
>>>>>> The old URL wasn't stable. I suspect the current URL will only be
>>>>>> stable for a few months so maybe we need another strategy for hosting
>>>>>> rootfs snapshots?
>>>>>>
>>>>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>>>> ---
>>>>>>    tests/avocado/boot_linux_console.py | 4 ++--
>>>>>>    1 file changed, 2 insertions(+), 2 deletions(-)
>>>>>>
>>>>>> diff --git a/tests/avocado/boot_linux_console.py
>>>>>> b/tests/avocado/boot_linux_console.py
>>>>>> index 4c9d551f47..5a2923c423 100644
>>>>>> --- a/tests/avocado/boot_linux_console.py
>>>>>> +++ b/tests/avocado/boot_linux_console.py
>>>>>> @@ -793,8 +793,8 @@ def test_arm_orangepi_sd(self):
>>>>>>            dtb_path =
>>>>>> '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
>>>>>>            dtb_path = self.extract_from_deb(deb_path, dtb_path)
>>>>>>            rootfs_url =
>>>>>> ('http://storage.kernelci.org/images/rootfs/buildroot/'
>>>>>> -                      'kci-2019.02/armel/base/rootfs.ext2.xz')
>>>>>> -        rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
>>>>>> +                      
>>>>>> 'buildroot-baseline/20221116.0/armel/rootfs.ext2.xz')
>>>>>> +        rootfs_hash = 'fae32f337c7b87547b10f42599acf109da8b6d9a'
>>>>> If Avocado doesn't find an artifact in its local cache, it will 
>>>>> fetch it
>>>>> from the URL.
>>>>> The cache might be populated with artifacts previously downloaded, but
>>>>> their URL is not valid anymore (my case for many tests).
>>>>> We can also add artifacts manually, see [1].
>>>>> I'd rather keep pre-existing tests if possible, to test older
>>>>> (kernel / user-space) images. We don't need to run all the tests all
>>>>> the time:
>>>>> tests can be filtered by tags (see [2]).
>>>>> My preference here is to refactor this test, adding the
>>>>> "kci-2019.02"
>>>>> and "baseline-20221116.0" releases. I can prepare the patch if you /
>>>>> Thomas don't object.
>>>>
>>>> IMHO we shouldn't keep tests in the upstream git repository where the
>>>> binaries are not available in public anymore. They won't get run by
>>>> new contributors anymore, and also could vanish from the disks of the
>>>> people who previously downloaded it, once they wipe their cache or
>>>> upgrade to a new installation, so the test code will sooner or later
>>>> be bitrotting. But if you want to keep the tests around on your hard
>>>> disk, you could also stick the test in a local branch on your hard
>>>> disk instead.
>>>
>>> CI/Workstation splits aside I tend to agree with Thomas here that having
>>> tests no one else can run will lead to an accretion of broken tests.
>>
>> Following this idea, should we remove all boards for which no open
>> source & GPL software is available? I.e:
>>
>> 40p                  IBM RS/6000 7020 (40p)
> 
> This machine can run debian :

IMHO having QEMU able to run anything an architecture can run seems way
more interesting/helpful rather than restricting it to just open source
projects.

>    qemu-system-ppc -M 40p -cpu 604 -nic user -hda ./prep.qcow2 -cdrom 
> ./zImage.hdd -serial mon:stdio -nographic
>    >> =============================================================
>    >> OpenBIOS 1.1 [Mar 7 2022 23:07]
>    >> Configuration device id QEMU version 1 machine id 0
>    >> CPUs: 0
>    >> Memory: 128M
>    >> UUID: 00000000-0000-0000-0000-000000000000
>    >> CPU type PowerPC,604
>    milliseconds isn't unique.
>    Welcome to OpenBIOS v1.1 built on Mar 7 2022 23:07
>    Trying hd:,\\:tbxi...
>    >> Not a bootable ELF image
>    >> switching to new context:
>    loaded at:     04000400 04015218
>    relocated to:  00800000 00814E18
>    board data at: 07C9E870 07CA527C
>    relocated to:  0080B130 00811B3C
>    zimage at:     0400B400 0411DC98
>    avail ram:     00400000 00800000
>    Linux/PPC load: console=/dev/ttyS0,9600 console=tty0 
> ether=5,0x210,eth0 ether=11,0x300,eth1 ramdisk_size=8192 root=/dev/sda3
>    Uncompressing Linux................................................done.
>    Now booting the kernel
>    Debian GNU/Linux 3.0 6015 ttyS0
>    6015 login:
> 
> Please keep it ! :)
> 
> and it also boots AIX 4.4/5.1 (with 2 small patches) but that's clearly
> not open source. It is downloadable from the net though, like many macos
> PPC images.
> 
> That said, we might have been putting too much in avocado and it takes
> ages to run (when it does not hit some random python issue).

w.r.t. "too much in avocado", are you referring to GitLab CI?

I see the following 2 use cases with Avocado:
  1/ Run tests locally
  2/ Run tests on CI
The set of tests used in 1/ and 2/ doesn't have to be the same...

1/ is very helpful for maintainers, to run tests specific to their
subsystems. Also useful during refactor when touching other subsystems,
to run their tests before sending a patch set.

2/ is the "gating" testing. With retrospective, it was a mistake to
start running avocado on CI without any filtering on what tests to run.
Instead of trying to explain my view here, I'd like to go back to Daniel
earlier proposal:
https://lore.kernel.org/qemu-devel/20200427152036.GI1244803@redhat.com/

Per this proposal, we should only run 'Tier 1' on Gitlab CI.
Daniel described "Tier 1" as "[test that] Will always work." I'd like to
amend with "test that run in less than 150 seconds" (or less). If a test
takes more, we can run it on our workstations, but we shouldn't waste
CI cycles with it.

I plan to post a series converting our current Avocado "opt-out" use:

   @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')

to and "opt-in" one using the 'gating-ci-tier:1' Avocado tag.

(if curious: https://gitlab.com/philmd/qemu/-/commits/gci_tier1_optin/)

>>> Given the tests themselves are standalone couldn't the prospective test
>>> hoarder keep their own personal repository to be run with the rest of 
>>> the
>>> in-tree code, something like:
>>>
>>>    cd my/test/zoo/repo
>>>    $(QEMU_BUILD)/tests/venv/bin/avocado run my_test_zoo.py
>>>
>>> for convenience we could maybe support an env variable so the existing
>>> test selection tags would work:
>>>
>>>    set -x QEMU_AVOCADO_EXTRA_TESTS /my/test/zoo/repo
>>>    ./tests/venv/bin/avocado list
>>>    ...
>>>    <list all tests in qemu src tree and extra>
>>>    ...
>>>
>>> ?
>>
>> Yes, this is what we use to test the Fuloong2E:
>>
>> $ git grep RESCUE_YL_PATH tests/avocado/
>> tests/avocado/machine_mips_fuloong2e.py:21: 
>> @skipUnless(os.getenv('RESCUE_YL_PATH'), 'RESCUE_YL_PATH not available')
>> tests/avocado/machine_mips_fuloong2e.py:34:        kernel_path = 
>> self.fetch_asset('file://' + os.getenv('RESCUE_YL_PATH'),
>>
>> The firmware is not open source / GPL but if you have a Fuloong2E board
>> you can dump it from the flash, then use it to test QEMU from hard reset
>> up to userland. Otherwise you are forced to use the -kernel argument.
>>
>>>> The other possibility is to upload the binaries to a new public
>>>> location in the web ... but for software that contains GPLed software,
>>>> you should then also make sure to provide the source code to comply
>>>> with the license.
>>>
>>> This is the traditional reason we've lent so hard on external hosting
>>> for binaries because the upstream doesn't want the hassle of maintaining
>>> that sort of zoo of binaries. That said we have tests where binaries are
>>> served from fileserver.linaro.org but its then only my problem to deal
>>> with GPL requirements and not the upstream.
>>
>> Maybe we are discussing 2 different topics. I am in possession of
>> old Solaris installation CDROMs and could boot some of them with
>> qemu-system-sparc. I want to automatize my testing, and wrote Avocado
>> scripts doing that. I suppose other QEMU users have similar CDROMs.
>> If I contribute my tests, they can run them. Isn't it in the interest
>> of the community to have such examples and tests available?
> 
> I think so.
> 
> Is it time to move some of the tests (and images) in an external tree ?
> That would be one way to keep them available for all. I like qemu-zoo.
> 
> It would certainly require some legal advice.
> 
> 
> C.
> 
> 
>>
>> Regards,
>>
>> Phil.
>>
> 

Re: [RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Daniel P. Berrangé]

On Thu, Nov 24, 2022 at 12:06:10AM +0100, Philippe Mathieu-Daudé wrote:
> On 23/11/22 19:49, Cédric Le Goater wrote:
> > On 11/23/22 19:13, Philippe Mathieu-Daudé wrote:
> > > On 23/11/22 15:12, Alex Bennée wrote:
> > > > Thomas Huth <thuth@redhat.com> writes:
> > > > > On 23/11/2022 12.15, Philippe Mathieu-Daudé wrote:
> > > > > > On 18/11/22 12:33, Alex Bennée wrote:
> > > > > > > The old URL wasn't stable. I suspect the current URL will only be
> > > > > > > stable for a few months so maybe we need another strategy for hosting
> > > > > > > rootfs snapshots?
> > > > > > > 
> > > > > > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > > > > > > ---
> > > > > > >    tests/avocado/boot_linux_console.py | 4 ++--
> > > > > > >    1 file changed, 2 insertions(+), 2 deletions(-)
> > > > > > > 
> > > > > > > diff --git a/tests/avocado/boot_linux_console.py
> > > > > > > b/tests/avocado/boot_linux_console.py
> > > > > > > index 4c9d551f47..5a2923c423 100644
> > > > > > > --- a/tests/avocado/boot_linux_console.py
> > > > > > > +++ b/tests/avocado/boot_linux_console.py
> > > > > > > @@ -793,8 +793,8 @@ def test_arm_orangepi_sd(self):
> > > > > > >            dtb_path =
> > > > > > > '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
> > > > > > >            dtb_path = self.extract_from_deb(deb_path, dtb_path)
> > > > > > >            rootfs_url =
> > > > > > > ('http://storage.kernelci.org/images/rootfs/buildroot/'
> > > > > > > -                      'kci-2019.02/armel/base/rootfs.ext2.xz')
> > > > > > > -        rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
> > > > > > > +
> > > > > > > 'buildroot-baseline/20221116.0/armel/rootfs.ext2.xz')
> > > > > > > +        rootfs_hash = 'fae32f337c7b87547b10f42599acf109da8b6d9a'
> > > > > > If Avocado doesn't find an artifact in its local cache,
> > > > > > it will fetch it
> > > > > > from the URL.
> > > > > > The cache might be populated with artifacts previously downloaded, but
> > > > > > their URL is not valid anymore (my case for many tests).
> > > > > > We can also add artifacts manually, see [1].
> > > > > > I'd rather keep pre-existing tests if possible, to test older
> > > > > > (kernel / user-space) images. We don't need to run all the tests all
> > > > > > the time:
> > > > > > tests can be filtered by tags (see [2]).
> > > > > > My preference here is to refactor this test, adding the
> > > > > > "kci-2019.02"
> > > > > > and "baseline-20221116.0" releases. I can prepare the patch if you /
> > > > > > Thomas don't object.
> > > > > 
> > > > > IMHO we shouldn't keep tests in the upstream git repository where the
> > > > > binaries are not available in public anymore. They won't get run by
> > > > > new contributors anymore, and also could vanish from the disks of the
> > > > > people who previously downloaded it, once they wipe their cache or
> > > > > upgrade to a new installation, so the test code will sooner or later
> > > > > be bitrotting. But if you want to keep the tests around on your hard
> > > > > disk, you could also stick the test in a local branch on your hard
> > > > > disk instead.
> > > > 
> > > > CI/Workstation splits aside I tend to agree with Thomas here that having
> > > > tests no one else can run will lead to an accretion of broken tests.
> > > 
> > > Following this idea, should we remove all boards for which no open
> > > source & GPL software is available? I.e:
> > > 
> > > 40p                  IBM RS/6000 7020 (40p)
> > 
> > This machine can run debian :
> 
> IMHO having QEMU able to run anything an architecture can run seems way
> more interesting/helpful rather than restricting it to just open source
> projects.
> 
> >    qemu-system-ppc -M 40p -cpu 604 -nic user -hda ./prep.qcow2 -cdrom
> > ./zImage.hdd -serial mon:stdio -nographic
> >    >> =============================================================
> >    >> OpenBIOS 1.1 [Mar 7 2022 23:07]
> >    >> Configuration device id QEMU version 1 machine id 0
> >    >> CPUs: 0
> >    >> Memory: 128M
> >    >> UUID: 00000000-0000-0000-0000-000000000000
> >    >> CPU type PowerPC,604
> >    milliseconds isn't unique.
> >    Welcome to OpenBIOS v1.1 built on Mar 7 2022 23:07
> >    Trying hd:,\\:tbxi...
> >    >> Not a bootable ELF image
> >    >> switching to new context:
> >    loaded at:     04000400 04015218
> >    relocated to:  00800000 00814E18
> >    board data at: 07C9E870 07CA527C
> >    relocated to:  0080B130 00811B3C
> >    zimage at:     0400B400 0411DC98
> >    avail ram:     00400000 00800000
> >    Linux/PPC load: console=/dev/ttyS0,9600 console=tty0
> > ether=5,0x210,eth0 ether=11,0x300,eth1 ramdisk_size=8192 root=/dev/sda3
> >    Uncompressing Linux................................................done.
> >    Now booting the kernel
> >    Debian GNU/Linux 3.0 6015 ttyS0
> >    6015 login:
> > 
> > Please keep it ! :)
> > 
> > and it also boots AIX 4.4/5.1 (with 2 small patches) but that's clearly
> > not open source. It is downloadable from the net though, like many macos
> > PPC images.
> > 
> > That said, we might have been putting too much in avocado and it takes
> > ages to run (when it does not hit some random python issue).
> 
> w.r.t. "too much in avocado", are you referring to GitLab CI?
> 
> I see the following 2 use cases with Avocado:
>  1/ Run tests locally
>  2/ Run tests on CI
> The set of tests used in 1/ and 2/ doesn't have to be the same...
> 
> 1/ is very helpful for maintainers, to run tests specific to their
> subsystems. Also useful during refactor when touching other subsystems,
> to run their tests before sending a patch set.
> 
> 2/ is the "gating" testing. With retrospective, it was a mistake to
> start running avocado on CI without any filtering on what tests to run.
> Instead of trying to explain my view here, I'd like to go back to Daniel
> earlier proposal:
> https://lore.kernel.org/qemu-devel/20200427152036.GI1244803@redhat.com/
> 
> Per this proposal, we should only run 'Tier 1' on Gitlab CI.
> Daniel described "Tier 1" as "[test that] Will always work."

The key part there is to make clear that testing does not determine
what code we accept into QEMU tree. It merely influences what quality
level we tell users the code has. Ideally we would test everything,
but realistically that's not viable, but we still want to take the
features.

>                                                              I'd like to
> amend with "test that run in less than 150 seconds" (or less). If a test
> takes more, we can run it on our workstations, but we shouldn't waste
> CI cycles with it.

I don't think we need to be so aggressive on time limits for
individual tests. What matters for CI is not the individual
test time, but the overall pipeline wallclock time.

If we want our pipelines to be no longer than 45 minutes,
it is still fine to have 4 tests that run 30 minutes each,
provided we have sufficient resources to run all 4 in parallel.
Keeping tests short is still a good thing, as it lets us run
more overall, but if some need extra time that's ok.

Above all else though, the top 5 requirements for any CI
test we add are reliability, reliability, reliability,
reliability and reliability.

We can't keep spending so much time chasing broken tests.
If the person merging QEMU pull requests just carries on
ignoring tests as they're so frequently broken, the value
of having the tests at all is drastically reduced, in
terms of what they can promise us about quality of the
code we ship.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Thomas Huth]

On 23/11/2022 19.13, Philippe Mathieu-Daudé wrote:
> On 23/11/22 15:12, Alex Bennée wrote:
>> Thomas Huth <thuth@redhat.com> writes:
>>> On 23/11/2022 12.15, Philippe Mathieu-Daudé wrote:
>>>> On 18/11/22 12:33, Alex Bennée wrote:
>>>>> The old URL wasn't stable. I suspect the current URL will only be
>>>>> stable for a few months so maybe we need another strategy for hosting
>>>>> rootfs snapshots?
>>>>>
>>>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>>> ---
>>>>>    tests/avocado/boot_linux_console.py | 4 ++--
>>>>>    1 file changed, 2 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/tests/avocado/boot_linux_console.py
>>>>> b/tests/avocado/boot_linux_console.py
>>>>> index 4c9d551f47..5a2923c423 100644
>>>>> --- a/tests/avocado/boot_linux_console.py
>>>>> +++ b/tests/avocado/boot_linux_console.py
>>>>> @@ -793,8 +793,8 @@ def test_arm_orangepi_sd(self):
>>>>>            dtb_path =
>>>>> '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
>>>>>            dtb_path = self.extract_from_deb(deb_path, dtb_path)
>>>>>            rootfs_url =
>>>>> ('http://storage.kernelci.org/images/rootfs/buildroot/'
>>>>> -                      'kci-2019.02/armel/base/rootfs.ext2.xz')
>>>>> -        rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
>>>>> +                      
>>>>> 'buildroot-baseline/20221116.0/armel/rootfs.ext2.xz')
>>>>> +        rootfs_hash = 'fae32f337c7b87547b10f42599acf109da8b6d9a'
>>>> If Avocado doesn't find an artifact in its local cache, it will fetch it
>>>> from the URL.
>>>> The cache might be populated with artifacts previously downloaded, but
>>>> their URL is not valid anymore (my case for many tests).
>>>> We can also add artifacts manually, see [1].
>>>> I'd rather keep pre-existing tests if possible, to test older
>>>> (kernel / user-space) images. We don't need to run all the tests all
>>>> the time:
>>>> tests can be filtered by tags (see [2]).
>>>> My preference here is to refactor this test, adding the
>>>> "kci-2019.02"
>>>> and "baseline-20221116.0" releases. I can prepare the patch if you /
>>>> Thomas don't object.
>>>
>>> IMHO we shouldn't keep tests in the upstream git repository where the
>>> binaries are not available in public anymore. They won't get run by
>>> new contributors anymore, and also could vanish from the disks of the
>>> people who previously downloaded it, once they wipe their cache or
>>> upgrade to a new installation, so the test code will sooner or later
>>> be bitrotting. But if you want to keep the tests around on your hard
>>> disk, you could also stick the test in a local branch on your hard
>>> disk instead.
>>
>> CI/Workstation splits aside I tend to agree with Thomas here that having
>> tests no one else can run will lead to an accretion of broken tests.
> 
> Following this idea, should we remove all boards for which no open
> source & GPL software is available?

That's certainly a different topic... but I see where you're heading to.

My point here is rather: There is an alternative, newer version of the 
kernel available which can be used to test the same thing. The older version 
disappeared from the net, so why should we bother trying to keep that test 
with that version supported if there is a newer version available?

And for things like firmware images and ISOs, some people still have the 
hardware and CD-ROMs around, or you could buy them second hand, so there is 
indeed a point in having tests for such systems. But for this old kernel 
here? How is somebody supposed to run the test if they didn't download it 
before it disappeared?

If you feel that there is really a benefit for keeping it around, then fine, 
please submit a patch that restores the test with the old kernel. But then 
please also add a comment how to get the kernel if necessary. ... well, 
maybe we just need an owner for the test ... thinking about it twice, I 
think that's what bothered me most about this: It's a test in 
boot_linux_console.py so there is no clear maintainer who could be asked 
about it if it bitrots when the binary is not available. Thus please put it 
into a different file with an entry in MAINTAINERS if you restore it.

So maybe the rule should be: Tests with files that are not publicly 
available in the internet are OK if they have a clear maintainer who could 
be asked about the state if necessary. Tests without available files and 
without maintainer are just too prone to bitrot in the future.

  Thomas

Re: [RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Daniel P. Berrangé]

On Thu, Nov 24, 2022 at 09:20:36AM +0100, Thomas Huth wrote:
> On 23/11/2022 19.13, Philippe Mathieu-Daudé wrote:
> > On 23/11/22 15:12, Alex Bennée wrote:
> > > Thomas Huth <thuth@redhat.com> writes:
> > > > On 23/11/2022 12.15, Philippe Mathieu-Daudé wrote:
> > > > > On 18/11/22 12:33, Alex Bennée wrote:
> > > > > > The old URL wasn't stable. I suspect the current URL will only be
> > > > > > stable for a few months so maybe we need another strategy for hosting
> > > > > > rootfs snapshots?
> > > > > > 
> > > > > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > > > > > ---
> > > > > >    tests/avocado/boot_linux_console.py | 4 ++--
> > > > > >    1 file changed, 2 insertions(+), 2 deletions(-)
> > > > > > 
> > > > > > diff --git a/tests/avocado/boot_linux_console.py
> > > > > > b/tests/avocado/boot_linux_console.py
> > > > > > index 4c9d551f47..5a2923c423 100644
> > > > > > --- a/tests/avocado/boot_linux_console.py
> > > > > > +++ b/tests/avocado/boot_linux_console.py
> > > > > > @@ -793,8 +793,8 @@ def test_arm_orangepi_sd(self):
> > > > > >            dtb_path =
> > > > > > '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
> > > > > >            dtb_path = self.extract_from_deb(deb_path, dtb_path)
> > > > > >            rootfs_url =
> > > > > > ('http://storage.kernelci.org/images/rootfs/buildroot/'
> > > > > > -                      'kci-2019.02/armel/base/rootfs.ext2.xz')
> > > > > > -        rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
> > > > > > +
> > > > > > 'buildroot-baseline/20221116.0/armel/rootfs.ext2.xz')
> > > > > > +        rootfs_hash = 'fae32f337c7b87547b10f42599acf109da8b6d9a'
> > > > > If Avocado doesn't find an artifact in its local cache, it will fetch it
> > > > > from the URL.
> > > > > The cache might be populated with artifacts previously downloaded, but
> > > > > their URL is not valid anymore (my case for many tests).
> > > > > We can also add artifacts manually, see [1].
> > > > > I'd rather keep pre-existing tests if possible, to test older
> > > > > (kernel / user-space) images. We don't need to run all the tests all
> > > > > the time:
> > > > > tests can be filtered by tags (see [2]).
> > > > > My preference here is to refactor this test, adding the
> > > > > "kci-2019.02"
> > > > > and "baseline-20221116.0" releases. I can prepare the patch if you /
> > > > > Thomas don't object.
> > > > 
> > > > IMHO we shouldn't keep tests in the upstream git repository where the
> > > > binaries are not available in public anymore. They won't get run by
> > > > new contributors anymore, and also could vanish from the disks of the
> > > > people who previously downloaded it, once they wipe their cache or
> > > > upgrade to a new installation, so the test code will sooner or later
> > > > be bitrotting. But if you want to keep the tests around on your hard
> > > > disk, you could also stick the test in a local branch on your hard
> > > > disk instead.
> > > 
> > > CI/Workstation splits aside I tend to agree with Thomas here that having
> > > tests no one else can run will lead to an accretion of broken tests.
> > 
> > Following this idea, should we remove all boards for which no open
> > source & GPL software is available?
> 
> That's certainly a different topic... but I see where you're heading to.
> 
> My point here is rather: There is an alternative, newer version of the
> kernel available which can be used to test the same thing. The older version
> disappeared from the net, so why should we bother trying to keep that test
> with that version supported if there is a newer version available?

In the ideal world we would test a representative sample of OS across a
wide vintage of years.

The real world though, we're massively struggling to keep avocado running
reliably, and so don't have resources to achieve ideal testing coverage.

We need to aggressively restrict our focus to testing that maximises
the value for our userbase. I think that means focusing on modern
non-EOL operating systems.  So yes, if an old image disappears, cull
it from testing and put something new in its place that is more likely
to actually be used by our userbase. 


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [RFC PATCH] tests/avocado: use new rootfs for orangepi test

[Daniel P. Berrangé]

On Wed, Nov 23, 2022 at 07:13:09PM +0100, Philippe Mathieu-Daudé wrote:
> On 23/11/22 15:12, Alex Bennée wrote:
> > Thomas Huth <thuth@redhat.com> writes:
> > > On 23/11/2022 12.15, Philippe Mathieu-Daudé wrote:
> > > > On 18/11/22 12:33, Alex Bennée wrote:
> > > > > The old URL wasn't stable. I suspect the current URL will only be
> > > > > stable for a few months so maybe we need another strategy for hosting
> > > > > rootfs snapshots?
> > > > > 
> > > > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > > > > ---
> > > > >    tests/avocado/boot_linux_console.py | 4 ++--
> > > > >    1 file changed, 2 insertions(+), 2 deletions(-)
> > > > > 
> > > > > diff --git a/tests/avocado/boot_linux_console.py
> > > > > b/tests/avocado/boot_linux_console.py
> > > > > index 4c9d551f47..5a2923c423 100644
> > > > > --- a/tests/avocado/boot_linux_console.py
> > > > > +++ b/tests/avocado/boot_linux_console.py
> > > > > @@ -793,8 +793,8 @@ def test_arm_orangepi_sd(self):
> > > > >            dtb_path =
> > > > > '/usr/lib/linux-image-current-sunxi/sun8i-h3-orangepi-pc.dtb'
> > > > >            dtb_path = self.extract_from_deb(deb_path, dtb_path)
> > > > >            rootfs_url =
> > > > > ('http://storage.kernelci.org/images/rootfs/buildroot/'
> > > > > -                      'kci-2019.02/armel/base/rootfs.ext2.xz')
> > > > > -        rootfs_hash = '692510cb625efda31640d1de0a8d60e26040f061'
> > > > > +                      'buildroot-baseline/20221116.0/armel/rootfs.ext2.xz')
> > > > > +        rootfs_hash = 'fae32f337c7b87547b10f42599acf109da8b6d9a'
> > > > If Avocado doesn't find an artifact in its local cache, it will fetch it
> > > > from the URL.
> > > > The cache might be populated with artifacts previously downloaded, but
> > > > their URL is not valid anymore (my case for many tests).
> > > > We can also add artifacts manually, see [1].
> > > > I'd rather keep pre-existing tests if possible, to test older
> > > > (kernel / user-space) images. We don't need to run all the tests all
> > > > the time:
> > > > tests can be filtered by tags (see [2]).
> > > > My preference here is to refactor this test, adding the
> > > > "kci-2019.02"
> > > > and "baseline-20221116.0" releases. I can prepare the patch if you /
> > > > Thomas don't object.
> > > 
> > > IMHO we shouldn't keep tests in the upstream git repository where the
> > > binaries are not available in public anymore. They won't get run by
> > > new contributors anymore, and also could vanish from the disks of the
> > > people who previously downloaded it, once they wipe their cache or
> > > upgrade to a new installation, so the test code will sooner or later
> > > be bitrotting. But if you want to keep the tests around on your hard
> > > disk, you could also stick the test in a local branch on your hard
> > > disk instead.
> > 
> > CI/Workstation splits aside I tend to agree with Thomas here that having
> > tests no one else can run will lead to an accretion of broken tests.
> 
> Following this idea, should we remove all boards for which no open
> source & GPL software is available? I.e:

No of course not, that's totally ridiculous.

Don't equate what scenarios we cover in CI, with what features
QEMU implements in code.

The CI coverage merely influences what we tell people about
the level of quality we can guarantee for respective features.


> > > The other possibility is to upload the binaries to a new public
> > > location in the web ... but for software that contains GPLed software,
> > > you should then also make sure to provide the source code to comply
> > > with the license.
> > 
> > This is the traditional reason we've lent so hard on external hosting
> > for binaries because the upstream doesn't want the hassle of maintaining
> > that sort of zoo of binaries. That said we have tests where binaries are
> > served from fileserver.linaro.org but its then only my problem to deal
> > with GPL requirements and not the upstream.
> 
> Maybe we are discussing 2 different topics. I am in possession of
> old Solaris installation CDROMs and could boot some of them with
> qemu-system-sparc. I want to automatize my testing, and wrote Avocado
> scripts doing that. I suppose other QEMU users have similar CDROMs.
> If I contribute my tests, they can run them. Isn't it in the interest
> of the community to have such examples and tests available?

Potentially yes, but remember that all code has a maintenance cost,
and we're massively struggling to keep even our current avocado
setup running reliably. The CI for Avocado is in a state of almost
constant brokeness throughout recent merges :-(

Given the instability and time invested chasing failures in current
tests we have, I don't think we can commit to adding arbitrary user
contributed tests to our gating CI.

I think we need to strictly focus our Avocado gating CI testing on
a small set of OS that give maximum value for our usersr. IMHO that
means primarily focus testing resources on modern non-EOL OS distros
and hardware platforms.

We could take tests for other old OS / old hardware platforms and
NOT run them in CI, but then they're at even greater risk of
bitrotting than the stuff we already struggle to keep running.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|