[PATCH v3 0/2] tests/avocado: updates for sbsa-ref testing

[PATCH v3 0/2] tests/avocado: updates for sbsa-ref testing

[Marcin Juszkiewicz]

We want to have some non-Linux OS in testing in case one of changes keep
Linux booting but crash elsewhere. So far OpenBSD was used for it but we
move to FreeBSD 14.x due to longer support cycles.

One OpenBSD stays - will be run on Cortex-A57 only. And only on local
runs.

At same time we add test to run on default cpu settings. For now it is
plain Neoverse-N2 but we are considering either disabling PAuth or going
with 'impdef' way.

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>

---
Changes in v3:
- kept OpenBSD/Cortex-A57 test for local runs (by request of Philippe)
- Link to v2: https://lore.kernel.org/r/20240624-b4-move-to-freebsd-v2-0-64ea7b04998f@linaro.org

---
Marcin Juszkiewicz (2):
      tests/avocado: sbsa-ref: add FreeBSD tests
      tests/avocado: add test for default sbsa-ref cpu

 tests/avocado/machine_aarch64_sbsaref.py | 90 +++++++++++++++++++++++---------
 1 file changed, 65 insertions(+), 25 deletions(-)
---
base-commit: c9ba79baca7c673098361e3a687f72d458e0d18a
change-id: 20240624-b4-move-to-freebsd-aafdeb75ef38

Best regards,
-- 
Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>

[PATCH v3 1/2] tests/avocado: sbsa-ref: add FreeBSD tests

[Marcin Juszkiewicz]

FreeBSD has longer support cycle for stable release (14.x EoL in 2028)
than OpenBSD (7.3 we used is already EoL). Also bugfixes are backported
so we can stay on 14.x for longer. Tests done on OpenBSD will now be
done using FreeBSD.

OpenBSD 7.3 stays with Cortex-A57 test for local runs only.

Moved from Neoverse-N1 to Neoverse-N2 as sbsa-ref defaults were changed.

Timeout messages expanded to mention being affected by PAuth emulation.

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
---
 tests/avocado/machine_aarch64_sbsaref.py | 75 ++++++++++++++++++++++++--------
 1 file changed, 56 insertions(+), 19 deletions(-)

diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index 6bb82f2a03..bb2c098aac 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -1,4 +1,4 @@
-# Functional test that boots a Linux kernel and checks the console
+# Functional test that boots a kernel and checks the console
 #
 # SPDX-FileCopyrightText: 2023-2024 Linaro Ltd.
 # SPDX-FileContributor: Philippe Mathieu-Daudé <philmd@linaro.org>
@@ -8,7 +8,7 @@
 
 import os
 
-from avocado import skipUnless
+from avocado import skipIf, skipUnless
 from avocado.utils import archive
 
 from avocado_qemu import QemuSystemTest
@@ -165,7 +165,8 @@ def test_sbsaref_alpine_linux_max_pauth_impdef(self):
         """
         self.boot_alpine_linux("max,pauth-impdef=on")
 
-    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'),
+                'Test might timeout due to PAuth emulation')
     def test_sbsaref_alpine_linux_max(self):
         """
         :avocado: tags=cpu:max
@@ -173,7 +174,6 @@ def test_sbsaref_alpine_linux_max(self):
         """
         self.boot_alpine_linux("max")
 
-
     # This tests the whole boot chain from EFI to Userspace
     # We only boot a whole OS for the current top level CPU and GIC
     # Other test profiles should use more minimal boots
@@ -200,6 +200,9 @@ def boot_openbsd73(self, cpu):
                                  "Welcome to the OpenBSD/arm64"
                                  " 7.3 installation program.")
 
+
+    # we keep OpenBSD 7.3 on Cortex-A57 by request
+    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
     def test_sbsaref_openbsd73_cortex_a57(self):
         """
         :avocado: tags=cpu:cortex-a57
@@ -207,32 +210,66 @@ def test_sbsaref_openbsd73_cortex_a57(self):
         """
         self.boot_openbsd73("cortex-a57")
 
-    def test_sbsaref_openbsd73_neoverse_n1(self):
+
+    # This tests the whole boot chain from EFI to Userspace
+    # We only boot a whole OS for the current top level CPU and GIC
+    # Other test profiles should use more minimal boots
+    def boot_freebsd(self, cpu):
+        self.fetch_firmware()
+
+        img_url = (
+            "https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/"
+            "14.1/FreeBSD-14.1-RELEASE-arm64-aarch64-bootonly.iso"
+        )
+
+        img_hash = "44cdbae275ef1bb6dab1d5fbb59473d4f741e1c8ea8a80fd9e906b531d6ad461"
+        img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash)
+
+        self.vm.set_console()
+        self.vm.add_args(
+            "-cpu",
+            cpu,
+            "-drive",
+            f"file={img_path},format=raw",
+        )
+
+        self.vm.launch()
+        wait_for_console_pattern(self, "Welcome to FreeBSD!")
+
+    def test_sbsaref_freebsd_cortex_a57(self):
         """
-        :avocado: tags=cpu:neoverse-n1
-        :avocado: tags=os:openbsd
+        :avocado: tags=cpu:cortex-a57
+        :avocado: tags=os:freebsd
+        """
+        self.boot_freebsd("cortex-a57")
+
+    # We use Neoverse-N2 as default cpu
+    def test_sbsaref_freebsd_neoverse_n2(self):
         """
-        self.boot_openbsd73("neoverse-n1")
+        :avocado: tags=cpu:neoverse-n2
+        :avocado: tags=os:freebsd
+        """
+        self.boot_freebsd("neoverse-n2")
 
-    def test_sbsaref_openbsd73_max_pauth_off(self):
+    def test_sbsaref_freebsd_max_pauth_off(self):
         """
         :avocado: tags=cpu:max
-        :avocado: tags=os:openbsd
+        :avocado: tags=os:freebsd
         """
-        self.boot_openbsd73("max,pauth=off")
+        self.boot_freebsd("max,pauth=off")
 
-    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
-    def test_sbsaref_openbsd73_max_pauth_impdef(self):
+    def test_sbsaref_freebsd_max_pauth_impdef(self):
         """
         :avocado: tags=cpu:max
-        :avocado: tags=os:openbsd
+        :avocado: tags=os:freebsd
         """
-        self.boot_openbsd73("max,pauth-impdef=on")
+        self.boot_freebsd("max,pauth-impdef=on")
 
-    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
-    def test_sbsaref_openbsd73_max(self):
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'),
+                'Test might timeout due to PAuth emulation')
+    def test_sbsaref_freebsd_max(self):
         """
         :avocado: tags=cpu:max
-        :avocado: tags=os:openbsd
+        :avocado: tags=os:freebsd
         """
-        self.boot_openbsd73("max")
+        self.boot_freebsd("max")

-- 
2.45.2

[PATCH v3 2/2] tests/avocado: add test for default sbsa-ref cpu

[Marcin Juszkiewicz]

We changed sbsa-ref cpu several times already and may do it again in a
future. To newer core or to enable/disable some properties.

This change switches Neoverse-N2 tests to 'let test default cpu' ones.

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
---
 tests/avocado/machine_aarch64_sbsaref.py | 29 ++++++++++++++++-------------
 1 file changed, 16 insertions(+), 13 deletions(-)

diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index bb2c098aac..43bb1860e0 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -115,7 +115,7 @@ def test_sbsaref_edk2_firmware(self):
     # This tests the whole boot chain from EFI to Userspace
     # We only boot a whole OS for the current top level CPU and GIC
     # Other test profiles should use more minimal boots
-    def boot_alpine_linux(self, cpu):
+    def boot_alpine_linux(self, cpu=False):
         self.fetch_firmware()
 
         iso_url = (
@@ -128,12 +128,14 @@ def boot_alpine_linux(self, cpu):
 
         self.vm.set_console()
         self.vm.add_args(
-            "-cpu",
-            cpu,
             "-drive",
             f"file={iso_path},format=raw",
         )
 
+        # let allow test which will use default cpu of platform
+        if cpu:
+            self.vm.add_args("-cpu", cpu)
+
         self.vm.launch()
         wait_for_console_pattern(self, "Welcome to Alpine Linux 3.17")
 
@@ -144,12 +146,12 @@ def test_sbsaref_alpine_linux_cortex_a57(self):
         """
         self.boot_alpine_linux("cortex-a57")
 
-    def test_sbsaref_alpine_linux_neoverse_n1(self):
+    # Let test whichever cpu is used as default
+    def test_sbsaref_alpine_linux_default(self):
         """
-        :avocado: tags=cpu:neoverse-n1
         :avocado: tags=os:linux
         """
-        self.boot_alpine_linux("neoverse-n1")
+        self.boot_alpine_linux()
 
     def test_sbsaref_alpine_linux_max_pauth_off(self):
         """
@@ -214,7 +216,7 @@ def test_sbsaref_openbsd73_cortex_a57(self):
     # This tests the whole boot chain from EFI to Userspace
     # We only boot a whole OS for the current top level CPU and GIC
     # Other test profiles should use more minimal boots
-    def boot_freebsd(self, cpu):
+    def boot_freebsd(self, cpu=False):
         self.fetch_firmware()
 
         img_url = (
@@ -227,12 +229,14 @@ def boot_freebsd(self, cpu):
 
         self.vm.set_console()
         self.vm.add_args(
-            "-cpu",
-            cpu,
             "-drive",
             f"file={img_path},format=raw",
         )
 
+        # let allow test which will use default cpu of platform
+        if cpu:
+            self.vm.add_args("-cpu", cpu)
+
         self.vm.launch()
         wait_for_console_pattern(self, "Welcome to FreeBSD!")
 
@@ -243,13 +247,12 @@ def test_sbsaref_freebsd_cortex_a57(self):
         """
         self.boot_freebsd("cortex-a57")
 
-    # We use Neoverse-N2 as default cpu
-    def test_sbsaref_freebsd_neoverse_n2(self):
+    # Let test whichever cpu is used as default
+    def test_sbsaref_freebsd_default(self):
         """
-        :avocado: tags=cpu:neoverse-n2
         :avocado: tags=os:freebsd
         """
-        self.boot_freebsd("neoverse-n2")
+        self.boot_freebsd()
 
     def test_sbsaref_freebsd_max_pauth_off(self):
         """

-- 
2.45.2

Re: [PATCH v3 1/2] tests/avocado: sbsa-ref: add FreeBSD tests

[Peter Maydell]

On Mon, 24 Jun 2024 at 17:20, Marcin Juszkiewicz
<marcin.juszkiewicz@linaro.org> wrote:
>
> FreeBSD has longer support cycle for stable release (14.x EoL in 2028)
> than OpenBSD (7.3 we used is already EoL). Also bugfixes are backported
> so we can stay on 14.x for longer. Tests done on OpenBSD will now be
> done using FreeBSD.
>
> OpenBSD 7.3 stays with Cortex-A57 test for local runs only.
>
> Moved from Neoverse-N1 to Neoverse-N2 as sbsa-ref defaults were changed.
>
> Timeout messages expanded to mention being affected by PAuth emulation.
>
> Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
> ---
>  tests/avocado/machine_aarch64_sbsaref.py | 75 ++++++++++++++++++++++++--------
>  1 file changed, 56 insertions(+), 19 deletions(-)
>
> diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
> index 6bb82f2a03..bb2c098aac 100644
> --- a/tests/avocado/machine_aarch64_sbsaref.py
> +++ b/tests/avocado/machine_aarch64_sbsaref.py
> @@ -1,4 +1,4 @@
> -# Functional test that boots a Linux kernel and checks the console
> +# Functional test that boots a kernel and checks the console
>  #
>  # SPDX-FileCopyrightText: 2023-2024 Linaro Ltd.
>  # SPDX-FileContributor: Philippe Mathieu-Daudé <philmd@linaro.org>
> @@ -8,7 +8,7 @@
>
>  import os
>
> -from avocado import skipUnless
> +from avocado import skipIf, skipUnless
>  from avocado.utils import archive
>
>  from avocado_qemu import QemuSystemTest
> @@ -165,7 +165,8 @@ def test_sbsaref_alpine_linux_max_pauth_impdef(self):
>          """
>          self.boot_alpine_linux("max,pauth-impdef=on")
>
> -    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
> +    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'),
> +                'Test might timeout due to PAuth emulation')
>      def test_sbsaref_alpine_linux_max(self):
>          """
>          :avocado: tags=cpu:max
> @@ -173,7 +174,6 @@ def test_sbsaref_alpine_linux_max(self):
>          """
>          self.boot_alpine_linux("max")
>
> -
>      # This tests the whole boot chain from EFI to Userspace
>      # We only boot a whole OS for the current top level CPU and GIC
>      # Other test profiles should use more minimal boots
> @@ -200,6 +200,9 @@ def boot_openbsd73(self, cpu):
>                                   "Welcome to the OpenBSD/arm64"
>                                   " 7.3 installation program.")
>
> +
> +    # we keep OpenBSD 7.3 on Cortex-A57 by request
> +    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')

Don't skip stuff if GITLAB_CI, please. If we don't want to
run it on the CI then we shouldn't be running it by default
when a local user runs check-avocado either.

Any skip should have a clear description of the reason why
we're skipping it.

thanks
-- PMM

Re: [PATCH v3 1/2] tests/avocado: sbsa-ref: add FreeBSD tests

[Philippe Mathieu-Daudé]

On 1/7/24 15:37, Peter Maydell wrote:
> On Mon, 24 Jun 2024 at 17:20, Marcin Juszkiewicz
> <marcin.juszkiewicz@linaro.org> wrote:
>>
>> FreeBSD has longer support cycle for stable release (14.x EoL in 2028)
>> than OpenBSD (7.3 we used is already EoL). Also bugfixes are backported
>> so we can stay on 14.x for longer. Tests done on OpenBSD will now be
>> done using FreeBSD.
>>
>> OpenBSD 7.3 stays with Cortex-A57 test for local runs only.
>>
>> Moved from Neoverse-N1 to Neoverse-N2 as sbsa-ref defaults were changed.
>>
>> Timeout messages expanded to mention being affected by PAuth emulation.
>>
>> Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
>> ---
>>   tests/avocado/machine_aarch64_sbsaref.py | 75 ++++++++++++++++++++++++--------
>>   1 file changed, 56 insertions(+), 19 deletions(-)
>>
>> diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
>> index 6bb82f2a03..bb2c098aac 100644
>> --- a/tests/avocado/machine_aarch64_sbsaref.py
>> +++ b/tests/avocado/machine_aarch64_sbsaref.py
>> @@ -1,4 +1,4 @@
>> -# Functional test that boots a Linux kernel and checks the console
>> +# Functional test that boots a kernel and checks the console
>>   #
>>   # SPDX-FileCopyrightText: 2023-2024 Linaro Ltd.
>>   # SPDX-FileContributor: Philippe Mathieu-Daudé <philmd@linaro.org>
>> @@ -8,7 +8,7 @@
>>
>>   import os
>>
>> -from avocado import skipUnless
>> +from avocado import skipIf, skipUnless
>>   from avocado.utils import archive
>>
>>   from avocado_qemu import QemuSystemTest
>> @@ -165,7 +165,8 @@ def test_sbsaref_alpine_linux_max_pauth_impdef(self):
>>           """
>>           self.boot_alpine_linux("max,pauth-impdef=on")
>>
>> -    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
>> +    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'),
>> +                'Test might timeout due to PAuth emulation')
>>       def test_sbsaref_alpine_linux_max(self):
>>           """
>>           :avocado: tags=cpu:max
>> @@ -173,7 +174,6 @@ def test_sbsaref_alpine_linux_max(self):
>>           """
>>           self.boot_alpine_linux("max")
>>
>> -
>>       # This tests the whole boot chain from EFI to Userspace
>>       # We only boot a whole OS for the current top level CPU and GIC
>>       # Other test profiles should use more minimal boots
>> @@ -200,6 +200,9 @@ def boot_openbsd73(self, cpu):
>>                                    "Welcome to the OpenBSD/arm64"
>>                                    " 7.3 installation program.")
>>
>> +
>> +    # we keep OpenBSD 7.3 on Cortex-A57 by request
>> +    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
> 
> Don't skip stuff if GITLAB_CI, please. If we don't want to
> run it on the CI then we shouldn't be running it by default
> when a local user runs check-avocado either.

I did request that.

> Any skip should have a clear description of the reason why
> we're skipping it.

Right. Maybe replace by:

   @skipUnless(os.getenv('AVOCADO_TEST_LEGACY_OS'), 'OpenBSD 7.3 is EoL')

Or AVOCADO_TEST_LEGACY_OPENBSD... Naming is hard.

Regards,

Phil.