[PATCH v2 0/4] tests/avocado: update sbsa-ref firmware to latest

[PATCH v2 0/4] tests/avocado: update sbsa-ref firmware to latest

[Marcin Juszkiewicz]

Updating sbsa-ref firmware for QEMU CI was manual task. Now it is
replaced by CI job run on CodeLinaro Gitlab instance.

This patchset updates to current state:

- Trusted Firmware v2.10.2 (latest LTS)
- Tianocore EDK2 stable202402 (latest release)

And Tianocore EDK2-platforms commit 085c2fb (edk2-platforms does not
have releases).

Firmware images were built using Debian 'bookworm' cross gcc 12.2.0
compiler.

And while I am in that file I dropped use of 'virtio-rng-pci' device as
sbsa-ref is supposed to emulate physical hardware.

OpenBSD updated to 7.4 version.

(1/8) test_sbsaref_edk2_firmware: PASS (2.49 s)
(2/8) test_sbsaref_alpine_linux_cortex_a57: PASS (23.78 s)
(3/8) test_sbsaref_alpine_linux_neoverse_n1: PASS (23.38 s)
(4/8) test_sbsaref_alpine_linux_max_pauth_off: PASS (23.31 s)
(5/8) test_sbsaref_alpine_linux_max_pauth_impdef: PASS (29.27 s)
(6/8) test_sbsaref_alpine_linux_max: PASS (80.13 s)
(7/8) test_sbsaref_openbsd_cortex_a57: PASS (16.01 s)
(8/8) test_sbsaref_openbsd_neoverse_n1: PASS (16.05 s)

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
---
Changes in v2:
- disabled 'max' tests on OpenBSD
- moved tags to 'one tag per line'
- added 'os:linux' tags to Alpine ones
- Link to v1: https://lore.kernel.org/r/20240313-sbsa-ref-firmware-update-v1-0-e166703c5424@linaro.org

---
Marcin Juszkiewicz (4):
      tests/avocado: update sbsa-ref firmware
      tests/avocado: drop virtio-rng from sbsa-ref tests
      tests/avocado: use OpenBSD 7.4 for sbsa-ref
      tests/avocado: sbsa-ref: add Alpine tests for misc 'max' setup

 tests/avocado/machine_aarch64_sbsaref.py | 113 ++++++++++++++++++++-----------
 1 file changed, 73 insertions(+), 40 deletions(-)
---
base-commit: 0748129684be2773117b0b8fc3c60161abdb7bb8
change-id: 20240313-sbsa-ref-firmware-update-7579d9f6d59b

Best regards,
-- 
Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>

[PATCH v2 1/4] tests/avocado: update sbsa-ref firmware

[Marcin Juszkiewicz]

We now have CI job to build those and publish in space with
readable urls.

Firmware is built using Debian 'bookworm' cross toolchain (gcc 12.2.0).

Used versions:

- Trusted Firmware v2.10.2
- Tianocore EDK2 stable202402
- Tianocore EDK2 Platforms code commit 085c2fb

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
---
 tests/avocado/machine_aarch64_sbsaref.py | 40 +++++++++++++++++---------------
 1 file changed, 21 insertions(+), 19 deletions(-)

diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index 528c7d2934..cbab793455 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -1,6 +1,6 @@
 # Functional test that boots a Linux kernel and checks the console
 #
-# SPDX-FileCopyrightText: 2023 Linaro Ltd.
+# SPDX-FileCopyrightText: 2023-2024 Linaro Ltd.
 # SPDX-FileContributor: Philippe Mathieu-Daudé <philmd@linaro.org>
 # SPDX-FileContributor: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
 #
@@ -32,34 +32,36 @@ def fetch_firmware(self):
         """
         Flash volumes generated using:
 
-        - Fedora GNU Toolchain version 13.2.1 20230728 (Red Hat 13.2.1-1)
+        Toolchain from Debian:
+        aarch64-linux-gnu-gcc (Debian 12.2.0-14) 12.2.0
 
-        - Trusted Firmware-A
-          https://github.com/ARM-software/arm-trusted-firmware/tree/7c3ff62d
+        Used components:
+
+        - Trusted Firmware 2.10.2
+        - Tianocore EDK2 stable202402
+        - Tianocore EDK2-platforms commit 085c2fb
 
-        - Tianocore EDK II
-          https://github.com/tianocore/edk2/tree/0f9283429dd4
-          https://github.com/tianocore/edk2/tree/ad1c0394b177
-          https://github.com/tianocore/edk2-platforms/tree/d03a60523a60
         """
 
         # Secure BootRom (TF-A code)
         fs0_xz_url = (
-            "https://fileserver.linaro.org/s/rE43RJyTfxPtBkc/"
-            "download/SBSA_FLASH0.fd.xz"
+            "https://artifacts.codelinaro.org/artifactory/linaro-419-sbsa-ref/"
+            "20240313-116475/edk2/SBSA_FLASH0.fd.xz"
         )
-        fs0_xz_hash = "cdb8e4ffdaaa79292b7b465693f9e5fae6b7062d"
-        tar_xz_path = self.fetch_asset(fs0_xz_url, asset_hash=fs0_xz_hash)
+        fs0_xz_hash = "637593749cc307dea7dc13265c32e5d020267552f22b18a31850b8429fc5e159"
+        tar_xz_path = self.fetch_asset(fs0_xz_url, asset_hash=fs0_xz_hash,
+                                      algorithm='sha256')
         archive.extract(tar_xz_path, self.workdir)
         fs0_path = os.path.join(self.workdir, "SBSA_FLASH0.fd")
 
         # Non-secure rom (UEFI and EFI variables)
         fs1_xz_url = (
-            "https://fileserver.linaro.org/s/AGWPDXbcqJTKS4R/"
-            "download/SBSA_FLASH1.fd.xz"
+            "https://artifacts.codelinaro.org/artifactory/linaro-419-sbsa-ref/"
+            "20240313-116475/edk2/SBSA_FLASH1.fd.xz"
         )
-        fs1_xz_hash = "411155ae6984334714dff08d5d628178e790c875"
-        tar_xz_path = self.fetch_asset(fs1_xz_url, asset_hash=fs1_xz_hash)
+        fs1_xz_hash = "cb0a5e8cf5e303c5d3dc106cfd5943ffe9714b86afddee7164c69ee1dd41991c"
+        tar_xz_path = self.fetch_asset(fs1_xz_url, asset_hash=fs1_xz_hash,
+                                      algorithm='sha256')
         archive.extract(tar_xz_path, self.workdir)
         fs1_path = os.path.join(self.workdir, "SBSA_FLASH1.fd")
 
@@ -96,15 +98,15 @@ def test_sbsaref_edk2_firmware(self):
 
         # AP Trusted ROM
         wait_for_console_pattern(self, "Booting Trusted Firmware")
-        wait_for_console_pattern(self, "BL1: v2.9(release):v2.9")
+        wait_for_console_pattern(self, "BL1: v2.10.2(release):")
         wait_for_console_pattern(self, "BL1: Booting BL2")
 
         # Trusted Boot Firmware
-        wait_for_console_pattern(self, "BL2: v2.9(release)")
+        wait_for_console_pattern(self, "BL2: v2.10.2(release)")
         wait_for_console_pattern(self, "Booting BL31")
 
         # EL3 Runtime Software
-        wait_for_console_pattern(self, "BL31: v2.9(release)")
+        wait_for_console_pattern(self, "BL31: v2.10.2(release)")
 
         # Non-trusted Firmware
         wait_for_console_pattern(self, "UEFI firmware (version 1.0")

-- 
2.44.0

[PATCH v2 2/4] tests/avocado: drop virtio-rng from sbsa-ref tests

[Marcin Juszkiewicz]

sbsa-ref is supposed to emulate real hardware so virtio-rng-pci
does not fit here

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
---
 tests/avocado/machine_aarch64_sbsaref.py | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index cbab793455..259225f15f 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -132,10 +132,6 @@ def boot_alpine_linux(self, cpu):
             cpu,
             "-drive",
             f"file={iso_path},format=raw",
-            "-device",
-            "virtio-rng-pci,rng=rng0",
-            "-object",
-            "rng-random,id=rng0,filename=/dev/urandom",
         )
 
         self.vm.launch()
@@ -179,10 +175,6 @@ def boot_openbsd73(self, cpu):
             cpu,
             "-drive",
             f"file={img_path},format=raw",
-            "-device",
-            "virtio-rng-pci,rng=rng0",
-            "-object",
-            "rng-random,id=rng0,filename=/dev/urandom",
         )
 
         self.vm.launch()

-- 
2.44.0

Re: [PATCH v2 2/4] tests/avocado: drop virtio-rng from sbsa-ref tests

[Alex Bennée]

Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> writes:

> sbsa-ref is supposed to emulate real hardware so virtio-rng-pci
> does not fit here

What is real anyway ;-)

VirtIO devices exist in real life and I would argue for PCI the device
appears very much like a normal PCI device. I could see the argument for
avoiding virtio-mmio devices which are a lot more VM-only in flavour.

>
> Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
> ---
>  tests/avocado/machine_aarch64_sbsaref.py | 8 --------
>  1 file changed, 8 deletions(-)
>
> diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
> index cbab793455..259225f15f 100644
> --- a/tests/avocado/machine_aarch64_sbsaref.py
> +++ b/tests/avocado/machine_aarch64_sbsaref.py
> @@ -132,10 +132,6 @@ def boot_alpine_linux(self, cpu):
>              cpu,
>              "-drive",
>              f"file={iso_path},format=raw",
> -            "-device",
> -            "virtio-rng-pci,rng=rng0",
> -            "-object",
> -            "rng-random,id=rng0,filename=/dev/urandom",
>          )
>  
>          self.vm.launch()
> @@ -179,10 +175,6 @@ def boot_openbsd73(self, cpu):
>              cpu,
>              "-drive",
>              f"file={img_path},format=raw",
> -            "-device",
> -            "virtio-rng-pci,rng=rng0",
> -            "-object",
> -            "rng-random,id=rng0,filename=/dev/urandom",
>          )
>  
>          self.vm.launch()

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v2 2/4] tests/avocado: drop virtio-rng from sbsa-ref tests

[Peter Maydell]

On Thu, 14 Mar 2024 at 12:11, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> writes:
>
> > sbsa-ref is supposed to emulate real hardware so virtio-rng-pci
> > does not fit here
>
> What is real anyway ;-)
>
> VirtIO devices exist in real life and I would argue for PCI the device
> appears very much like a normal PCI device. I could see the argument for
> avoiding virtio-mmio devices which are a lot more VM-only in flavour.

But our platform for testing virtio-rng-pci should be the 'virt'
board (and our avocado test for virt does indeed have that device).
What exactly are we adding by testing it also in sbsa-ref,
where the typical user will *not* be using it (if they wanted
a virtio setup they would be using 'virt')?

thanks
-- PMM

Re: [PATCH v2 2/4] tests/avocado: drop virtio-rng from sbsa-ref tests

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Thu, 14 Mar 2024 at 12:11, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> writes:
>>
>> > sbsa-ref is supposed to emulate real hardware so virtio-rng-pci
>> > does not fit here
>>
>> What is real anyway ;-)
>>
>> VirtIO devices exist in real life and I would argue for PCI the device
>> appears very much like a normal PCI device. I could see the argument for
>> avoiding virtio-mmio devices which are a lot more VM-only in flavour.
>
> But our platform for testing virtio-rng-pci should be the 'virt'
> board (and our avocado test for virt does indeed have that device).
> What exactly are we adding by testing it also in sbsa-ref,
> where the typical user will *not* be using it (if they wanted
> a virtio setup they would be using 'virt')?

I assume just ensuring the board has enough entropy to boot cleanly. If
it doesn't need it then it doesn't matter I guess.

>
> thanks
> -- PMM

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

[PATCH v2 3/4] tests/avocado: use OpenBSD 7.4 for sbsa-ref

[Marcin Juszkiewicz]

7.4 was released in October 2023, time to update before 7.3 gets dropped.

Disabled tests for 'max' cpu as OpenBSD fails there.

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
---
 tests/avocado/machine_aarch64_sbsaref.py | 47 +++++++++++++++++++++++---------
 1 file changed, 34 insertions(+), 13 deletions(-)

diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index 259225f15f..0e52dc5854 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -159,14 +159,14 @@ def test_sbsaref_alpine_linux_max(self):
     # This tests the whole boot chain from EFI to Userspace
     # We only boot a whole OS for the current top level CPU and GIC
     # Other test profiles should use more minimal boots
-    def boot_openbsd73(self, cpu):
+    def boot_openbsd(self, cpu):
         self.fetch_firmware()
 
         img_url = (
-            "https://cdn.openbsd.org/pub/OpenBSD/7.3/arm64/miniroot73.img"
+            "https://cdn.openbsd.org/pub/OpenBSD/7.4/arm64/miniroot74.img"
         )
 
-        img_hash = "7fc2c75401d6f01fbfa25f4953f72ad7d7c18650056d30755c44b9c129b707e5"
+        img_hash = "7b08b2ce081cff6408d183f7152ddcfd2779912104866e4fdf6ae2d864b51142"
         img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash)
 
         self.vm.set_console()
@@ -180,23 +180,44 @@ def boot_openbsd73(self, cpu):
         self.vm.launch()
         wait_for_console_pattern(self,
                                  "Welcome to the OpenBSD/arm64"
-                                 " 7.3 installation program.")
+                                 " 7.4 installation program.")
 
-    def test_sbsaref_openbsd73_cortex_a57(self):
+    def test_sbsaref_openbsd_cortex_a57(self):
         """
         :avocado: tags=cpu:cortex-a57
+        :avocado: tags=os:openbsd
         """
-        self.boot_openbsd73("cortex-a57")
+        self.boot_openbsd("cortex-a57")
 
-    def test_sbsaref_openbsd73_neoverse_n1(self):
+    def test_sbsaref_openbsd_neoverse_n1(self):
         """
         :avocado: tags=cpu:neoverse-n1
+        :avocado: tags=os:openbsd
         """
-        self.boot_openbsd73("neoverse-n1")
+        self.boot_openbsd("neoverse-n1")
 
-    def test_sbsaref_openbsd73_max(self):
-        """
-        :avocado: tags=cpu:max
-        """
-        self.boot_openbsd73("max")
+# OpenBSD 7.4 does not boot on current max cpu.
+#
+#   def test_sbsaref_openbsd_max_pauth_off(self):
+#       """
+#       :avocado: tags=cpu:max
+#       :avocado: tags=os:openbsd
+#       """
+#       self.boot_openbsd("max,pauth=off")
+
+#   @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
+#   def test_sbsaref_openbsd_max_pauth_impdef(self):
+#       """
+#       :avocado: tags=cpu:max
+#       :avocado: tags=os:openbsd
+#       """
+#       self.boot_openbsd("max,pauth-impdef=on")
+
+#   @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
+#   def test_sbsaref_openbsd_max(self):
+#       """
+#       :avocado: tags=cpu:max
+#       :avocado: tags=os:openbsd
+#       """
+#       self.boot_openbsd("max")
 

-- 
2.44.0

Re: [PATCH v2 3/4] tests/avocado: use OpenBSD 7.4 for sbsa-ref

[Alex Bennée]

Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> writes:

> 7.4 was released in October 2023, time to update before 7.3 gets dropped.
>
> Disabled tests for 'max' cpu as OpenBSD fails there.
>
> Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
> ---
>  tests/avocado/machine_aarch64_sbsaref.py | 47 +++++++++++++++++++++++---------
>  1 file changed, 34 insertions(+), 13 deletions(-)
>
> diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
> index 259225f15f..0e52dc5854 100644
> --- a/tests/avocado/machine_aarch64_sbsaref.py
> +++ b/tests/avocado/machine_aarch64_sbsaref.py
> @@ -159,14 +159,14 @@ def test_sbsaref_alpine_linux_max(self):
>      # This tests the whole boot chain from EFI to Userspace
>      # We only boot a whole OS for the current top level CPU and GIC
>      # Other test profiles should use more minimal boots
> -    def boot_openbsd73(self, cpu):
> +    def boot_openbsd(self, cpu):
>          self.fetch_firmware()
>  
>          img_url = (
> -            "https://cdn.openbsd.org/pub/OpenBSD/7.3/arm64/miniroot73.img"
> +            "https://cdn.openbsd.org/pub/OpenBSD/7.4/arm64/miniroot74.img"
>          )
>  
> -        img_hash = "7fc2c75401d6f01fbfa25f4953f72ad7d7c18650056d30755c44b9c129b707e5"
> +        img_hash = "7b08b2ce081cff6408d183f7152ddcfd2779912104866e4fdf6ae2d864b51142"
>          img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash)
>  
>          self.vm.set_console()
> @@ -180,23 +180,44 @@ def boot_openbsd73(self, cpu):
>          self.vm.launch()
>          wait_for_console_pattern(self,
>                                   "Welcome to the OpenBSD/arm64"
> -                                 " 7.3 installation program.")
> +                                 " 7.4 installation program.")
>  
> -    def test_sbsaref_openbsd73_cortex_a57(self):
> +    def test_sbsaref_openbsd_cortex_a57(self):
>          """
>          :avocado: tags=cpu:cortex-a57
> +        :avocado: tags=os:openbsd
>          """
> -        self.boot_openbsd73("cortex-a57")
> +        self.boot_openbsd("cortex-a57")
>  
> -    def test_sbsaref_openbsd73_neoverse_n1(self):
> +    def test_sbsaref_openbsd_neoverse_n1(self):
>          """
>          :avocado: tags=cpu:neoverse-n1
> +        :avocado: tags=os:openbsd
>          """
> -        self.boot_openbsd73("neoverse-n1")
> +        self.boot_openbsd("neoverse-n1")
>  
> -    def test_sbsaref_openbsd73_max(self):
> -        """
> -        :avocado: tags=cpu:max
> -        """
> -        self.boot_openbsd73("max")
> +# OpenBSD 7.4 does not boot on current max cpu.
> +#
> +#   def test_sbsaref_openbsd_max_pauth_off(self):
> +#       """
> +#       :avocado: tags=cpu:max
> +#       :avocado: tags=os:openbsd
> +#       """
> +#       self.boot_openbsd("max,pauth=off")

If we are not going to delete the entries then at least use a @skip
instead of commenting. Maybe:

  @skip("Potential un-diagnosed upstream bug?")

but it would be nice to figure out exactly where is breaks.

> +
> +#   @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
> +#   def test_sbsaref_openbsd_max_pauth_impdef(self):
> +#       """
> +#       :avocado: tags=cpu:max
> +#       :avocado: tags=os:openbsd
> +#       """
> +#       self.boot_openbsd("max,pauth-impdef=on")
> +
> +#   @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
> +#   def test_sbsaref_openbsd_max(self):
> +#       """
> +#       :avocado: tags=cpu:max
> +#       :avocado: tags=os:openbsd
> +#       """
> +#       self.boot_openbsd("max")

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v2 3/4] tests/avocado: use OpenBSD 7.4 for sbsa-ref

[Marcin Juszkiewicz]

W dniu 14.03.2024 o 13:14, Alex Bennée pisze:
>> +# OpenBSD 7.4 does not boot on current max cpu.
>> +#
>> +#   def test_sbsaref_openbsd_max_pauth_off(self):
>> +#       """
>> +#       :avocado: tags=cpu:max
>> +#       :avocado: tags=os:openbsd
>> +#       """
>> +#       self.boot_openbsd("max,pauth=off")
> If we are not going to delete the entries then at least use a @skip
> instead of commenting. Maybe:
> 
>    @skip("Potential un-diagnosed upstream bug?")
> 
> but it would be nice to figure out exactly where is breaks.

I am going to subscribe to openbsd mailing list and ask there.

OpenBSD 7.3 works, 7.4/7.5-current does not.
And it is on Neoverse-V1/N2 and max cpu types.

Re: [PATCH v2 3/4] tests/avocado: use OpenBSD 7.4 for sbsa-ref

[Philippe Mathieu-Daudé]

On 14/3/24 13:14, Alex Bennée wrote:
> Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> writes:
> 
>> 7.4 was released in October 2023, time to update before 7.3 gets dropped.
>>
>> Disabled tests for 'max' cpu as OpenBSD fails there.
>>
>> Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
>> ---
>>   tests/avocado/machine_aarch64_sbsaref.py | 47 +++++++++++++++++++++++---------
>>   1 file changed, 34 insertions(+), 13 deletions(-)
>>
>> diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
>> index 259225f15f..0e52dc5854 100644
>> --- a/tests/avocado/machine_aarch64_sbsaref.py
>> +++ b/tests/avocado/machine_aarch64_sbsaref.py
>> @@ -159,14 +159,14 @@ def test_sbsaref_alpine_linux_max(self):
>>       # This tests the whole boot chain from EFI to Userspace
>>       # We only boot a whole OS for the current top level CPU and GIC
>>       # Other test profiles should use more minimal boots
>> -    def boot_openbsd73(self, cpu):
>> +    def boot_openbsd(self, cpu):
>>           self.fetch_firmware()
>>   
>>           img_url = (
>> -            "https://cdn.openbsd.org/pub/OpenBSD/7.3/arm64/miniroot73.img"
>> +            "https://cdn.openbsd.org/pub/OpenBSD/7.4/arm64/miniroot74.img"
>>           )
>>   
>> -        img_hash = "7fc2c75401d6f01fbfa25f4953f72ad7d7c18650056d30755c44b9c129b707e5"
>> +        img_hash = "7b08b2ce081cff6408d183f7152ddcfd2779912104866e4fdf6ae2d864b51142"
>>           img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash)
>>   
>>           self.vm.set_console()
>> @@ -180,23 +180,44 @@ def boot_openbsd73(self, cpu):
>>           self.vm.launch()
>>           wait_for_console_pattern(self,
>>                                    "Welcome to the OpenBSD/arm64"
>> -                                 " 7.3 installation program.")
>> +                                 " 7.4 installation program.")
>>   
>> -    def test_sbsaref_openbsd73_cortex_a57(self):
>> +    def test_sbsaref_openbsd_cortex_a57(self):
>>           """
>>           :avocado: tags=cpu:cortex-a57
>> +        :avocado: tags=os:openbsd
>>           """
>> -        self.boot_openbsd73("cortex-a57")
>> +        self.boot_openbsd("cortex-a57")
>>   
>> -    def test_sbsaref_openbsd73_neoverse_n1(self):
>> +    def test_sbsaref_openbsd_neoverse_n1(self):
>>           """
>>           :avocado: tags=cpu:neoverse-n1
>> +        :avocado: tags=os:openbsd
>>           """
>> -        self.boot_openbsd73("neoverse-n1")
>> +        self.boot_openbsd("neoverse-n1")
>>   
>> -    def test_sbsaref_openbsd73_max(self):
>> -        """
>> -        :avocado: tags=cpu:max
>> -        """
>> -        self.boot_openbsd73("max")
>> +# OpenBSD 7.4 does not boot on current max cpu.
>> +#
>> +#   def test_sbsaref_openbsd_max_pauth_off(self):
>> +#       """
>> +#       :avocado: tags=cpu:max
>> +#       :avocado: tags=os:openbsd
>> +#       """
>> +#       self.boot_openbsd("max,pauth=off")
> 
> If we are not going to delete the entries then at least use a @skip
> instead of commenting. Maybe:
> 
>    @skip("Potential un-diagnosed upstream bug?")

Daniel or Peter suggested to open a GitLab issue and use

     @skip("https://gitlab.com/qemu-project/qemu/-/issues/xyz")

to track progress.

> but it would be nice to figure out exactly where is breaks.

Re: [PATCH v2 3/4] tests/avocado: use OpenBSD 7.4 for sbsa-ref

[Alex Bennée]

Philippe Mathieu-Daudé <philmd@linaro.org> writes:

> On 14/3/24 13:14, Alex Bennée wrote:
>> Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org> writes:
>> 
>>> 7.4 was released in October 2023, time to update before 7.3 gets dropped.
>>>
>>> Disabled tests for 'max' cpu as OpenBSD fails there.
>>>
>>> Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
>>> ---
>>>   tests/avocado/machine_aarch64_sbsaref.py | 47 +++++++++++++++++++++++---------
>>>   1 file changed, 34 insertions(+), 13 deletions(-)
>>>
>>> diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
>>> index 259225f15f..0e52dc5854 100644
>>> --- a/tests/avocado/machine_aarch64_sbsaref.py
>>> +++ b/tests/avocado/machine_aarch64_sbsaref.py
>>> @@ -159,14 +159,14 @@ def test_sbsaref_alpine_linux_max(self):
>>>       # This tests the whole boot chain from EFI to Userspace
>>>       # We only boot a whole OS for the current top level CPU and GIC
>>>       # Other test profiles should use more minimal boots
>>> -    def boot_openbsd73(self, cpu):
>>> +    def boot_openbsd(self, cpu):
>>>           self.fetch_firmware()
>>>             img_url = (
>>> -            "https://cdn.openbsd.org/pub/OpenBSD/7.3/arm64/miniroot73.img"
>>> +            "https://cdn.openbsd.org/pub/OpenBSD/7.4/arm64/miniroot74.img"
>>>           )
>>>   -        img_hash =
>>> "7fc2c75401d6f01fbfa25f4953f72ad7d7c18650056d30755c44b9c129b707e5"
>>> +        img_hash = "7b08b2ce081cff6408d183f7152ddcfd2779912104866e4fdf6ae2d864b51142"
>>>           img_path = self.fetch_asset(img_url, algorithm="sha256", asset_hash=img_hash)
>>>             self.vm.set_console()
>>> @@ -180,23 +180,44 @@ def boot_openbsd73(self, cpu):
>>>           self.vm.launch()
>>>           wait_for_console_pattern(self,
>>>                                    "Welcome to the OpenBSD/arm64"
>>> -                                 " 7.3 installation program.")
>>> +                                 " 7.4 installation program.")
>>>   -    def test_sbsaref_openbsd73_cortex_a57(self):
>>> +    def test_sbsaref_openbsd_cortex_a57(self):
>>>           """
>>>           :avocado: tags=cpu:cortex-a57
>>> +        :avocado: tags=os:openbsd
>>>           """
>>> -        self.boot_openbsd73("cortex-a57")
>>> +        self.boot_openbsd("cortex-a57")
>>>   -    def test_sbsaref_openbsd73_neoverse_n1(self):
>>> +    def test_sbsaref_openbsd_neoverse_n1(self):
>>>           """
>>>           :avocado: tags=cpu:neoverse-n1
>>> +        :avocado: tags=os:openbsd
>>>           """
>>> -        self.boot_openbsd73("neoverse-n1")
>>> +        self.boot_openbsd("neoverse-n1")
>>>   -    def test_sbsaref_openbsd73_max(self):
>>> -        """
>>> -        :avocado: tags=cpu:max
>>> -        """
>>> -        self.boot_openbsd73("max")
>>> +# OpenBSD 7.4 does not boot on current max cpu.
>>> +#
>>> +#   def test_sbsaref_openbsd_max_pauth_off(self):
>>> +#       """
>>> +#       :avocado: tags=cpu:max
>>> +#       :avocado: tags=os:openbsd
>>> +#       """
>>> +#       self.boot_openbsd("max,pauth=off")
>> If we are not going to delete the entries then at least use a @skip
>> instead of commenting. Maybe:
>>    @skip("Potential un-diagnosed upstream bug?")
>
> Daniel or Peter suggested to open a GitLab issue and use
>
>     @skip("https://gitlab.com/qemu-project/qemu/-/issues/xyz")
>
> to track progress.

That's a good idea. Are you going to respin?

>
>> but it would be nice to figure out exactly where is breaks.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v2 3/4] tests/avocado: use OpenBSD 7.4 for sbsa-ref

[Marcin Juszkiewicz]

W dniu 14.03.2024 o 15:56, Alex Bennée pisze:
>>> If we are not going to delete the entries then at least use a @skip
>>> instead of commenting. Maybe:
>>>     @skip("Potential un-diagnosed upstream bug?")
>> Daniel or Peter suggested to open a GitLab issue and use
>>
>>      @skip("https://gitlab.com/qemu-project/qemu/-/issues/xyz")
>>
>> to track progress.
> That's a good idea. Are you going to respin?

Opened https://gitlab.com/qemu-project/qemu/-/issues/2224 to track 
problem. Subscribed to arm@openbsd mailing list.

Will walk the dog and then mail them with problem.

And respin patch series tomorrow.

[PATCH v2 4/4] tests/avocado: sbsa-ref: add Alpine tests for misc 'max' setup

[Marcin Juszkiewicz]

PAuth makes run timeout on CI so add tests using 'max' without it
and with impdef one.

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
---
 tests/avocado/machine_aarch64_sbsaref.py | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
index 0e52dc5854..0dd2eff0de 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -140,18 +140,36 @@ def boot_alpine_linux(self, cpu):
     def test_sbsaref_alpine_linux_cortex_a57(self):
         """
         :avocado: tags=cpu:cortex-a57
+        :avocado: tags=os:linux
         """
         self.boot_alpine_linux("cortex-a57")
 
     def test_sbsaref_alpine_linux_neoverse_n1(self):
         """
         :avocado: tags=cpu:neoverse-n1
+        :avocado: tags=os:linux
         """
         self.boot_alpine_linux("neoverse-n1")
 
+    def test_sbsaref_alpine_linux_max_pauth_off(self):
+        """
+        :avocado: tags=cpu:max
+        :avocado: tags=os:linux
+        """
+        self.boot_alpine_linux("max,pauth=off")
+
+    def test_sbsaref_alpine_linux_max_pauth_impdef(self):
+        """
+        :avocado: tags=cpu:max
+        :avocado: tags=os:linux
+        """
+        self.boot_alpine_linux("max,pauth-impdef=on")
+
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
     def test_sbsaref_alpine_linux_max(self):
         """
         :avocado: tags=cpu:max
+        :avocado: tags=os:linux
         """
         self.boot_alpine_linux("max")
 

-- 
2.44.0