Re: [PATCH 0/3] tests/avocado: update sbsa-ref firmware to latest

["Philippe Mathieu-Daudé" <philmd@linaro.org>]

Hi Marcin,

On 13/3/24 11:08, Marcin Juszkiewicz wrote:
> Updating sbsa-ref firmware for QEMU CI was manual task. Now it is
> replaced by CI job run on CodeLinaro Gitlab instance.
> 
> This patchset updates to current state:
> 
> - Trusted Firmware v2.10.2 (latest LTS)
> - Tianocore EDK2 stable202402 (latest release)
> 
> And Tianocore EDK2-platforms commit 085c2fb (edk2-platforms does not
> have releases).
> 
> Firmware images were built using Debian 'bookworm' cross gcc 12.2.0
> compiler.
> 
> And while I am in that file I dropped use of 'virtio-rng-pci' device as
> sbsa-ref is supposed to emulate physical hardware.
> 
> OpenBSD updated to 7.4 version.
> 
> Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
> ---
> Marcin Juszkiewicz (3):
>        tests/avocado: update sbsa-ref firmware
>        tests/avocado: drop virtio-rng from sbsa-ref tests
>        tests/avocado: use OpenBSD 7.4 for sbsa-ref

Hmm 'max' is timeouting:

  (1/7) test_sbsaref_edk2_firmware: PASS (9.99 s)
  (2/7) test_sbsaref_alpine_linux_cortex_a57: PASS (54.82 s)
  (3/7) test_sbsaref_alpine_linux_neoverse_n1: PASS (53.09 s)
  (4/7) test_sbsaref_alpine_linux_max: INTERRUPTED: Runner error 
occurred: Timeout reached... (180.51 s)
  (5/7) test_sbsaref_openbsd_cortex_a57: PASS (33.45 s)
  (6/7) test_sbsaref_openbsd_neoverse_n1: PASS (22.15 s)
  (7/7) test_sbsaref_openbsd_max: INTERRUPTED: Runner error occurred: 
Timeout reached... (180.42 s)
RESULTS    : PASS 5 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 2 | 
CANCEL 0
JOB TIME   : 537.48 s

Using impdef pauth ...

-- >8 --
diff --git a/tests/avocado/machine_aarch64_sbsaref.py 
b/tests/avocado/machine_aarch64_sbsaref.py
index 94c9f81d72..d349ce73ae 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -156 +156 @@ def test_sbsaref_alpine_linux_max(self):
-        self.boot_alpine_linux("max")
+        self.boot_alpine_linux("max,pauth-impdef=on")
@@ -201 +201 @@ def test_sbsaref_openbsd_max(self):
-        self.boot_openbsd("max")
+        self.boot_openbsd("max,pauth-impdef=on")
---

... I still get:

  (1/7) test_sbsaref_edk2_firmware: PASS (5.56 s)
  (2/7) test_sbsaref_alpine_linux_cortex_a57: PASS (53.09 s)
  (3/7) test_sbsaref_alpine_linux_neoverse_n1: PASS (53.00 s)
  (4/7) test_sbsaref_alpine_linux_max: PASS (70.99 s)
  (5/7) test_sbsaref_openbsd_cortex_a57: PASS (22.63 s)
  (6/7) test_sbsaref_openbsd_neoverse_n1: PASS (22.56 s)
  (7/7) test_sbsaref_openbsd_max: INTERRUPTED: Timeout reached... (180.43 s)
RESULTS    : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 1 | 
CANCEL 0
JOB TIME   : 409.29 s

What about disabling path on openbsd and using @skip tags?

-- >8 --
diff --git a/tests/avocado/machine_aarch64_sbsaref.py 
b/tests/avocado/machine_aarch64_sbsaref.py
index 94c9f81d72..c6ee7332a9 100644
--- a/tests/avocado/machine_aarch64_sbsaref.py
+++ b/tests/avocado/machine_aarch64_sbsaref.py
@@ -142,2 +142,3 @@ def test_sbsaref_alpine_linux_cortex_a57(self):
          :avocado: tags=cpu:cortex-a57
+        :avocado: tags=os:linux
          """
@@ -148,2 +149,3 @@ def test_sbsaref_alpine_linux_neoverse_n1(self):
          :avocado: tags=cpu:neoverse-n1
+        :avocado: tags=os:linux
          """
@@ -151,2 +153,17 @@ def test_sbsaref_alpine_linux_neoverse_n1(self):

+    def test_sbsaref_alpine_linux_max_pauth_off(self):
+        """
+        :avocado: tags=cpu:max
+        :avocado: tags=os:linux
+        """
+        self.boot_alpine_linux("max,pauth=off")
+
+    def test_sbsaref_alpine_linux_max_pauth_impdef(self):
+        """
+        :avocado: tags=cpu:max
+        :avocado: tags=os:linux
+        """
+        self.boot_alpine_linux("max,pauth-impdef=on")
+
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might 
timeout')
      def test_sbsaref_alpine_linux_max(self):
@@ -154,2 +171,3 @@ def test_sbsaref_alpine_linux_max(self):
          :avocado: tags=cpu:max
+        :avocado: tags=os:linux
          """
@@ -186,3 +204,4 @@ def test_sbsaref_openbsd_cortex_a57(self):
          """
-        :avocado: tags=cpu:cortex-a57,os:openbsd
+        :avocado: tags=cpu:cortex-a57
+        :avocado: tags=os:openbsd
          """
@@ -192,3 +211,4 @@ def test_sbsaref_openbsd_neoverse_n1(self):
          """
-        :avocado: tags=cpu:neoverse-n1,os:openbsd
+        :avocado: tags=cpu:neoverse-n1
+        :avocado: tags=os:openbsd
          """
@@ -196,7 +216,24 @@ def test_sbsaref_openbsd_neoverse_n1(self):

+    def test_sbsaref_openbsd_max_pauth_off(self):
+        """
+        :avocado: tags=cpu:max
+        :avocado: tags=os:openbsd
+        """
+        self.boot_openbsd("max,pauth=off")
+
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might 
timeout')
+    def test_sbsaref_openbsd_max_pauth_impdef(self):
+        """
+        :avocado: tags=cpu:max
+        :avocado: tags=os:openbsd
+        """
+        self.boot_openbsd("max,pauth-impdef=on")
+
+    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might 
timeout')
      def test_sbsaref_openbsd_max(self):
          """
-        :avocado: tags=cpu:max,os:openbsd
+        :avocado: tags=cpu:max
+        :avocado: tags=os:openbsd
          """
-        self.boot_openbsd("max")
+        self.boot_openbsd("max,pauth-impdef=on")

---

However OpenBSD still timeouts at 180s:

  (01/11) test_sbsaref_edk2_firmware: PASS (7.25 s)
  (02/11) test_sbsaref_alpine_linux_cortex_a57: PASS (54.30 s)
  (03/11) test_sbsaref_alpine_linux_neoverse_n1: PASS (52.25 s)
  (04/11) test_sbsaref_alpine_linux_max_pauth_off: PASS (53.31 s)
  (05/11) test_sbsaref_alpine_linux_max_pauth_impdef: PASS (72.72 s)
  (06/11) test_sbsaref_alpine_linux_max: SKIP: Test might timeout
  (07/11) test_sbsaref_openbsd_cortex_a57: PASS (22.37 s)
  (08/11) test_sbsaref_openbsd_neoverse_n1: PASS (22.98 s)
  (09/11) test_sbsaref_openbsd_max_pauth_off: INTERRUPTED: Timeout 
reached... (180.54 s)
  (10/11) test_sbsaref_openbsd_max_pauth_impdef: SKIP: Test might timeout
  (11/11) test_sbsaref_openbsd_max: SKIP: Test might timeout

Regards,

Phil.