Re: [PATCH v3 1/2] tests/avocado: sbsa-ref: add FreeBSD tests

[Peter Maydell <peter.maydell@linaro.org>]

On Mon, 24 Jun 2024 at 17:20, Marcin Juszkiewicz
<marcin.juszkiewicz@linaro.org> wrote:
>
> FreeBSD has longer support cycle for stable release (14.x EoL in 2028)
> than OpenBSD (7.3 we used is already EoL). Also bugfixes are backported
> so we can stay on 14.x for longer. Tests done on OpenBSD will now be
> done using FreeBSD.
>
> OpenBSD 7.3 stays with Cortex-A57 test for local runs only.
>
> Moved from Neoverse-N1 to Neoverse-N2 as sbsa-ref defaults were changed.
>
> Timeout messages expanded to mention being affected by PAuth emulation.
>
> Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
> ---
>  tests/avocado/machine_aarch64_sbsaref.py | 75 ++++++++++++++++++++++++--------
>  1 file changed, 56 insertions(+), 19 deletions(-)
>
> diff --git a/tests/avocado/machine_aarch64_sbsaref.py b/tests/avocado/machine_aarch64_sbsaref.py
> index 6bb82f2a03..bb2c098aac 100644
> --- a/tests/avocado/machine_aarch64_sbsaref.py
> +++ b/tests/avocado/machine_aarch64_sbsaref.py
> @@ -1,4 +1,4 @@
> -# Functional test that boots a Linux kernel and checks the console
> +# Functional test that boots a kernel and checks the console
>  #
>  # SPDX-FileCopyrightText: 2023-2024 Linaro Ltd.
>  # SPDX-FileContributor: Philippe Mathieu-Daudé <philmd@linaro.org>
> @@ -8,7 +8,7 @@
>
>  import os
>
> -from avocado import skipUnless
> +from avocado import skipIf, skipUnless
>  from avocado.utils import archive
>
>  from avocado_qemu import QemuSystemTest
> @@ -165,7 +165,8 @@ def test_sbsaref_alpine_linux_max_pauth_impdef(self):
>          """
>          self.boot_alpine_linux("max,pauth-impdef=on")
>
> -    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'), 'Test might timeout')
> +    @skipUnless(os.getenv('AVOCADO_TIMEOUT_EXPECTED'),
> +                'Test might timeout due to PAuth emulation')
>      def test_sbsaref_alpine_linux_max(self):
>          """
>          :avocado: tags=cpu:max
> @@ -173,7 +174,6 @@ def test_sbsaref_alpine_linux_max(self):
>          """
>          self.boot_alpine_linux("max")
>
> -
>      # This tests the whole boot chain from EFI to Userspace
>      # We only boot a whole OS for the current top level CPU and GIC
>      # Other test profiles should use more minimal boots
> @@ -200,6 +200,9 @@ def boot_openbsd73(self, cpu):
>                                   "Welcome to the OpenBSD/arm64"
>                                   " 7.3 installation program.")
>
> +
> +    # we keep OpenBSD 7.3 on Cortex-A57 by request
> +    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')

Don't skip stuff if GITLAB_CI, please. If we don't want to
run it on the CI then we shouldn't be running it by default
when a local user runs check-avocado either.

Any skip should have a clear description of the reason why
we're skipping it.

thanks
-- PMM