[PATCH 0/8] automation/cirrus-ci: add smoke tests of the FreeBSD builds

[PATCH 0/8] automation/cirrus-ci: add smoke tests of the FreeBSD builds

[Roger Pau Monne]

Hello,

The aim of the series is to introduce a basic set of smoke tests using
QEMU and XTF, to ensure some minimal run-time testing of the LLVM
toolchain generated Xen binary.

Roger Pau Monne (8):
  automation/cirrus-ci: add timestamps
  automation/cirrus-ci: update FreeBSD to 13.5
  automation/cirrus-ci: store Xen Kconfig before doing a build
  automation/cirrus-ci: use matrix keyword to generate per-version build
    tasks
  automation/cirrus-ci: build XTF
  automation/cirrus-ci: store XTF and Xen build artifacts
  automation/console.exp: do not assume expect is always at /usr/bin/
  automation/cirrus-ci: add smoke tests for the FreeBSD builds

 .cirrus.yml                    | 170 +++++++++++++++++++++++++--------
 CHANGELOG.md                   |   1 +
 automation/scripts/console.exp |   2 +-
 3 files changed, 133 insertions(+), 40 deletions(-)

-- 
2.48.1

[PATCH 1/8] automation/cirrus-ci: add timestamps

[Roger Pau Monne]

Such timestamps can still be disabled from the Web UI using a tick box.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
 .cirrus.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.cirrus.yml b/.cirrus.yml
index 1a39f5026f9a..aa195ba370ac 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -14,6 +14,7 @@ freebsd_template: &FREEBSD_ENV
     APPEND_LIB: /usr/local/lib
     APPEND_INCLUDES: /usr/local/include
     CIRRUS_CLONE_DEPTH: 1
+    CIRRUS_LOG_TIMESTAMP: true
 
 freebsd_artifacts: &FREEBSD_ARTIFACTS
   always:
-- 
2.48.1

Re: [PATCH 1/8] automation/cirrus-ci: add timestamps

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 687 bytes --]

On Mon, 17 Mar 2025, Roger Pau Monne wrote:
> Such timestamps can still be disabled from the Web UI using a tick box.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>

> ---
>  .cirrus.yml | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/.cirrus.yml b/.cirrus.yml
> index 1a39f5026f9a..aa195ba370ac 100644
> --- a/.cirrus.yml
> +++ b/.cirrus.yml
> @@ -14,6 +14,7 @@ freebsd_template: &FREEBSD_ENV
>      APPEND_LIB: /usr/local/lib
>      APPEND_INCLUDES: /usr/local/include
>      CIRRUS_CLONE_DEPTH: 1
> +    CIRRUS_LOG_TIMESTAMP: true
>  
>  freebsd_artifacts: &FREEBSD_ARTIFACTS
>    always:
> -- 
> 2.48.1
> 

[PATCH 2/8] automation/cirrus-ci: update FreeBSD to 13.5

[Roger Pau Monne]

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
 .cirrus.yml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index aa195ba370ac..ce7588b7b6b1 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -1,7 +1,7 @@
 # https://cirrus-ci.org/guide/tips-and-tricks/#sharing-configuration-between-tasks
 freebsd_13: &FREEBSD_13
   freebsd_instance:
-    image_family: freebsd-13-4
+    image_family: freebsd-13-5
 freebsd_14: &FREEBSD_14
   freebsd_instance:
     image_family: freebsd-14-2
-- 
2.48.1

Re: [PATCH 2/8] automation/cirrus-ci: update FreeBSD to 13.5

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]

On Mon, 17 Mar 2025, Roger Pau Monne wrote:
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>

> ---
>  .cirrus.yml | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/.cirrus.yml b/.cirrus.yml
> index aa195ba370ac..ce7588b7b6b1 100644
> --- a/.cirrus.yml
> +++ b/.cirrus.yml
> @@ -1,7 +1,7 @@
>  # https://cirrus-ci.org/guide/tips-and-tricks/#sharing-configuration-between-tasks
>  freebsd_13: &FREEBSD_13
>    freebsd_instance:
> -    image_family: freebsd-13-4
> +    image_family: freebsd-13-5
>  freebsd_14: &FREEBSD_14
>    freebsd_instance:
>      image_family: freebsd-14-2
> -- 
> 2.48.1
> 

[PATCH 3/8] automation/cirrus-ci: store Xen Kconfig before doing a build

[Roger Pau Monne]

In case the build fails or gets stuck, store the Kconfig file ahead of
starting the build.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
 .cirrus.yml | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index ce7588b7b6b1..490f214f0460 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -16,7 +16,7 @@ freebsd_template: &FREEBSD_ENV
     CIRRUS_CLONE_DEPTH: 1
     CIRRUS_LOG_TIMESTAMP: true
 
-freebsd_artifacts: &FREEBSD_ARTIFACTS
+freebsd_configure_artifacts: &FREEBSD_CONFIGURE_ARTIFACTS
   always:
     rename_script:
       - cp xen/.config xen-config
@@ -32,25 +32,30 @@ freebsd_full_build_template: &FREEBSD_FULL_BUILD_TEMPLATE
                                  yajl lzo2 pixman argp-standalone
                                  libxml2 glib git python3
 
-  build_script:
+  configure_script:
     - cc --version
     - ./configure --with-system-seabios=/usr/local/share/seabios/bios.bin
-    - gmake -j`sysctl -n hw.ncpu` clang=y
+    - gmake -j`sysctl -n hw.ncpu` -C xen clang=y defconfig
+
+  << : *FREEBSD_CONFIGURE_ARTIFACTS
 
-  << : *FREEBSD_ARTIFACTS
+  build_script:
+    - gmake -j`sysctl -n hw.ncpu` clang=y
 
 freebsd_randconfig_template: &FREEBSD_RANDCONFIG_TEMPLATE
   << : *FREEBSD_ENV
 
   install_script: pkg install -y gmake python3 bison
 
-  build_script:
+  configure_script:
     - cc --version
     - gmake -j`sysctl -n hw.ncpu` -C xen clang=y \
             KCONFIG_ALLCONFIG=tools/kconfig/allrandom.config randconfig
-    - gmake -j`sysctl -n hw.ncpu` build-xen clang=y
 
-  << : *FREEBSD_ARTIFACTS
+  << : *FREEBSD_CONFIGURE_ARTIFACTS
+
+  build_script:
+    - gmake -j`sysctl -n hw.ncpu` build-xen clang=y
 
 task:
   name: 'FreeBSD 13: full build'
-- 
2.48.1

Re: [PATCH 3/8] automation/cirrus-ci: store Xen Kconfig before doing a build

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 254 bytes --]

On Mon, 17 Mar 2025, Roger Pau Monne wrote:
> In case the build fails or gets stuck, store the Kconfig file ahead of
> starting the build.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>

[PATCH 4/8] automation/cirrus-ci: use matrix keyword to generate per-version build tasks

[Roger Pau Monne]

Move the current logic to use the matrix keyword to generate a task for
each version of FreeBSD we want to build Xen on.  The matrix keyword
however cannot be used in YAML aliases, so it needs to be explicitly used
inside of each task, which creates a bit of duplication.  At least abstract
the FreeBSD minor version numbers to avoid repetition of image names.

Note that the full build uses matrix over an env variable instead of using
it directly in image_family.  This is so that the alias can also be set
based on the FreeBSD version, in preparation for adding further tasks that
will depend on the full build having finished.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
I could alternatively name FREEBSD_15 FREEBSD_CURRENT or FREEBSD_HEAD or
something versionless, however I don't know about the other versions.

Assuming there will only be 3 versions tested at all times I could use:
FREEBSD_HEAD, FREEBSD_PRODUCTION and FREEBSD_LEGACY, as to avoid having to
specify versions in the task definitions.
---
 .cirrus.yml | 78 +++++++++++++++++++++++++----------------------------
 1 file changed, 37 insertions(+), 41 deletions(-)

diff --git a/.cirrus.yml b/.cirrus.yml
index 490f214f0460..bda4dbe86a7c 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -1,13 +1,9 @@
 # https://cirrus-ci.org/guide/tips-and-tricks/#sharing-configuration-between-tasks
-freebsd_13: &FREEBSD_13
-  freebsd_instance:
-    image_family: freebsd-13-5
-freebsd_14: &FREEBSD_14
-  freebsd_instance:
-    image_family: freebsd-14-2
-freebsd_15: &FREEBSD_15
-  freebsd_instance:
-    image_family: freebsd-15-0-snap
+freebsd_versions: &FREEBSD_VERSIONS
+  env:
+    FREEBSD_13: freebsd-13-5
+    FREEBSD_14: freebsd-14-2
+    FREEBSD_15: freebsd-15-0-snap
 
 freebsd_template: &FREEBSD_ENV
   environment:
@@ -24,7 +20,24 @@ freebsd_configure_artifacts: &FREEBSD_CONFIGURE_ARTIFACTS
       path: xen-config
       type: text/plain
 
-freebsd_full_build_template: &FREEBSD_FULL_BUILD_TEMPLATE
+task:
+  name: 'FreeBSD: full build'
+
+  # It's not possible to use the matrix keyword in YAML aliases, as they
+  # keyword usage is restricted to task, docker_builder or pipe.  Attempting to
+  # use a YAML alias with the duplicated keys doesn't work either. Use an env
+  # variable so the version can also be appended to the task alias.
+  << : *FREEBSD_VERSIONS
+  env:
+    matrix:
+      FREEBSD_VERSION: $FREEBSD_13
+      FREEBSD_VERSION: $FREEBSD_14
+      FREEBSD_VERSION: $FREEBSD_15
+
+  alias: freebsd_full_$FREEBSD_VERSION
+  freebsd_instance:
+    image_family: $FREEBSD_VERSION
+
   << : *FREEBSD_ENV
 
   install_script: pkg install -y seabios gmake ninja bash
@@ -42,7 +55,20 @@ freebsd_full_build_template: &FREEBSD_FULL_BUILD_TEMPLATE
   build_script:
     - gmake -j`sysctl -n hw.ncpu` clang=y
 
-freebsd_randconfig_template: &FREEBSD_RANDCONFIG_TEMPLATE
+task:
+  name: 'FreeBSD: randconfig build'
+
+  # It's not possible to use the matrix keyword in YAML aliases, as they
+  # keyword usage is restricted to task, docker_builder or pipe.  Attempting to
+  # use a YAML alias with the duplicated `image_family` keys doesn't work
+  # either. Abstract the version numbers at least.
+  << : *FREEBSD_VERSIONS
+  freebsd_instance:
+    matrix:
+      image_family: $FREEBSD_13
+      image_family: $FREEBSD_14
+      image_family: $FREEBSD_15
+
   << : *FREEBSD_ENV
 
   install_script: pkg install -y gmake python3 bison
@@ -56,33 +82,3 @@ freebsd_randconfig_template: &FREEBSD_RANDCONFIG_TEMPLATE
 
   build_script:
     - gmake -j`sysctl -n hw.ncpu` build-xen clang=y
-
-task:
-  name: 'FreeBSD 13: full build'
-  << : *FREEBSD_13
-  << : *FREEBSD_FULL_BUILD_TEMPLATE
-
-task:
-  name: 'FreeBSD 14: full build'
-  << : *FREEBSD_14
-  << : *FREEBSD_FULL_BUILD_TEMPLATE
-
-task:
-  name: 'FreeBSD 15: full build'
-  << : *FREEBSD_15
-  << : *FREEBSD_FULL_BUILD_TEMPLATE
-
-task:
-  name: 'FreeBSD 13: randconfig'
-  << : *FREEBSD_13
-  << : *FREEBSD_RANDCONFIG_TEMPLATE
-
-task:
-  name: 'FreeBSD 14: randconfig'
-  << : *FREEBSD_14
-  << : *FREEBSD_RANDCONFIG_TEMPLATE
-
-task:
-  name: 'FreeBSD 15: randconfig'
-  << : *FREEBSD_15
-  << : *FREEBSD_RANDCONFIG_TEMPLATE
-- 
2.48.1

Re: [PATCH 4/8] automation/cirrus-ci: use matrix keyword to generate per-version build tasks

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1226 bytes --]

On Mon, 17 Mar 2025, Roger Pau Monne wrote:
> Move the current logic to use the matrix keyword to generate a task for
> each version of FreeBSD we want to build Xen on.  The matrix keyword
> however cannot be used in YAML aliases, so it needs to be explicitly used
> inside of each task, which creates a bit of duplication.  At least abstract
> the FreeBSD minor version numbers to avoid repetition of image names.
> 
> Note that the full build uses matrix over an env variable instead of using
> it directly in image_family.  This is so that the alias can also be set
> based on the FreeBSD version, in preparation for adding further tasks that
> will depend on the full build having finished.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
> I could alternatively name FREEBSD_15 FREEBSD_CURRENT or FREEBSD_HEAD or
> something versionless, however I don't know about the other versions.
> 
> Assuming there will only be 3 versions tested at all times I could use:
> FREEBSD_HEAD, FREEBSD_PRODUCTION and FREEBSD_LEGACY, as to avoid having to
> specify versions in the task definitions.

I think it is OK but I'll leave it to others to comment as well

Acked-by: Stefano Stabellini <sstabellini@kernel.org>

[PATCH 5/8] automation/cirrus-ci: build XTF

[Roger Pau Monne]

In preparation for using the XTF selftests to smoke test the FreeBSD based
Xen builds.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
I've used the current version of FreeBSD to build XTF, but it might be
safer to use the latest production (release) version instead.
---
 .cirrus.yml | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/.cirrus.yml b/.cirrus.yml
index bda4dbe86a7c..d8a322acd54c 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -12,6 +12,12 @@ freebsd_template: &FREEBSD_ENV
     CIRRUS_CLONE_DEPTH: 1
     CIRRUS_LOG_TIMESTAMP: true
 
+freebsd_template_latest: &FREEBSD_ENV_LATEST
+  << : *FREEBSD_VERSIONS
+  freebsd_instance:
+    image_family: $FREEBSD_15
+  << : *FREEBSD_ENV
+
 freebsd_configure_artifacts: &FREEBSD_CONFIGURE_ARTIFACTS
   always:
     rename_script:
@@ -82,3 +88,21 @@ task:
 
   build_script:
     - gmake -j`sysctl -n hw.ncpu` build-xen clang=y
+
+task:
+  name: 'FreeBSD: XTF build'
+  alias: xtf
+
+  << : *FREEBSD_ENV_LATEST
+
+  clone_script:
+    - pkg install -y git
+    - git clone --depth 1 https://xenbits.xen.org/git-http/xtf.git
+
+  install_script: pkg install -y gmake
+
+  build_script:
+    - cd xtf
+    - cc --version
+    - git rev-parse HEAD
+    - gmake -j`sysctl -n hw.ncpu` LLVM=y
-- 
2.48.1

Re: [PATCH 5/8] automation/cirrus-ci: build XTF

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1508 bytes --]

On Mon, 17 Mar 2025, Roger Pau Monne wrote:
> In preparation for using the XTF selftests to smoke test the FreeBSD based
> Xen builds.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
> I've used the current version of FreeBSD to build XTF, but it might be
> safer to use the latest production (release) version instead.

Yes, I think so, especially as the latest is a rolling distro?


> ---
>  .cirrus.yml | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
> 
> diff --git a/.cirrus.yml b/.cirrus.yml
> index bda4dbe86a7c..d8a322acd54c 100644
> --- a/.cirrus.yml
> +++ b/.cirrus.yml
> @@ -12,6 +12,12 @@ freebsd_template: &FREEBSD_ENV
>      CIRRUS_CLONE_DEPTH: 1
>      CIRRUS_LOG_TIMESTAMP: true
>  
> +freebsd_template_latest: &FREEBSD_ENV_LATEST
> +  << : *FREEBSD_VERSIONS
> +  freebsd_instance:
> +    image_family: $FREEBSD_15
> +  << : *FREEBSD_ENV
> +
>  freebsd_configure_artifacts: &FREEBSD_CONFIGURE_ARTIFACTS
>    always:
>      rename_script:
> @@ -82,3 +88,21 @@ task:
>  
>    build_script:
>      - gmake -j`sysctl -n hw.ncpu` build-xen clang=y
> +
> +task:
> +  name: 'FreeBSD: XTF build'
> +  alias: xtf
> +
> +  << : *FREEBSD_ENV_LATEST
> +
> +  clone_script:
> +    - pkg install -y git
> +    - git clone --depth 1 https://xenbits.xen.org/git-http/xtf.git
> +
> +  install_script: pkg install -y gmake
> +
> +  build_script:
> +    - cd xtf
> +    - cc --version
> +    - git rev-parse HEAD
> +    - gmake -j`sysctl -n hw.ncpu` LLVM=y
> -- 
> 2.48.1
> 

Re: [PATCH 5/8] automation/cirrus-ci: build XTF

[Roger Pau Monné]

On Mon, Mar 17, 2025 at 05:07:22PM -0700, Stefano Stabellini wrote:
> On Mon, 17 Mar 2025, Roger Pau Monne wrote:
> > In preparation for using the XTF selftests to smoke test the FreeBSD based
> > Xen builds.
> > 
> > Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> > ---
> > I've used the current version of FreeBSD to build XTF, but it might be
> > safer to use the latest production (release) version instead.
> 
> Yes, I think so, especially as the latest is a rolling distro?

Kind of - it's even weirder, as part of the libraries are bundled with
the FreeBSD base system (together with the kernel).  When using current
there could be a window where the packages are built against stale
libraries, because the ones in base have been updated, but the
packages haven't been rebuilt yet.

I will switch to use the last production release.

Thanks, Roger.

[PATCH 6/8] automation/cirrus-ci: store XTF and Xen build artifacts

[Roger Pau Monne]

In preparation for adding some smoke tests that will consume those outputs.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
 .cirrus.yml | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/.cirrus.yml b/.cirrus.yml
index d8a322acd54c..c74d5cf6eb9a 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -61,6 +61,14 @@ task:
   build_script:
     - gmake -j`sysctl -n hw.ncpu` clang=y
 
+  xen_artifacts:
+    path: xen/xen
+    type: application/octet-stream
+
+  debug_artifacts:
+    path: xen/xen-syms
+    type: application/octet-stream
+
 task:
   name: 'FreeBSD: randconfig build'
 
@@ -106,3 +114,7 @@ task:
     - cc --version
     - git rev-parse HEAD
     - gmake -j`sysctl -n hw.ncpu` LLVM=y
+
+  xtf_artifacts:
+    path: xtf/tests/selftest/test-*-selftest
+    type: application/octet-stream
-- 
2.48.1

Re: [PATCH 6/8] automation/cirrus-ci: store XTF and Xen build artifacts

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 236 bytes --]

On Mon, 17 Mar 2025, Roger Pau Monne wrote:
> In preparation for adding some smoke tests that will consume those outputs.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>

Acked-by: Stefano Stabellini <sstabellini@kernel.org>

[PATCH 7/8] automation/console.exp: do not assume expect is always at /usr/bin/

[Roger Pau Monne]

Instead use env to find the location of expect.

Additionally do not use the -f flag, as it's only meaningful when passing
arguments on the command line, which we never do for console.exp.  From the
expect 5.45.4 man page:

> The -f flag prefaces a file from which to read commands from.  The flag
> itself is optional as it is only useful when using the #! notation (see
> above), so  that other arguments may be supplied on the command line.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
 automation/scripts/console.exp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/automation/scripts/console.exp b/automation/scripts/console.exp
index 310543c33e5e..31ce97b91b63 100755
--- a/automation/scripts/console.exp
+++ b/automation/scripts/console.exp
@@ -1,4 +1,4 @@
-#!/usr/bin/expect -f
+#!/usr/bin/env expect
 
 if {[info exists env(TEST_TIMEOUT_OVERRIDE)]} {
     set timeout $env(TEST_TIMEOUT_OVERRIDE)
-- 
2.48.1

Re: [PATCH 7/8] automation/console.exp: do not assume expect is always at /usr/bin/

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1167 bytes --]

On Mon, 17 Mar 2025, Roger Pau Monne wrote:
> Instead use env to find the location of expect.
> 
> Additionally do not use the -f flag, as it's only meaningful when passing
> arguments on the command line, which we never do for console.exp.  From the
> expect 5.45.4 man page:
> 
> > The -f flag prefaces a file from which to read commands from.  The flag
> > itself is optional as it is only useful when using the #! notation (see
> > above), so  that other arguments may be supplied on the command line.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>

Do you have a link to a successful Gitlab and Cirrus CI?

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

> ---
>  automation/scripts/console.exp | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/automation/scripts/console.exp b/automation/scripts/console.exp
> index 310543c33e5e..31ce97b91b63 100755
> --- a/automation/scripts/console.exp
> +++ b/automation/scripts/console.exp
> @@ -1,4 +1,4 @@
> -#!/usr/bin/expect -f
> +#!/usr/bin/env expect
>  
>  if {[info exists env(TEST_TIMEOUT_OVERRIDE)]} {
>      set timeout $env(TEST_TIMEOUT_OVERRIDE)
> -- 
> 2.48.1
> 

Re: [PATCH 7/8] automation/console.exp: do not assume expect is always at /usr/bin/

[Roger Pau Monné]

On Mon, Mar 17, 2025 at 04:48:05PM -0700, Stefano Stabellini wrote:
> On Mon, 17 Mar 2025, Roger Pau Monne wrote:
> > Instead use env to find the location of expect.
> > 
> > Additionally do not use the -f flag, as it's only meaningful when passing
> > arguments on the command line, which we never do for console.exp.  From the
> > expect 5.45.4 man page:
> > 
> > > The -f flag prefaces a file from which to read commands from.  The flag
> > > itself is optional as it is only useful when using the #! notation (see
> > > above), so  that other arguments may be supplied on the command line.
> > 
> > Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> 
> Do you have a link to a successful Gitlab and Cirrus CI?

Yes.  They are here:

https://cirrus-ci.com/build/4981406134173696
https://gitlab.com/xen-project/people/royger/xen/-/pipelines/1719913362

> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

Thanks, Roger.

[PATCH 8/8] automation/cirrus-ci: add smoke tests for the FreeBSD builds

[Roger Pau Monne]

Introduce a basic set of smoke tests using the XTF selftest image, and run
them on QEMU.  Use the matrix keyword to create a different task for each
XTF flavor on each FreeBSD build.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
I've used the current version of FreeBSD to run the tests, but it might be
safer to use the latest production (release) version instead.
---
 .cirrus.yml  | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 CHANGELOG.md |  1 +
 2 files changed, 55 insertions(+)

diff --git a/.cirrus.yml b/.cirrus.yml
index c74d5cf6eb9a..9c8b4b235cfd 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -5,6 +5,8 @@ freebsd_versions: &FREEBSD_VERSIONS
     FREEBSD_14: freebsd-14-2
     FREEBSD_15: freebsd-15-0-snap
 
+# Build jobs
+
 freebsd_template: &FREEBSD_ENV
   environment:
     APPEND_LIB: /usr/local/lib
@@ -118,3 +120,55 @@ task:
   xtf_artifacts:
     path: xtf/tests/selftest/test-*-selftest
     type: application/octet-stream
+
+# Test jobs
+
+task:
+  name: 'FreeBSD: XTF selftest'
+
+  << : *FREEBSD_ENV_LATEST
+
+  env:
+    matrix:
+      FREEBSD_BUILD: $FREEBSD_13
+      FREEBSD_BUILD: $FREEBSD_14
+      FREEBSD_BUILD: $FREEBSD_15
+
+  depends_on:
+    - freebsd_full_$FREEBSD_BUILD
+    - xtf
+
+  install_script: pkg install -y qemu-nox11 expect
+
+  env:
+    matrix:
+      XTF_ARCH: hvm32
+      XTF_ARCH: hvm32pae
+      XTF_ARCH: hvm32pse
+      XTF_ARCH: hvm64
+      XTF_ARCH: pv64
+
+  fetch_script:
+    - fetch https://api.cirrus-ci.com/v1/artifact/build/$CIRRUS_BUILD_ID/xtf/xtf.zip
+    - fetch https://api.cirrus-ci.com/v1/artifact/build/$CIRRUS_BUILD_ID/freebsd_full_$FREEBSD_BUILD/xen.zip
+    - unzip xtf.zip
+    - unzip xen.zip
+
+  test_script: |
+    case "$XTF_ARCH" in \
+      *hvm*) \
+        XEN_EXTRA_CMD="dom0=pvh dom0-iommu=none" \
+        ;; \
+    esac
+    export TEST_CMD="qemu-system-x86_64 -kernel xen/xen -initrd xtf/tests/selftest/test-${XTF_ARCH}-selftest \
+                                        -append \"loglvl=all console=com1 noreboot console_timestamps=boot dom0=verbose ${XEN_EXTRA_CMD}\" \
+                                        -m 512 -nographic -monitor none -serial stdio"
+    export TEST_LOG="serial-${FREEBSD_BUILD}-${XTF_ARCH}.txt"
+    export PASSED="Test result: SUCCESS"
+    export TEST_TIMEOUT=120
+    ./automation/scripts/console.exp | sed 's/\r\+$//'
+
+  always:
+    serial_artifacts:
+      path: serial-*.txt
+      type: text/plain
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7201c484f899..16c14e933c80 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -12,6 +12,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
  - On x86:
    - Option to attempt to fixup p2m page-faults on PVH dom0.
    - Resizable BARs is supported for PVH dom0.
+   - Smoke tests for the FreeBSD Xen builds in Cirrus CI.
 
 ### Removed
 
-- 
2.48.1

Re: [PATCH 8/8] automation/cirrus-ci: add smoke tests for the FreeBSD builds

[Oleksii Kurochko]

[-- Attachment #1: Type: text/plain, Size: 3182 bytes --]


On 3/17/25 1:16 PM, Roger Pau Monne wrote:
> Introduce a basic set of smoke tests using the XTF selftest image, and run
> them on QEMU.  Use the matrix keyword to create a different task for each
> XTF flavor on each FreeBSD build.
>
> Signed-off-by: Roger Pau Monné<roger.pau@citrix.com>
> ---
> I've used the current version of FreeBSD to run the tests, but it might be
> safer to use the latest production (release) version instead.
> ---
>   .cirrus.yml  | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>   CHANGELOG.md |  1 +

Acked-by: Oleksii Kurochko<oleksii.kurochko@gmail.com>
~ Oleksii

>   2 files changed, 55 insertions(+)
>
> diff --git a/.cirrus.yml b/.cirrus.yml
> index c74d5cf6eb9a..9c8b4b235cfd 100644
> --- a/.cirrus.yml
> +++ b/.cirrus.yml
> @@ -5,6 +5,8 @@ freebsd_versions: &FREEBSD_VERSIONS
>       FREEBSD_14: freebsd-14-2
>       FREEBSD_15: freebsd-15-0-snap
>   
> +# Build jobs
> +
>   freebsd_template: &FREEBSD_ENV
>     environment:
>       APPEND_LIB: /usr/local/lib
> @@ -118,3 +120,55 @@ task:
>     xtf_artifacts:
>       path: xtf/tests/selftest/test-*-selftest
>       type: application/octet-stream
> +
> +# Test jobs
> +
> +task:
> +  name: 'FreeBSD: XTF selftest'
> +
> +  << : *FREEBSD_ENV_LATEST
> +
> +  env:
> +    matrix:
> +      FREEBSD_BUILD: $FREEBSD_13
> +      FREEBSD_BUILD: $FREEBSD_14
> +      FREEBSD_BUILD: $FREEBSD_15
> +
> +  depends_on:
> +    - freebsd_full_$FREEBSD_BUILD
> +    - xtf
> +
> +  install_script: pkg install -y qemu-nox11 expect
> +
> +  env:
> +    matrix:
> +      XTF_ARCH: hvm32
> +      XTF_ARCH: hvm32pae
> +      XTF_ARCH: hvm32pse
> +      XTF_ARCH: hvm64
> +      XTF_ARCH: pv64
> +
> +  fetch_script:
> +    - fetchhttps://api.cirrus-ci.com/v1/artifact/build/$CIRRUS_BUILD_ID/xtf/xtf.zip
> +    - fetchhttps://api.cirrus-ci.com/v1/artifact/build/$CIRRUS_BUILD_ID/freebsd_full_$FREEBSD_BUILD/xen.zip
> +    - unzip xtf.zip
> +    - unzip xen.zip
> +
> +  test_script: |
> +    case "$XTF_ARCH" in \
> +      *hvm*) \
> +        XEN_EXTRA_CMD="dom0=pvh dom0-iommu=none" \
> +        ;; \
> +    esac
> +    export TEST_CMD="qemu-system-x86_64 -kernel xen/xen -initrd xtf/tests/selftest/test-${XTF_ARCH}-selftest \
> +                                        -append \"loglvl=all console=com1 noreboot console_timestamps=boot dom0=verbose ${XEN_EXTRA_CMD}\" \
> +                                        -m 512 -nographic -monitor none -serial stdio"
> +    export TEST_LOG="serial-${FREEBSD_BUILD}-${XTF_ARCH}.txt"
> +    export PASSED="Test result: SUCCESS"
> +    export TEST_TIMEOUT=120
> +    ./automation/scripts/console.exp | sed 's/\r\+$//'
> +
> +  always:
> +    serial_artifacts:
> +      path: serial-*.txt
> +      type: text/plain
> diff --git a/CHANGELOG.md b/CHANGELOG.md
> index 7201c484f899..16c14e933c80 100644
> --- a/CHANGELOG.md
> +++ b/CHANGELOG.md
> @@ -12,6 +12,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
>    - On x86:
>      - Option to attempt to fixup p2m page-faults on PVH dom0.
>      - Resizable BARs is supported for PVH dom0.
> +   - Smoke tests for the FreeBSD Xen builds in Cirrus CI.
>   
>   ### Removed
>   

[-- Attachment #2: Type: text/html, Size: 4038 bytes --]

Re: [PATCH 8/8] automation/cirrus-ci: add smoke tests for the FreeBSD builds

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

On Mon, 17 Mar 2025, Roger Pau Monne wrote:
> Introduce a basic set of smoke tests using the XTF selftest image, and run
> them on QEMU.  Use the matrix keyword to create a different task for each
> XTF flavor on each FreeBSD build.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
> I've used the current version of FreeBSD to run the tests, but it might be
> safer to use the latest production (release) version instead.

It is better to avoid using "current" (I called it "latest" in my
previous email) if it is rolling. Otherwise, it is OK.

Other than that, it is great.

Re: [PATCH 8/8] automation/cirrus-ci: add smoke tests for the FreeBSD builds

[Roger Pau Monné]

On Mon, Mar 17, 2025 at 05:09:25PM -0700, Stefano Stabellini wrote:
> On Mon, 17 Mar 2025, Roger Pau Monne wrote:
> > Introduce a basic set of smoke tests using the XTF selftest image, and run
> > them on QEMU.  Use the matrix keyword to create a different task for each
> > XTF flavor on each FreeBSD build.
> > 
> > Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> > ---
> > I've used the current version of FreeBSD to run the tests, but it might be
> > safer to use the latest production (release) version instead.
> 
> It is better to avoid using "current" (I called it "latest" in my
> previous email) if it is rolling. Otherwise, it is OK.

Yes, will adjust to use production instead of current and resend the
tail of the series.

> Other than that, it is great.

Thanks.  Afterwards I would like to add a specific UBSAN enabled
build, and do some more testing with it.  I found clang UBSAN more
complete than gcc (specially with the UB pointer arithmetic checks).

Roger.