[PATCH for-4.19 v2 00/12] CI: part 3 (slimline, and PPC/RISCV fixes)

[PATCH for-4.19 v2 00/12] CI: part 3 (slimline, and PPC/RISCV fixes)

[Andrew Cooper]

Patch 1 fixes a bug in the containerize script

Patches 2-5 remove useless/obsolete testing

Patch 6 fixes a bug with the archlinux testing configuration

Patches 7-9 adjust the PPC64 testing

Patches 10-11 adjust the RISCV64 testing

Patch 12 refreshes the custom GCC-IBT container


All changes here follow best-guidance for dockerfiles (non-root, heredocs for
improved legibility), and naming consistency improvements discussed previously
on the Committers call.  Breifly that's:

  $DISTRO-$VERSION(numeric)-$ARCH-*

which sort more nicely and don't require e.g. people to remember which order
buster/bullseye/bookworm are in terms of debain release.

For x86, this involves inserting an $ARCH of x86_64.  Right now, x86_64 is
implied by the absence, and all other variations have to state one.


Still to come:

 * OpenSUSE container fixes (resolves the final rolling distro vs
   allowed_failure problem)
 * Ubuntu/debian rename/trim
 * Add new jobs for latest releases (Ubuntu, Alpine, Fedora)

Andrew Cooper (12):
  CI: Fix CONTAINER_UID0=1 scripts/containerize

  CI: Remove useless/misleading randconfig jobs
  CI: Drop Debian Jessie dockerfiles
  CI: Drop Debian Stretch testing
  CI: Drop Ubuntu Trusty testing

  CI: Mark Archlinux/x86 as allowing failures

  CI: Introduce a debian:12-ppc64le container
  CI: Use debian:12-ppc64le for both build and test
  CI: Refresh bullseye-ppc64le as debian:11-ppc64le

  CI: Introduce debian:11/12-riscv64 containers
  CI: Swap to debian for riscv64 build and test

  CI: Refresh and upgrade the GCC-IBT container

 .../archlinux/current-riscv64.dockerfile      |  22 --
 automation/build/debian/11-ppc64le.dockerfile |  33 +++
 automation/build/debian/11-riscv64.dockerfile |  33 +++
 automation/build/debian/12-ppc64le.dockerfile |  36 +++
 automation/build/debian/12-riscv64.dockerfile |  36 +++
 ...ockerfile => 12-x86_64-gcc-ibt.dockerfile} |  81 +++---
 .../build/debian/bullseye-ppc64le.dockerfile  |  32 ---
 .../build/debian/jessie-i386.dockerfile       |  55 -----
 automation/build/debian/jessie.dockerfile     |  52 ----
 .../build/debian/stretch-i386.dockerfile      |  57 -----
 automation/build/debian/stretch.dockerfile    |  60 -----
 automation/build/ubuntu/trusty.dockerfile     |  51 ----
 automation/gitlab-ci/build.yaml               | 233 +++++++-----------
 automation/gitlab-ci/test.yaml                |  13 +-
 automation/scripts/containerize               |  15 +-
 automation/scripts/qemu-smoke-ppc64le.sh      |   4 +-
 .../qemu-system-ppc64/8.1.0-ppc64.dockerfile  |  38 ---
 17 files changed, 282 insertions(+), 569 deletions(-)
 delete mode 100644 automation/build/archlinux/current-riscv64.dockerfile
 create mode 100644 automation/build/debian/11-ppc64le.dockerfile
 create mode 100644 automation/build/debian/11-riscv64.dockerfile
 create mode 100644 automation/build/debian/12-ppc64le.dockerfile
 create mode 100644 automation/build/debian/12-riscv64.dockerfile
 rename automation/build/debian/{buster-gcc-ibt.dockerfile => 12-x86_64-gcc-ibt.dockerfile} (50%)
 delete mode 100644 automation/build/debian/bullseye-ppc64le.dockerfile
 delete mode 100644 automation/build/debian/jessie-i386.dockerfile
 delete mode 100644 automation/build/debian/jessie.dockerfile
 delete mode 100644 automation/build/debian/stretch-i386.dockerfile
 delete mode 100644 automation/build/debian/stretch.dockerfile
 delete mode 100644 automation/build/ubuntu/trusty.dockerfile
 delete mode 100644 automation/tests-artifacts/qemu-system-ppc64/8.1.0-ppc64.dockerfile

-- 
2.39.2

[PATCH 01/12] CI: Fix CONTAINER_UID0=1 scripts/containerize

[Andrew Cooper]

Right now, most build containers use root.  Archlinux, Fedora and Yocto set up
a regular user called `user`.

For those containers, trying to containerize as root fails, because
CONTAINER_UID0=1 does nothing, whereas CONTAINER_UID0=0 forces the user away
from root.

To make CONTAINER_UID0=1 work reliably, force to root if requested.

Fixes: 17fbe6504dfd ("automation: introduce a new variable to control container user")
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>

v2:
 * Add fixes tag
---
 automation/scripts/containerize | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index acdef1b54813..03bc4837350c 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -51,7 +51,7 @@ esac
 
 # Use this variable to control whether root should be used
 case "_${CONTAINER_UID0}" in
-    _1)   userarg= ;;
+    _1)   userarg="-u 0" ;;
     _0|_) userarg="-u $(id -u) $userns_podman" ;;
 esac
 
-- 
2.39.2

[PATCH 02/12] CI: Remove useless/misleading randconfig jobs

[Andrew Cooper]

Randconfig builds pick CONFIG_DEBUG with 50% probability.  Therefore
$foo{,-debug}-randconfig are two identical jobs with misleading names.

Furthermore, arm64 has a randconfig jobs for both the alpine and bookworm
build environments.  Both use GCC 12.2, so we have 4 identical jobs.  Delete 3
of them.

This leaves us with 4 primary randconfig jobs:

  gitlab-ci/build.yaml:381:debian-bookworm-gcc-arm32-randconfig:
  gitlab-ci/build.yaml:429:alpine-3.18-gcc-arm64-randconfig:
  gitlab-ci/build.yaml:495:archlinux-current-gcc-riscv64-randconfig:
  gitlab-ci/build.yaml:640:debian-bookworm-gcc-randconfig:

as PPC64 doesn't want randconfig right now, and buster-gcc-ibt is a special
job with a custom compiler.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>

Here is an example debug-randconfig with a release builds:
  https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7312848876

If we want to run multiple identical randconfig jobs, that's spelt

  parallel: 5

in the configuration, and here is an example of what such a run looks like:
  https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1368824041

Notice the randconfig jobs have a 5 in place of a retry button, and show a
submenu when clicked on.
---
 automation/gitlab-ci/build.yaml | 39 ---------------------------------
 1 file changed, 39 deletions(-)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index ff5c9055d1f0..da7b8885aa30 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -385,13 +385,6 @@ debian-bookworm-gcc-arm32-randconfig:
     HYPERVISOR_ONLY: y
     RANDCONFIG: y
 
-debian-bookworm-gcc-arm32-debug-randconfig:
-  extends: .gcc-arm32-cross-build-debug
-  variables:
-    CONTAINER: debian:bookworm-arm64v8-arm32-gcc
-    HYPERVISOR_ONLY: y
-    RANDCONFIG: y
-
 debian-bookworm-gcc-arm32-debug-staticmem:
   extends: .gcc-arm32-cross-build-debug
   variables:
@@ -423,18 +416,6 @@ debian-bookworm-gcc-debug-arm64:
   variables:
     CONTAINER: debian:bookworm-arm64v8
 
-debian-bookworm-gcc-arm64-randconfig:
-  extends: .gcc-arm64-build
-  variables:
-    CONTAINER: debian:bookworm-arm64v8
-    RANDCONFIG: y
-
-debian-bookworm-gcc-debug-arm64-randconfig:
-  extends: .gcc-arm64-build-debug
-  variables:
-    CONTAINER: debian:bookworm-arm64v8
-    RANDCONFIG: y
-
 alpine-3.18-gcc-arm64:
   extends: .gcc-arm64-build
   variables:
@@ -451,12 +432,6 @@ alpine-3.18-gcc-arm64-randconfig:
     CONTAINER: alpine:3.18-arm64v8
     RANDCONFIG: y
 
-alpine-3.18-gcc-debug-arm64-randconfig:
-  extends: .gcc-arm64-build-debug
-  variables:
-    CONTAINER: alpine:3.18-arm64v8
-    RANDCONFIG: y
-
 alpine-3.18-gcc-debug-arm64-staticmem:
   extends: .gcc-arm64-build-debug
   variables:
@@ -525,14 +500,6 @@ archlinux-current-gcc-riscv64-randconfig:
     RANDCONFIG: y
     <<: *riscv-fixed-randconfig
 
-archlinux-current-gcc-riscv64-debug-randconfig:
-  extends: .gcc-riscv64-cross-build-debug
-  variables:
-    CONTAINER: archlinux:current-riscv64
-    KBUILD_DEFCONFIG: tiny64_defconfig
-    RANDCONFIG: y
-    <<: *riscv-fixed-randconfig
-
 # Power cross-build
 debian-bullseye-gcc-ppc64le:
   extends: .gcc-ppc64le-cross-build
@@ -676,12 +643,6 @@ debian-bookworm-gcc-randconfig:
     CONTAINER: debian:bookworm
     RANDCONFIG: y
 
-debian-bookworm-gcc-debug-randconfig:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: debian:bookworm
-    RANDCONFIG: y
-
 debian-bookworm-32-clang-debug:
   extends: .clang-x86-32-build-debug
   variables:
-- 
2.39.2

[PATCH 03/12] CI: Drop Debian Jessie dockerfiles

[Andrew Cooper]

These were removed from testing in Xen 4.18.

Fixes: 3817e3c1b4b8 ("automation: Remove testing on Debian Jessie")
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
---
 .../build/debian/jessie-i386.dockerfile       | 55 -------------------
 automation/build/debian/jessie.dockerfile     | 52 ------------------
 automation/scripts/containerize               |  2 -
 3 files changed, 109 deletions(-)
 delete mode 100644 automation/build/debian/jessie-i386.dockerfile
 delete mode 100644 automation/build/debian/jessie.dockerfile

diff --git a/automation/build/debian/jessie-i386.dockerfile b/automation/build/debian/jessie-i386.dockerfile
deleted file mode 100644
index dfa6dae06cb8..000000000000
--- a/automation/build/debian/jessie-i386.dockerfile
+++ /dev/null
@@ -1,55 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/i386 debian/eol:jessie
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-ENTRYPOINT ["linux32"]
-
-# replace repos in archive as release is EOL
-COPY <<"END" /etc/apt/sources.list
-deb http://archive.debian.org/debian/ jessie main contrib non-free
-deb http://archive.debian.org/debian/ jessie-backports main contrib non-free
-deb http://archive.debian.org/debian-security/ jessie/updates main contrib non-free
-END
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        zlib1g-dev \
-        libncurses5-dev \
-        python-dev \
-        uuid-dev \
-        libyajl-dev \
-        libaio-dev \
-        libglib2.0-dev \
-        clang \
-        libpixman-1-dev \
-        pkg-config \
-        flex \
-        bison \
-        acpica-tools \
-        bin86 \
-        bcc \
-        liblzma-dev \
-        libc6-dev \
-        libnl-3-dev \
-        ocaml-nox \
-        libfindlib-ocaml-dev \
-        markdown \
-        transfig \
-        pandoc \
-        checkpolicy \
-        wget \
-        git \
-        nasm \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/build/debian/jessie.dockerfile b/automation/build/debian/jessie.dockerfile
deleted file mode 100644
index 82dae9ec70c4..000000000000
--- a/automation/build/debian/jessie.dockerfile
+++ /dev/null
@@ -1,52 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 debian/eol:jessie
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-# replace repos in archive as release is EOL
-COPY <<"END" /etc/apt/sources.list
-deb http://archive.debian.org/debian/ jessie main contrib non-free
-deb http://archive.debian.org/debian/ jessie-backports main contrib non-free
-deb http://archive.debian.org/debian-security/ jessie/updates main contrib non-free
-END
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        zlib1g-dev \
-        libncurses5-dev \
-        python-dev \
-        uuid-dev \
-        libyajl-dev \
-        libaio-dev \
-        libglib2.0-dev \
-        clang \
-        libpixman-1-dev \
-        pkg-config \
-        flex \
-        bison \
-        acpica-tools \
-        bin86 \
-        bcc \
-        liblzma-dev \
-        libnl-3-dev \
-        ocaml-nox \
-        libfindlib-ocaml-dev \
-        markdown \
-        transfig \
-        pandoc \
-        checkpolicy \
-        wget \
-        git \
-        nasm \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 03bc4837350c..57db42d10cb5 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -31,8 +31,6 @@ case "_${CONTAINER}" in
     _centos7) CONTAINER="${BASE}/centos:7" ;;
     _fedora) CONTAINER="${BASE}/fedora:29";;
     _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
-    _jessie) CONTAINER="${BASE}/debian:jessie" ;;
-    _jessie-i386) CONTAINER="${BASE}/debian:jessie-i386" ;;
     _bullseye-ppc64le) CONTAINER="${BASE}/debian:bullseye-ppc64le" ;;
     _stretch|_) CONTAINER="${BASE}/debian:stretch" ;;
     _stretch-i386) CONTAINER="${BASE}/debian:stretch-i386" ;;
-- 
2.39.2

[PATCH 04/12] CI: Drop Debian Stretch testing

[Andrew Cooper]

Debian stretch is also End of Life.  Update a couple of test steps to use
bookworm instead.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
---
 .../build/debian/stretch-i386.dockerfile      | 57 ------------------
 automation/build/debian/stretch.dockerfile    | 60 -------------------
 automation/gitlab-ci/build.yaml               | 34 +----------
 automation/gitlab-ci/test.yaml                |  4 +-
 automation/scripts/containerize               |  2 -
 5 files changed, 4 insertions(+), 153 deletions(-)
 delete mode 100644 automation/build/debian/stretch-i386.dockerfile
 delete mode 100644 automation/build/debian/stretch.dockerfile

diff --git a/automation/build/debian/stretch-i386.dockerfile b/automation/build/debian/stretch-i386.dockerfile
deleted file mode 100644
index 905096ab0ef1..000000000000
--- a/automation/build/debian/stretch-i386.dockerfile
+++ /dev/null
@@ -1,57 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/i386 debian:stretch
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-ENTRYPOINT ["linux32"]
-
-# replace repos in archive as release is EOL
-COPY <<"END" /etc/apt/sources.list
-deb http://archive.debian.org/debian/ stretch main contrib non-free
-deb http://archive.debian.org/debian/ stretch-backports main contrib non-free
-deb http://archive.debian.org/debian-security/ stretch/updates main contrib non-free
-END
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        zlib1g-dev \
-        libncurses5-dev \
-        python-dev \
-        python3-dev \
-        uuid-dev \
-        libyajl-dev \
-        libaio-dev \
-        libglib2.0-dev \
-        clang \
-        libpixman-1-dev \
-        pkg-config \
-        flex \
-        bison \
-        acpica-tools \
-        bin86 \
-        bcc \
-        liblzma-dev \
-        libc6-dev \
-        libnl-3-dev \
-        ocaml-nox \
-        libfindlib-ocaml-dev \
-        markdown \
-        transfig \
-        pandoc \
-        checkpolicy \
-        wget \
-        git \
-        nasm \
-        apt-transport-https \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/build/debian/stretch.dockerfile b/automation/build/debian/stretch.dockerfile
deleted file mode 100644
index b99d229acf14..000000000000
--- a/automation/build/debian/stretch.dockerfile
+++ /dev/null
@@ -1,60 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 debian:stretch
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-# replace repos in archive as release is EOL
-COPY <<"END" /etc/apt/sources.list
-deb http://archive.debian.org/debian/ stretch main contrib non-free
-deb http://archive.debian.org/debian/ stretch-backports main contrib non-free
-deb http://archive.debian.org/debian-security/ stretch/updates main contrib non-free
-END
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        zlib1g-dev \
-        libncurses5-dev \
-        python-dev \
-        python3-dev \
-        uuid-dev \
-        libyajl-dev \
-        libaio-dev \
-        libglib2.0-dev \
-        clang \
-        libpixman-1-dev \
-        pkg-config \
-        flex \
-        bison \
-        acpica-tools \
-        bin86 \
-        bcc \
-        liblzma-dev \
-        libnl-3-dev \
-        ocaml-nox \
-        libfindlib-ocaml-dev \
-        markdown \
-        transfig \
-        pandoc \
-        checkpolicy \
-        wget \
-        git \
-        nasm \
-        gnupg \
-        apt-transport-https \
-        # for test phase, qemu-smoke-* jobs
-        qemu-system-x86 \
-        # for test phase, qemu-alpine-* jobs
-        cpio \
-        busybox-static \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index da7b8885aa30..2c6eb17bef50 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -354,10 +354,10 @@ alpine-3.18-gcc-debug:
   variables:
     CONTAINER: alpine:3.18
 
-debian-stretch-gcc-debug:
+debian-bookworm-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: debian:stretch
+    CONTAINER: debian:bookworm
 
 debian-bookworm-clang-debug:
   extends: .clang-x86-64-build-debug
@@ -589,31 +589,6 @@ centos-7-gcc-debug:
   variables:
     CONTAINER: centos:7
 
-debian-stretch-gcc:
-  extends: .gcc-x86-64-build
-  variables:
-    CONTAINER: debian:stretch
-
-debian-stretch-clang:
-  extends: .clang-x86-64-build
-  variables:
-    CONTAINER: debian:stretch
-
-debian-stretch-clang-debug:
-  extends: .clang-x86-64-build-debug
-  variables:
-    CONTAINER: debian:stretch
-
-debian-stretch-32-clang-debug:
-  extends: .clang-x86-32-build-debug
-  variables:
-    CONTAINER: debian:stretch-i386
-
-debian-stretch-32-gcc-debug:
-  extends: .gcc-x86-32-build-debug
-  variables:
-    CONTAINER: debian:stretch-i386
-
 debian-buster-gcc-ibt:
   extends: .gcc-x86-64-build
   variables:
@@ -632,11 +607,6 @@ debian-bookworm-gcc:
   variables:
     CONTAINER: debian:bookworm
 
-debian-bookworm-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: debian:bookworm
-
 debian-bookworm-gcc-randconfig:
   extends: .gcc-x86-64-build
   variables:
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 898d2adc8c5b..d50e522f9f79 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -412,7 +412,7 @@ qemu-smoke-x86-64-gcc:
   script:
     - ./automation/scripts/qemu-smoke-x86-64.sh pv 2>&1 | tee ${LOGFILE}
   needs:
-    - debian-stretch-gcc-debug
+    - debian-bookworm-gcc-debug
 
 qemu-smoke-x86-64-clang:
   extends: .qemu-x86-64
@@ -426,7 +426,7 @@ qemu-smoke-x86-64-gcc-pvh:
   script:
     - ./automation/scripts/qemu-smoke-x86-64.sh pvh 2>&1 | tee ${LOGFILE}
   needs:
-    - debian-stretch-gcc-debug
+    - debian-bookworm-gcc-debug
 
 qemu-smoke-x86-64-clang-pvh:
   extends: .qemu-x86-64
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 57db42d10cb5..5140765cee98 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -32,8 +32,6 @@ case "_${CONTAINER}" in
     _fedora) CONTAINER="${BASE}/fedora:29";;
     _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
     _bullseye-ppc64le) CONTAINER="${BASE}/debian:bullseye-ppc64le" ;;
-    _stretch|_) CONTAINER="${BASE}/debian:stretch" ;;
-    _stretch-i386) CONTAINER="${BASE}/debian:stretch-i386" ;;
     _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
     _bookworm|_) CONTAINER="${BASE}/debian:bookworm" ;;
     _bookworm-i386) CONTAINER="${BASE}/debian:bookworm-i386" ;;
-- 
2.39.2

[PATCH 05/12] CI: Drop Ubuntu Trusty testing

[Andrew Cooper]

This is also End of Life.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
---
 automation/build/ubuntu/trusty.dockerfile | 51 -----------------------
 automation/gitlab-ci/build.yaml           | 12 ------
 automation/scripts/containerize           |  1 -
 3 files changed, 64 deletions(-)
 delete mode 100644 automation/build/ubuntu/trusty.dockerfile

diff --git a/automation/build/ubuntu/trusty.dockerfile b/automation/build/ubuntu/trusty.dockerfile
deleted file mode 100644
index 1cda1bbd3fea..000000000000
--- a/automation/build/ubuntu/trusty.dockerfile
+++ /dev/null
@@ -1,51 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 ubuntu:14.04
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        zlib1g-dev \
-        libncurses5-dev \
-        python-dev \
-        python3-dev \
-        uuid-dev \
-        libyajl-dev \
-        libaio-dev \
-        libglib2.0-dev \
-        clang \
-        libpixman-1-dev \
-        pkg-config \
-        flex \
-        bison \
-        acpica-tools \
-        bin86 \
-        bcc \
-        liblzma-dev \
-        libnl-3-dev \
-        ocaml-nox \
-        libfindlib-ocaml-dev \
-        markdown \
-        transfig \
-        pandoc \
-        checkpolicy \
-        wget \
-        git \
-        nasm \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
-
-# Remove expired certificate that Let's Encrypt certificates used to relie on.
-# (Not needed anymore)
-RUN sed -i 's#mozilla/DST_Root_CA_X3\.crt#!\0#' /etc/ca-certificates.conf && \
-    update-ca-certificates
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 2c6eb17bef50..3e299f34f461 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -633,18 +633,6 @@ fedora-gcc-debug:
   variables:
     CONTAINER: fedora:29
 
-# Ubuntu Trusty's Clang is 3.4 while Xen requires 3.5
-
-ubuntu-trusty-gcc:
-  extends: .gcc-x86-64-build
-  variables:
-    CONTAINER: ubuntu:trusty
-
-ubuntu-trusty-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: ubuntu:trusty
-
 ubuntu-xenial-clang:
   extends: .clang-x86-64-build
   variables:
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 5140765cee98..e9ad20aa0c80 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -39,7 +39,6 @@ case "_${CONTAINER}" in
     _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
     _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
-    _trusty) CONTAINER="${BASE}/ubuntu:trusty" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/suse:opensuse-tumbleweed" ;;
-- 
2.39.2

[PATCH 06/12] CI: Mark Archlinux/x86 as allowing failures

[Andrew Cooper]

Archlinux is a rolling distro, and is subject to arbitrary changes as the
container is rebuilt.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
---
 automation/gitlab-ci/build.yaml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 3e299f34f461..1446bc1d0ea7 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -573,11 +573,13 @@ archlinux-gcc:
   extends: .gcc-x86-64-build
   variables:
     CONTAINER: archlinux:current
+  allow_failure: true
 
 archlinux-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
     CONTAINER: archlinux:current
+  allow_failure: true
 
 centos-7-gcc:
   extends: .gcc-x86-64-build
-- 
2.39.2

[PATCH 07/12] CI: Introduce a debian:12-ppc64le container

[Andrew Cooper]

Make it non-root by default, and set XEN_TARGET_ARCH=ppc64 too.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
---
 automation/build/debian/12-ppc64le.dockerfile | 36 +++++++++++++++++++
 automation/scripts/containerize               |  1 +
 2 files changed, 37 insertions(+)
 create mode 100644 automation/build/debian/12-ppc64le.dockerfile

diff --git a/automation/build/debian/12-ppc64le.dockerfile b/automation/build/debian/12-ppc64le.dockerfile
new file mode 100644
index 000000000000..3b311e675ef5
--- /dev/null
+++ b/automation/build/debian/12-ppc64le.dockerfile
@@ -0,0 +1,36 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 debian:bookworm-slim
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV CROSS_COMPILE=powerpc64le-linux-gnu-
+ENV XEN_TARGET_ARCH=ppc64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        flex
+        gcc-powerpc64le-linux-gnu
+        python3-minimal
+
+        # Qemu for test phase
+        qemu-system-ppc
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index e9ad20aa0c80..aa3433f0dc98 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -32,6 +32,7 @@ case "_${CONTAINER}" in
     _fedora) CONTAINER="${BASE}/fedora:29";;
     _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
     _bullseye-ppc64le) CONTAINER="${BASE}/debian:bullseye-ppc64le" ;;
+    _bookworm-ppc64le) CONTAINER="${BASE}/debian:12-ppc64le" ;;
     _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
     _bookworm|_) CONTAINER="${BASE}/debian:bookworm" ;;
     _bookworm-i386) CONTAINER="${BASE}/debian:bookworm-i386" ;;
-- 
2.39.2

[PATCH 08/12] CI: Use debian:12-ppc64le for both build and test

[Andrew Cooper]

qemu-system-ppc64/8.1.0-ppc64 was added because bullseye's Qemu didn't
understand the powernv9 machine.

However, bookworm's Qemu does, and this is preforable to maintaining a random
build of Qemu ourselves.

Add bookworm builds, and retain bullseye to keep our lower-bound toolchain
testing.

Rename the jobs to follow to sort coherently ($DISTRO-$VERSION-$ARCH-*) and
reposition the jobs to optimise starting the smoke test.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
---
 automation/gitlab-ci/build.yaml               | 59 ++++++++++---------
 automation/gitlab-ci/test.yaml                |  5 +-
 automation/scripts/qemu-smoke-ppc64le.sh      |  4 +-
 .../qemu-system-ppc64/8.1.0-ppc64.dockerfile  | 38 ------------
 4 files changed, 34 insertions(+), 72 deletions(-)
 delete mode 100644 automation/tests-artifacts/qemu-system-ppc64/8.1.0-ppc64.dockerfile

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 1446bc1d0ea7..316dc4345f29 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -302,20 +302,6 @@ qemu-system-aarch64-6.0.0-arm32-export:
   tags:
     - arm64
 
-# ppc64 test artifacts
-
-qemu-system-ppc64-8.1.0-ppc64-export:
-  extends: .test-jobs-artifact-common
-  image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-ppc64:8.1.0-ppc64
-  script:
-    - mkdir binaries && cp /qemu-system-ppc64 /skiboot.lid binaries/
-  artifacts:
-    paths:
-      - binaries/qemu-system-ppc64
-      - binaries/skiboot.lid
-  tags:
-    - x86_64
-
 # x86_64 test artifacts
 
 alpine-3.18-rootfs-export:
@@ -364,6 +350,13 @@ debian-bookworm-clang-debug:
   variables:
     CONTAINER: debian:bookworm
 
+debian-12-ppc64le-gcc-debug:
+  extends: .gcc-ppc64le-cross-build-debug
+  variables:
+    CONTAINER: debian:12-ppc64le
+    KBUILD_DEFCONFIG: ppc64_defconfig
+    HYPERVISOR_ONLY: y
+
 # Arm32 cross-build
 
 debian-bookworm-gcc-arm32:
@@ -500,21 +493,6 @@ archlinux-current-gcc-riscv64-randconfig:
     RANDCONFIG: y
     <<: *riscv-fixed-randconfig
 
-# Power cross-build
-debian-bullseye-gcc-ppc64le:
-  extends: .gcc-ppc64le-cross-build
-  variables:
-    CONTAINER: debian:bullseye-ppc64le
-    KBUILD_DEFCONFIG: ppc64_defconfig
-    HYPERVISOR_ONLY: y
-
-debian-bullseye-gcc-ppc64le-debug:
-  extends: .gcc-ppc64le-cross-build-debug
-  variables:
-    CONTAINER: debian:bullseye-ppc64le
-    KBUILD_DEFCONFIG: ppc64_defconfig
-    HYPERVISOR_ONLY: y
-
 # Yocto test jobs
 yocto-qemuarm64:
   extends: .yocto-test-arm64
@@ -738,3 +716,26 @@ opensuse-tumbleweed-gcc-debug:
   variables:
     CONTAINER: suse:opensuse-tumbleweed
   allow_failure: true
+
+# PowerPC builds (x86 cross)
+debian-bullseye-gcc-ppc64le:
+  extends: .gcc-ppc64le-cross-build
+  variables:
+    CONTAINER: debian:bullseye-ppc64le
+    KBUILD_DEFCONFIG: ppc64_defconfig
+    HYPERVISOR_ONLY: y
+
+debian-bullseye-gcc-ppc64le-debug:
+  extends: .gcc-ppc64le-cross-build-debug
+  variables:
+    CONTAINER: debian:bullseye-ppc64le
+    KBUILD_DEFCONFIG: ppc64_defconfig
+    HYPERVISOR_ONLY: y
+
+debian-12-ppc64le-gcc:
+  extends: .gcc-ppc64le-cross-build
+  variables:
+    CONTAINER: debian:12-ppc64le
+    KBUILD_DEFCONFIG: ppc64_defconfig
+    HYPERVISOR_ONLY: y
+
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index d50e522f9f79..70c946f87799 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -69,7 +69,7 @@
 .qemu-ppc64le:
   extends: .test-jobs-common
   variables:
-    CONTAINER: debian:bullseye-ppc64le
+    CONTAINER: debian:12-ppc64le
     LOGFILE: qemu-smoke-ppc64le.log
   artifacts:
     paths:
@@ -447,5 +447,4 @@ qemu-smoke-ppc64le-powernv9-gcc:
   script:
     - ./automation/scripts/qemu-smoke-ppc64le.sh powernv9 2>&1 | tee ${LOGFILE}
   needs:
-    - qemu-system-ppc64-8.1.0-ppc64-export
-    - debian-bullseye-gcc-ppc64le-debug
+    - debian-12-ppc64le-gcc-debug
diff --git a/automation/scripts/qemu-smoke-ppc64le.sh b/automation/scripts/qemu-smoke-ppc64le.sh
index 701d81107d18..9088881b731b 100755
--- a/automation/scripts/qemu-smoke-ppc64le.sh
+++ b/automation/scripts/qemu-smoke-ppc64le.sh
@@ -12,8 +12,8 @@ rm -f ${serial_log}
 set +e
 
 timeout -k 1 20 \
-binaries/qemu-system-ppc64 \
-    -bios binaries/skiboot.lid \
+qemu-system-ppc64 \
+    -bios skiboot.lid \
     -M $machine \
     -m 2g \
     -smp 1 \
diff --git a/automation/tests-artifacts/qemu-system-ppc64/8.1.0-ppc64.dockerfile b/automation/tests-artifacts/qemu-system-ppc64/8.1.0-ppc64.dockerfile
deleted file mode 100644
index e28d68649239..000000000000
--- a/automation/tests-artifacts/qemu-system-ppc64/8.1.0-ppc64.dockerfile
+++ /dev/null
@@ -1,38 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 debian:bullseye-slim
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV QEMU_VERSION=8.1.0
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        curl \
-        python3 \
-        python3-pip \
-        python3-elementpath \
-        ninja-build \
-        pkg-config \
-        libglib2.0-dev \
-        libpixman-1-dev \
-        && \
-    \
-    curl -fsSLO https://download.qemu.org/qemu-"$QEMU_VERSION".tar.xz && \
-    tar xvJf qemu-"$QEMU_VERSION".tar.xz && \
-    cd qemu-"$QEMU_VERSION" && \
-    ./configure --target-list=ppc64-softmmu && \
-    make -j$(nproc) && \
-    cp ./build/qemu-system-ppc64 / && \
-    cp ./build/qemu-bundle/usr/local/share/qemu/skiboot.lid / && \
-    cd /build && \
-    rm -rf qemu-"$QEMU_VERSION"* && \
-    apt-get autoremove -y && \
-    apt-get clean && \
-    rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
-- 
2.39.2

[PATCH 09/12] CI: Refresh bullseye-ppc64le as debian:11-ppc64le

[Andrew Cooper]

... in the style of bookworm-ppc64le

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
---
 automation/build/debian/11-ppc64le.dockerfile | 33 +++++++++++++++++++
 .../build/debian/bullseye-ppc64le.dockerfile  | 32 ------------------
 automation/gitlab-ci/build.yaml               |  8 ++---
 automation/scripts/containerize               |  2 +-
 4 files changed, 38 insertions(+), 37 deletions(-)
 create mode 100644 automation/build/debian/11-ppc64le.dockerfile
 delete mode 100644 automation/build/debian/bullseye-ppc64le.dockerfile

diff --git a/automation/build/debian/11-ppc64le.dockerfile b/automation/build/debian/11-ppc64le.dockerfile
new file mode 100644
index 000000000000..d846b670d02c
--- /dev/null
+++ b/automation/build/debian/11-ppc64le.dockerfile
@@ -0,0 +1,33 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 debian:bullseye-slim
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV CROSS_COMPILE=powerpc64le-linux-gnu-
+ENV XEN_TARGET_ARCH=ppc64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        flex
+        gcc-powerpc64le-linux-gnu
+        python3-minimal
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/debian/bullseye-ppc64le.dockerfile b/automation/build/debian/bullseye-ppc64le.dockerfile
deleted file mode 100644
index e166d205f38c..000000000000
--- a/automation/build/debian/bullseye-ppc64le.dockerfile
+++ /dev/null
@@ -1,32 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 debian:bullseye-slim
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-# Add compiler path
-ENV CROSS_COMPILE powerpc64le-linux-gnu-
-
-RUN mkdir /build
-WORKDIR /build
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes --no-install-recommends install \
-        bison \
-        build-essential \
-        checkpolicy \
-        flex \
-        gawk \
-        gcc-powerpc64le-linux-gnu \
-        make \
-        python3-minimal \
-        # QEMU runtime dependencies for test phase
-        libglib2.0-0 \
-        libpixman-1-0 \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 316dc4345f29..e081664c4e95 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -718,17 +718,17 @@ opensuse-tumbleweed-gcc-debug:
   allow_failure: true
 
 # PowerPC builds (x86 cross)
-debian-bullseye-gcc-ppc64le:
+debian-11-ppc64le-gcc:
   extends: .gcc-ppc64le-cross-build
   variables:
-    CONTAINER: debian:bullseye-ppc64le
+    CONTAINER: debian:11-ppc64le
     KBUILD_DEFCONFIG: ppc64_defconfig
     HYPERVISOR_ONLY: y
 
-debian-bullseye-gcc-ppc64le-debug:
+debian-11-ppc64le-gcc-debug:
   extends: .gcc-ppc64le-cross-build-debug
   variables:
-    CONTAINER: debian:bullseye-ppc64le
+    CONTAINER: debian:11-ppc64le
     KBUILD_DEFCONFIG: ppc64_defconfig
     HYPERVISOR_ONLY: y
 
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index aa3433f0dc98..5c3f6782d9cd 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -31,7 +31,7 @@ case "_${CONTAINER}" in
     _centos7) CONTAINER="${BASE}/centos:7" ;;
     _fedora) CONTAINER="${BASE}/fedora:29";;
     _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
-    _bullseye-ppc64le) CONTAINER="${BASE}/debian:bullseye-ppc64le" ;;
+    _bullseye-ppc64le) CONTAINER="${BASE}/debian:11-ppc64le" ;;
     _bookworm-ppc64le) CONTAINER="${BASE}/debian:12-ppc64le" ;;
     _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
     _bookworm|_) CONTAINER="${BASE}/debian:bookworm" ;;
-- 
2.39.2

[PATCH 10/12] CI: Introduce debian:11/12-riscv64 containers

[Andrew Cooper]

For starters, they're slightly smaller:

  $ docker image list <snip>
  registry.gitlab.com/xen-project/xen/debian      12-riscv64         772MB
  registry.gitlab.com/xen-project/xen/debian      11-riscv64         422MB
  registry.gitlab.com/xen-project/xen/archlinux   current-riscv64    2.32GB

They also not rolling distros, so will be more predicatable testing for the
stable trees in the future.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
---
 automation/build/debian/11-riscv64.dockerfile | 33 +++++++++++++++++
 automation/build/debian/12-riscv64.dockerfile | 36 +++++++++++++++++++
 2 files changed, 69 insertions(+)
 create mode 100644 automation/build/debian/11-riscv64.dockerfile
 create mode 100644 automation/build/debian/12-riscv64.dockerfile

diff --git a/automation/build/debian/11-riscv64.dockerfile b/automation/build/debian/11-riscv64.dockerfile
new file mode 100644
index 000000000000..1c99bc89ea1a
--- /dev/null
+++ b/automation/build/debian/11-riscv64.dockerfile
@@ -0,0 +1,33 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 debian:bullseye-slim
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV CROSS_COMPILE=riscv64-linux-gnu-
+ENV XEN_TARGET_ARCH=riscv64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        flex
+        gcc-riscv64-linux-gnu
+        python3-minimal
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/debian/12-riscv64.dockerfile b/automation/build/debian/12-riscv64.dockerfile
new file mode 100644
index 000000000000..03fd4b03059b
--- /dev/null
+++ b/automation/build/debian/12-riscv64.dockerfile
@@ -0,0 +1,36 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 debian:bookworm-slim
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV CROSS_COMPILE=riscv64-linux-gnu-
+ENV XEN_TARGET_ARCH=riscv64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        flex
+        gcc-riscv64-linux-gnu
+        python3-minimal
+
+        # Qemu for test phase
+        qemu-system-riscv64
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
-- 
2.39.2

[PATCH 11/12] CI: Swap to debian for riscv64 build and test

[Andrew Cooper]

The containers are both much smaller, with stable toolchains over time, and
this at least means we're not doing all testing with a single compiler.

Rename the jobs to follow to sort coherently ($DISTRO-$VERSION-$ARCH-*) and
reposition the jobs to optimise starting the smoke test.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
---
 .../archlinux/current-riscv64.dockerfile      | 22 -----
 automation/gitlab-ci/build.yaml               | 83 +++++++++++--------
 automation/gitlab-ci/test.yaml                |  4 +-
 automation/scripts/containerize               |  3 +-
 4 files changed, 52 insertions(+), 60 deletions(-)
 delete mode 100644 automation/build/archlinux/current-riscv64.dockerfile

diff --git a/automation/build/archlinux/current-riscv64.dockerfile b/automation/build/archlinux/current-riscv64.dockerfile
deleted file mode 100644
index f7770bf82a78..000000000000
--- a/automation/build/archlinux/current-riscv64.dockerfile
+++ /dev/null
@@ -1,22 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 archlinux
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-# Packages needed for the build
-RUN pacman --noconfirm --needed -Syu \
-    base-devel \
-    git \
-    inetutils \
-    riscv64-linux-gnu-binutils \
-    riscv64-linux-gnu-gcc \
-    riscv64-linux-gnu-glibc \
-    # For test phase
-    qemu-system-riscv
-
-# Add compiler path
-ENV CROSS_COMPILE=riscv64-linux-gnu-
-
-RUN useradd --create-home user
-USER user
-WORKDIR /build
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index e081664c4e95..4b9d80cc5632 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -357,6 +357,13 @@ debian-12-ppc64le-gcc-debug:
     KBUILD_DEFCONFIG: ppc64_defconfig
     HYPERVISOR_ONLY: y
 
+debian-12-riscv64-gcc-debug:
+  extends: .gcc-riscv64-cross-build-debug
+  variables:
+    CONTAINER: debian:12-riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    HYPERVISOR_ONLY: y
+
 # Arm32 cross-build
 
 debian-bookworm-gcc-arm32:
@@ -458,41 +465,6 @@ alpine-3.18-gcc-debug-arm64-earlyprintk:
       CONFIG_EARLY_UART_CHOICE_PL011=y
       CONFIG_EARLY_UART_BASE_ADDRESS=0x9000000
 
-# RISC-V 64 cross-build
-.riscv-fixed-randconfig:
-  variables: &riscv-fixed-randconfig
-    EXTRA_FIXED_RANDCONFIG: |
-      CONFIG_BOOT_TIME_CPUPOOLS=n
-      CONFIG_COVERAGE=n
-      CONFIG_EXPERT=y
-      CONFIG_GRANT_TABLE=n
-      CONFIG_MEM_ACCESS=n
-      CONFIG_PERF_COUNTERS=n
-      CONFIG_LIVEPATCH=n
-      CONFIG_XSM=n
-
-archlinux-current-gcc-riscv64:
-  extends: .gcc-riscv64-cross-build
-  variables:
-    CONTAINER: archlinux:current-riscv64
-    KBUILD_DEFCONFIG: tiny64_defconfig
-    HYPERVISOR_ONLY: y
-
-archlinux-current-gcc-riscv64-debug:
-  extends: .gcc-riscv64-cross-build-debug
-  variables:
-    CONTAINER: archlinux:current-riscv64
-    KBUILD_DEFCONFIG: tiny64_defconfig
-    HYPERVISOR_ONLY: y
-
-archlinux-current-gcc-riscv64-randconfig:
-  extends: .gcc-riscv64-cross-build
-  variables:
-    CONTAINER: archlinux:current-riscv64
-    KBUILD_DEFCONFIG: tiny64_defconfig
-    RANDCONFIG: y
-    <<: *riscv-fixed-randconfig
-
 # Yocto test jobs
 yocto-qemuarm64:
   extends: .yocto-test-arm64
@@ -739,3 +711,44 @@ debian-12-ppc64le-gcc:
     KBUILD_DEFCONFIG: ppc64_defconfig
     HYPERVISOR_ONLY: y
 
+# RISC-V 64 cross-build
+debian-11-riscv64-gcc:
+  extends: .gcc-riscv64-cross-build
+  variables:
+    CONTAINER: debian:11-riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    HYPERVISOR_ONLY: y
+
+debian-11-riscv64-gcc-debug:
+  extends: .gcc-riscv64-cross-build-debug
+  variables:
+    CONTAINER: debian:11-riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    HYPERVISOR_ONLY: y
+
+debian-12-riscv64-gcc:
+  extends: .gcc-riscv64-cross-build
+  variables:
+    CONTAINER: debian:12-riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    HYPERVISOR_ONLY: y
+
+.riscv-fixed-randconfig:
+  variables: &riscv-fixed-randconfig
+    EXTRA_FIXED_RANDCONFIG: |
+      CONFIG_BOOT_TIME_CPUPOOLS=n
+      CONFIG_COVERAGE=n
+      CONFIG_EXPERT=y
+      CONFIG_GRANT_TABLE=n
+      CONFIG_MEM_ACCESS=n
+      CONFIG_PERF_COUNTERS=n
+      CONFIG_LIVEPATCH=n
+      CONFIG_XSM=n
+
+debian-12-riscv64-gcc-randconfig:
+  extends: .gcc-riscv64-cross-build
+  variables:
+    CONTAINER: debian:12-riscv64
+    KBUILD_DEFCONFIG: tiny64_defconfig
+    RANDCONFIG: y
+    <<: *riscv-fixed-randconfig
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 70c946f87799..f7e1753ead34 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -56,7 +56,7 @@
 .qemu-riscv64:
   extends: .test-jobs-common
   variables:
-    CONTAINER: archlinux:current-riscv64
+    CONTAINER: debian:12
     LOGFILE: qemu-smoke-riscv64.log
   artifacts:
     paths:
@@ -440,7 +440,7 @@ qemu-smoke-riscv64-gcc:
   script:
     - ./automation/scripts/qemu-smoke-riscv64.sh 2>&1 | tee ${LOGFILE}
   needs:
-    - archlinux-current-gcc-riscv64-debug
+    - debian-12-riscv64-gcc-debug
 
 qemu-smoke-ppc64le-powernv9-gcc:
   extends: .qemu-ppc64le
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 5c3f6782d9cd..6dbf55c8f89a 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -27,12 +27,13 @@ case "_${CONTAINER}" in
     _alpine) CONTAINER="${BASE}/alpine:3.18" ;;
     _alpine-arm64v8) CONTAINER="${BASE}/alpine:3.18-arm64v8" ;;
     _archlinux|_arch) CONTAINER="${BASE}/archlinux:current" ;;
-    _riscv64) CONTAINER="${BASE}/archlinux:current-riscv64" ;;
     _centos7) CONTAINER="${BASE}/centos:7" ;;
     _fedora) CONTAINER="${BASE}/fedora:29";;
     _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
     _bullseye-ppc64le) CONTAINER="${BASE}/debian:11-ppc64le" ;;
     _bookworm-ppc64le) CONTAINER="${BASE}/debian:12-ppc64le" ;;
+    _bullseye-riscv64) CONTAINER="${BASE}/debian:11-riscv64" ;;
+    _bookworm-riscv64) CONTAINER="${BASE}/debian:12-riscv64" ;;
     _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
     _bookworm|_) CONTAINER="${BASE}/debian:bookworm" ;;
     _bookworm-i386) CONTAINER="${BASE}/debian:bookworm-i386" ;;
-- 
2.39.2

[PATCH 12/12] CI: Refresh and upgrade the GCC-IBT container

[Andrew Cooper]

Upgrade from Debian buster to bookworm, GCC 11.3 to 11.4 and to be a non-root
container.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
---
 ...ockerfile => 12-x86_64-gcc-ibt.dockerfile} | 81 +++++++++++--------
 automation/gitlab-ci/build.yaml               |  4 +-
 automation/scripts/containerize               |  2 +-
 3 files changed, 49 insertions(+), 38 deletions(-)
 rename automation/build/debian/{buster-gcc-ibt.dockerfile => 12-x86_64-gcc-ibt.dockerfile} (50%)

diff --git a/automation/build/debian/buster-gcc-ibt.dockerfile b/automation/build/debian/12-x86_64-gcc-ibt.dockerfile
similarity index 50%
rename from automation/build/debian/buster-gcc-ibt.dockerfile
rename to automation/build/debian/12-x86_64-gcc-ibt.dockerfile
index ed9367aafbec..3ce60e7d9466 100644
--- a/automation/build/debian/buster-gcc-ibt.dockerfile
+++ b/automation/build/debian/12-x86_64-gcc-ibt.dockerfile
@@ -1,28 +1,33 @@
 # syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 debian:buster-slim AS builder
+FROM --platform=linux/amd64 debian:bookworm-slim AS builder
 
 ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
 
-RUN apt-get update && \
-    apt-get --quiet --yes --no-install-recommends install \
-        bison \
-        build-essential \
-        ca-certificates \
-        flex \
-        g++-multilib \
-        libc6-dev-i386 \
-        libgmp-dev \
-        libisl-dev \
-        libmpc-dev \
-        libmpfr-dev \
-        patch \
+RUN <<EOF
+#!/bin/bash
+    set -e
+    apt-get -y update
+
+    DEPS=(
+        bison
+        build-essential
+        ca-certificates
+        flex
+        g++-multilib
+        libc6-dev-i386
+        libgmp-dev
+        libisl-dev
+        libmpc-dev
+        libmpfr-dev
+        patch
         wget
+    )
 
-RUN mkdir /build
-WORKDIR /build
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
 
-RUN wget -q https://ftp.gnu.org/gnu/gcc/gcc-11.3.0/gcc-11.3.0.tar.xz -O - | tar xJ --strip=1
+RUN wget -q https://ftp.gnu.org/gnu/gcc/gcc-11.4.0/gcc-11.4.0.tar.xz -O - | tar xJ --strip=1
 RUN wget -q https://xenbits.xen.org/people/andrewcoop/gcc-11.2-Add-fcf-check-attribute-yes-no.patch -O - | patch -p1
 RUN ./configure \
         --prefix=/opt/gcc-11-ibt \
@@ -42,29 +47,35 @@ RUN ./configure \
 RUN make -j`nproc` && make -j`nproc` install
 
 
-FROM --platform=linux/amd64 debian:buster-slim
+FROM --platform=linux/amd64 debian:bookworm-slim
 COPY --from=builder /opt/gcc-11-ibt /opt/gcc-11-ibt
 
 LABEL maintainer.name="The Xen Project" \
       maintainer.email="xen-devel@lists.xenproject.org"
 
 ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
 ENV PATH="/opt/gcc-11-ibt/bin:${PATH}"
 
-RUN mkdir /build
-WORKDIR /build
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
 
-RUN apt-get update && \
-    apt-get --quiet --yes --no-install-recommends install \
-        bison \
-        build-essential \
-        checkpolicy \
-        flex \
-        gawk \
-        make \
-        python3-minimal \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        flex
+        python3-minimal
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 4b9d80cc5632..b4139414bc8e 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -541,10 +541,10 @@ centos-7-gcc-debug:
   variables:
     CONTAINER: centos:7
 
-debian-buster-gcc-ibt:
+debian-12-x86_64-gcc-ibt:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: debian:buster-gcc-ibt
+    CONTAINER: debian:12-x86_64-gcc-ibt
     RANDCONFIG: y
     EXTRA_FIXED_RANDCONFIG: |
       CONFIG_XEN_IBT=y
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 6dbf55c8f89a..4d5669c5ad01 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -34,7 +34,7 @@ case "_${CONTAINER}" in
     _bookworm-ppc64le) CONTAINER="${BASE}/debian:12-ppc64le" ;;
     _bullseye-riscv64) CONTAINER="${BASE}/debian:11-riscv64" ;;
     _bookworm-riscv64) CONTAINER="${BASE}/debian:12-riscv64" ;;
-    _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
+    _bookworm-x86_64-gcc-ibt) CONTAINER="${BASE}/debian:12-x86_64-gcc-ibt" ;;
     _bookworm|_) CONTAINER="${BASE}/debian:bookworm" ;;
     _bookworm-i386) CONTAINER="${BASE}/debian:bookworm-i386" ;;
     _bookworm-arm64v8-arm32-gcc) CONTAINER="${BASE}/debian:bookworm-arm64v8-arm32-gcc" ;;
-- 
2.39.2

Re: [PATCH 01/12] CI: Fix CONTAINER_UID0=1 scripts/containerize

[Anthony PERARD]

On Thu, Jul 11, 2024 at 12:15:06PM +0100, Andrew Cooper wrote:
> Right now, most build containers use root.  Archlinux, Fedora and Yocto set up
> a regular user called `user`.
> 
> For those containers, trying to containerize as root fails, because
> CONTAINER_UID0=1 does nothing, whereas CONTAINER_UID0=0 forces the user away
> from root.
> 
> To make CONTAINER_UID0=1 work reliably, force to root if requested.
> 
> Fixes: 17fbe6504dfd ("automation: introduce a new variable to control container user")
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 02/12] CI: Remove useless/misleading randconfig jobs

[Anthony PERARD]

On Thu, Jul 11, 2024 at 12:15:07PM +0100, Andrew Cooper wrote:
> Randconfig builds pick CONFIG_DEBUG with 50% probability.  Therefore
> $foo{,-debug}-randconfig are two identical jobs with misleading names.
> 
> Furthermore, arm64 has a randconfig jobs for both the alpine and bookworm
> build environments.  Both use GCC 12.2, so we have 4 identical jobs.  Delete 3
> of them.
> 
> This leaves us with 4 primary randconfig jobs:
> 
>   gitlab-ci/build.yaml:381:debian-bookworm-gcc-arm32-randconfig:
>   gitlab-ci/build.yaml:429:alpine-3.18-gcc-arm64-randconfig:
>   gitlab-ci/build.yaml:495:archlinux-current-gcc-riscv64-randconfig:
>   gitlab-ci/build.yaml:640:debian-bookworm-gcc-randconfig:
> 
> as PPC64 doesn't want randconfig right now, and buster-gcc-ibt is a special
> job with a custom compiler.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

> If we want to run multiple identical randconfig jobs, that's spelt
> 
>   parallel: 5
> 
> in the configuration, and here is an example of what such a run looks like:
>   https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1368824041
> 
> Notice the randconfig jobs have a 5 in place of a retry button, and show a
> submenu when clicked on.

We already have "parallel: 2" on the whole pipeline, give or take, but
those are quick jobs so it might be worth it to run them a few more time
on a pipeline.

Cheers,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 04/12] CI: Drop Debian Stretch testing

[Anthony PERARD]

On Thu, Jul 11, 2024 at 12:15:09PM +0100, Andrew Cooper wrote:
> Debian stretch is also End of Life.  Update a couple of test steps to use
> bookworm instead.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 05/12] CI: Drop Ubuntu Trusty testing

[Anthony PERARD]

On Thu, Jul 11, 2024 at 12:15:10PM +0100, Andrew Cooper wrote:
> This is also End of Life.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH for-4.19 v2 00/12] CI: part 3 (slimline, and PPC/RISCV fixes)

[Oleksii]

Release-Acked-By: Oleksii Kurochko <oleksii.kurochko@gmail.com>

~ Oleksii

On Thu, 2024-07-11 at 12:15 +0100, Andrew Cooper wrote:
> Patch 1 fixes a bug in the containerize script
> 
> Patches 2-5 remove useless/obsolete testing
> 
> Patch 6 fixes a bug with the archlinux testing configuration
> 
> Patches 7-9 adjust the PPC64 testing
> 
> Patches 10-11 adjust the RISCV64 testing
> 
> Patch 12 refreshes the custom GCC-IBT container
> 
> 
> All changes here follow best-guidance for dockerfiles (non-root,
> heredocs for
> improved legibility), and naming consistency improvements discussed
> previously
> on the Committers call.  Breifly that's:
> 
>   $DISTRO-$VERSION(numeric)-$ARCH-*
> 
> which sort more nicely and don't require e.g. people to remember
> which order
> buster/bullseye/bookworm are in terms of debain release.
> 
> For x86, this involves inserting an $ARCH of x86_64.  Right now,
> x86_64 is
> implied by the absence, and all other variations have to state one.
> 
> 
> Still to come:
> 
>  * OpenSUSE container fixes (resolves the final rolling distro vs
>    allowed_failure problem)
>  * Ubuntu/debian rename/trim
>  * Add new jobs for latest releases (Ubuntu, Alpine, Fedora)
> 
> Andrew Cooper (12):
>   CI: Fix CONTAINER_UID0=1 scripts/containerize
> 
>   CI: Remove useless/misleading randconfig jobs
>   CI: Drop Debian Jessie dockerfiles
>   CI: Drop Debian Stretch testing
>   CI: Drop Ubuntu Trusty testing
> 
>   CI: Mark Archlinux/x86 as allowing failures
> 
>   CI: Introduce a debian:12-ppc64le container
>   CI: Use debian:12-ppc64le for both build and test
>   CI: Refresh bullseye-ppc64le as debian:11-ppc64le
> 
>   CI: Introduce debian:11/12-riscv64 containers
>   CI: Swap to debian for riscv64 build and test
> 
>   CI: Refresh and upgrade the GCC-IBT container
> 
>  .../archlinux/current-riscv64.dockerfile      |  22 --
>  automation/build/debian/11-ppc64le.dockerfile |  33 +++
>  automation/build/debian/11-riscv64.dockerfile |  33 +++
>  automation/build/debian/12-ppc64le.dockerfile |  36 +++
>  automation/build/debian/12-riscv64.dockerfile |  36 +++
>  ...ockerfile => 12-x86_64-gcc-ibt.dockerfile} |  81 +++---
>  .../build/debian/bullseye-ppc64le.dockerfile  |  32 ---
>  .../build/debian/jessie-i386.dockerfile       |  55 -----
>  automation/build/debian/jessie.dockerfile     |  52 ----
>  .../build/debian/stretch-i386.dockerfile      |  57 -----
>  automation/build/debian/stretch.dockerfile    |  60 -----
>  automation/build/ubuntu/trusty.dockerfile     |  51 ----
>  automation/gitlab-ci/build.yaml               | 233 +++++++---------
> --
>  automation/gitlab-ci/test.yaml                |  13 +-
>  automation/scripts/containerize               |  15 +-
>  automation/scripts/qemu-smoke-ppc64le.sh      |   4 +-
>  .../qemu-system-ppc64/8.1.0-ppc64.dockerfile  |  38 ---
>  17 files changed, 282 insertions(+), 569 deletions(-)
>  delete mode 100644 automation/build/archlinux/current-
> riscv64.dockerfile
>  create mode 100644 automation/build/debian/11-ppc64le.dockerfile
>  create mode 100644 automation/build/debian/11-riscv64.dockerfile
>  create mode 100644 automation/build/debian/12-ppc64le.dockerfile
>  create mode 100644 automation/build/debian/12-riscv64.dockerfile
>  rename automation/build/debian/{buster-gcc-ibt.dockerfile => 12-
> x86_64-gcc-ibt.dockerfile} (50%)
>  delete mode 100644 automation/build/debian/bullseye-
> ppc64le.dockerfile
>  delete mode 100644 automation/build/debian/jessie-i386.dockerfile
>  delete mode 100644 automation/build/debian/jessie.dockerfile
>  delete mode 100644 automation/build/debian/stretch-i386.dockerfile
>  delete mode 100644 automation/build/debian/stretch.dockerfile
>  delete mode 100644 automation/build/ubuntu/trusty.dockerfile
>  delete mode 100644 automation/tests-artifacts/qemu-system-
> ppc64/8.1.0-ppc64.dockerfile
> 

Re: [PATCH 06/12] CI: Mark Archlinux/x86 as allowing failures

[Anthony PERARD]

On Thu, Jul 11, 2024 at 12:15:11PM +0100, Andrew Cooper wrote:
> Archlinux is a rolling distro, and is subject to arbitrary changes as the
> container is rebuilt.

I disagree about the changes been arbitrary, it is just new version of
software. :-)

But I guess "arbirary changes" could be a description for package been
removed from the main/official archive, and left for the users to build
themselfs.

> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Anyway:
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

(Next, automatically rebuild the container every month, I could probably
revive my old patch series doing that)

Cheers,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 07/12] CI: Introduce a debian:12-ppc64le container

[Anthony PERARD]

On Thu, Jul 11, 2024 at 12:15:12PM +0100, Andrew Cooper wrote:
> diff --git a/automation/build/debian/12-ppc64le.dockerfile b/automation/build/debian/12-ppc64le.dockerfile
> new file mode 100644
> index 000000000000..3b311e675ef5
> --- /dev/null
> +++ b/automation/build/debian/12-ppc64le.dockerfile
> @@ -0,0 +1,36 @@
...
> +ENV CROSS_COMPILE=powerpc64le-linux-gnu-
> +ENV XEN_TARGET_ARCH=ppc64

I don't really like these two envvars. The second one is already set in
the env by gitlab, the first one could in theory also come from
gitlab-ci.

But I guess that comment comes from the fact that I don't really know
why we would want a second bookworm containers with only a few packages.

> +
> +RUN <<EOF
> +#!/bin/bash
> +    set -e
> +
> +    useradd --create-home user
> +
> +    apt-get -y update
> +
> +    DEPS=(
> +        # Xen
> +        bison
> +        build-essential
> +        checkpolicy
> +        flex
> +        gcc-powerpc64le-linux-gnu
> +        python3-minimal
> +
> +        # Qemu for test phase
> +        qemu-system-ppc
> +    )
> +
> +    apt-get -y --no-install-recommends install "${DEPS[@]}"
> +    rm -rf /var/lib/apt/lists/*

Other containers do also `apt-get clean` is this not needed? Or maybe
the original containers is setup to not save packages?

/me looking into /var/cache

Looks empty enough, so I guess `apt-get clean` is optional.

Changes looks good enough, so:
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Cheers,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 02/12] CI: Remove useless/misleading randconfig jobs

[Andrew Cooper]

On 11/07/2024 2:09 pm, Anthony PERARD wrote:
> On Thu, Jul 11, 2024 at 12:15:07PM +0100, Andrew Cooper wrote:
>> Randconfig builds pick CONFIG_DEBUG with 50% probability.  Therefore
>> $foo{,-debug}-randconfig are two identical jobs with misleading names.
>>
>> Furthermore, arm64 has a randconfig jobs for both the alpine and bookworm
>> build environments.  Both use GCC 12.2, so we have 4 identical jobs.  Delete 3
>> of them.
>>
>> This leaves us with 4 primary randconfig jobs:
>>
>>   gitlab-ci/build.yaml:381:debian-bookworm-gcc-arm32-randconfig:
>>   gitlab-ci/build.yaml:429:alpine-3.18-gcc-arm64-randconfig:
>>   gitlab-ci/build.yaml:495:archlinux-current-gcc-riscv64-randconfig:
>>   gitlab-ci/build.yaml:640:debian-bookworm-gcc-randconfig:
>>
>> as PPC64 doesn't want randconfig right now, and buster-gcc-ibt is a special
>> job with a custom compiler.
>>
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks.

>
> Thanks,
>
>> If we want to run multiple identical randconfig jobs, that's spelt
>>
>>   parallel: 5
>>
>> in the configuration, and here is an example of what such a run looks like:
>>   https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1368824041
>>
>> Notice the randconfig jobs have a 5 in place of a retry button, and show a
>> submenu when clicked on.
> We already have "parallel: 2" on the whole pipeline, give or take, but
> those are quick jobs so it might be worth it to run them a few more time
> on a pipeline.

I'm afraid I don't follow.

The runners themselves are set up to run concurrent jobs.  I could well
believe that's spelt parallel: in the runner config files.

Here, its really jobs:parallel which is the way of saying "I want 5
copies of this job to run in the pipeline".

Obviously, 5 is arbitrary.  I was thinking of having a scheduled (once
per day, at 3AM or whenever) running randconfig only with a parallel:20
or so, so we can make good use of otherwise-idle time.

~Andrew

Re: [PATCH 06/12] CI: Mark Archlinux/x86 as allowing failures

[Andrew Cooper]

On 11/07/2024 2:37 pm, Anthony PERARD wrote:
> On Thu, Jul 11, 2024 at 12:15:11PM +0100, Andrew Cooper wrote:
>> Archlinux is a rolling distro, and is subject to arbitrary changes as the
>> container is rebuilt.
> I disagree about the changes been arbitrary, it is just new version of
> software. :-)
>
> But I guess "arbirary changes" could be a description for package been
> removed from the main/official archive, and left for the users to build
> themselfs.

Specifically, rebuilding the container for staging has a side effect of
jumping the toolchain version under all stable trees.

I'll see about tweaking the wording.


>
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Anyway:
> Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks.

> (Next, automatically rebuild the container every month, I could probably
> revive my old patch series doing that)

That's also on the list, but not for 4.19.  Qemu rebuild their
containers as the first pipeline stage, and while I think that's
overkill, I was going to see if I could turn it into a weekly(?)
scheduled job.

~Andrew

Re: [PATCH 03/12] CI: Drop Debian Jessie dockerfiles

[Anthony PERARD]

On Thu, Jul 11, 2024 at 12:15:08PM +0100, Andrew Cooper wrote:
> These were removed from testing in Xen 4.18.
> 
> Fixes: 3817e3c1b4b8 ("automation: Remove testing on Debian Jessie")
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 02/12] CI: Remove useless/misleading randconfig jobs

[Anthony PERARD]

On Thu, Jul 11, 2024 at 04:16:29PM +0100, Andrew Cooper wrote:
> On 11/07/2024 2:09 pm, Anthony PERARD wrote:
> > We already have "parallel: 2" on the whole pipeline, give or take, but
> > those are quick jobs so it might be worth it to run them a few more time
> > on a pipeline.
>
> I'm afraid I don't follow.

Sorry, I just meant that the whole test suite is run twice, once in
https://gitlab.com/xen-project/hardware/xen/-/pipelines
and once in
https://gitlab.com/xen-project/xen/-/pipelines
(with just a few tests that are differents)

> Obviously, 5 is arbitrary.  I was thinking of having a scheduled (once
> per day, at 3AM or whenever) running randconfig only with a parallel:20
> or so, so we can make good use of otherwise-idle time.

That sounds like a good idea.

Cheers,

--

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 07/12] CI: Introduce a debian:12-ppc64le container

[Andrew Cooper]

On 11/07/2024 3:41 pm, Anthony PERARD wrote:
> On Thu, Jul 11, 2024 at 12:15:12PM +0100, Andrew Cooper wrote:
>> diff --git a/automation/build/debian/12-ppc64le.dockerfile b/automation/build/debian/12-ppc64le.dockerfile
>> new file mode 100644
>> index 000000000000..3b311e675ef5
>> --- /dev/null
>> +++ b/automation/build/debian/12-ppc64le.dockerfile
>> @@ -0,0 +1,36 @@
> ...
>> +ENV CROSS_COMPILE=powerpc64le-linux-gnu-
>> +ENV XEN_TARGET_ARCH=ppc64
> I don't really like these two envvars. The second one is already set in
> the env by gitlab, the first one could in theory also come from
> gitlab-ci.

The first one is pre-existing.  (It shows as new because enough of the
file changed for it not to be deemed a rename).

The second is something I added, for improved usability.

> But I guess that comment comes from the fact that I don't really know
> why we would want a second bookworm containers with only a few packages.

Right now (for better or worse) - we have single purpose containers for
a single arch.

For PPC and RISCV, where we're only doing a cross-build of the
hypervisor, the dependencies are almost identical.  But the moment we
start doing userspace, it will diverge quickly.

Part of the work I'm doing is trimming the containers down.  Container
size is a relevant factor for how long the CI tests take.  Also, with
smaller containers, the "oops, out of disk space, delete everything"
logic on various runners will trigger less frequently, so we get better
cache utilisation.

We do not IMO want to have containers with a union of all deps of all
architectures.  They'd be giant.

So while we do have single-purpose containers, these ENV vars are
~mandatory for using the container for it's intended purpose, and it's
far more usable for a human when you can just enter the container and
type `make`.
>> +
>> +RUN <<EOF
>> +#!/bin/bash
>> +    set -e
>> +
>> +    useradd --create-home user
>> +
>> +    apt-get -y update
>> +
>> +    DEPS=(
>> +        # Xen
>> +        bison
>> +        build-essential
>> +        checkpolicy
>> +        flex
>> +        gcc-powerpc64le-linux-gnu
>> +        python3-minimal
>> +
>> +        # Qemu for test phase
>> +        qemu-system-ppc
>> +    )
>> +
>> +    apt-get -y --no-install-recommends install "${DEPS[@]}"
>> +    rm -rf /var/lib/apt/lists/*
> Other containers do also `apt-get clean` is this not needed? Or maybe
> the original containers is setup to not save packages?
>
> /me looking into /var/cache
>
> Looks empty enough, so I guess `apt-get clean` is optional.

https://docs.docker.com/build/building/best-practices/ notes that the
official Debian and Ubuntu images automatically clean themselves, so you
don't need to do it manually.

https://github.com/moby/moby/blob/03e2923e42446dbb830c654d0eec323a0b4ef02a/contrib/mkimage/debootstrap#L82-L105

> Changes looks good enough, so:
> Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks.

~Andrew

Re: [PATCH 02/12] CI: Remove useless/misleading randconfig jobs

[Andrew Cooper]

On 11/07/2024 4:50 pm, Anthony PERARD wrote:
> On Thu, Jul 11, 2024 at 04:16:29PM +0100, Andrew Cooper wrote:
>> On 11/07/2024 2:09 pm, Anthony PERARD wrote:
>>> We already have "parallel: 2" on the whole pipeline, give or take, but
>>> those are quick jobs so it might be worth it to run them a few more time
>>> on a pipeline.
>> I'm afraid I don't follow.
> Sorry, I just meant that the whole test suite is run twice, once in
> https://gitlab.com/xen-project/hardware/xen/-/pipelines
> and once in
> https://gitlab.com/xen-project/xen/-/pipelines
> (with just a few tests that are differents)

And the rest.  Every time I re-sync my tree, and probably the same for
the other maintainers.

However, these have nothing on the overhead from patchew, which would be
far more palatable if we didn't essentially throw away the result by not
sending it anywhere useful.

~Andrew

Re: [PATCH 08/12] CI: Use debian:12-ppc64le for both build and test

[Anthony PERARD]

On Thu, Jul 11, 2024 at 12:15:13PM +0100, Andrew Cooper wrote:
> qemu-system-ppc64/8.1.0-ppc64 was added because bullseye's Qemu didn't

QEMU is normally spelled "QEMU" unless we are in camel case context, or
lower case context.

> understand the powernv9 machine.
> 
> However, bookworm's Qemu does, and this is preforable to maintaining a random

"preforable", that's an intended mispell right? :-)

> build of Qemu ourselves.
> 
> Add bookworm builds, and retain bullseye to keep our lower-bound toolchain
> testing.
> 
> Rename the jobs to follow to sort coherently ($DISTRO-$VERSION-$ARCH-*) and

There's no renamed jobs in this patch, but the new jobs does follow the
new naming schema. Did you intend to rename both
"debian-bullseye-gcc-ppc64le*" jobs?

> reposition the jobs to optimise starting the smoke test.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Patch is mostly ok, so:
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 09/12] CI: Refresh bullseye-ppc64le as debian:11-ppc64le

[Anthony PERARD]

On Thu, Jul 11, 2024 at 12:15:14PM +0100, Andrew Cooper wrote:
> ... in the style of bookworm-ppc64le
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 10/12] CI: Introduce debian:11/12-riscv64 containers

[Anthony PERARD]

On Thu, Jul 11, 2024 at 12:15:15PM +0100, Andrew Cooper wrote:
> For starters, they're slightly smaller:
> 
>   $ docker image list <snip>
>   registry.gitlab.com/xen-project/xen/debian      12-riscv64         772MB
>   registry.gitlab.com/xen-project/xen/debian      11-riscv64         422MB
>   registry.gitlab.com/xen-project/xen/archlinux   current-riscv64    2.32GB
> 
> They also not rolling distros, so will be more predicatable testing for the
> stable trees in the future.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 11/12] CI: Swap to debian for riscv64 build and test

[Anthony PERARD]

On Thu, Jul 11, 2024 at 12:15:16PM +0100, Andrew Cooper wrote:
> The containers are both much smaller, with stable toolchains over time, and
> this at least means we're not doing all testing with a single compiler.
> 
> Rename the jobs to follow to sort coherently ($DISTRO-$VERSION-$ARCH-*) and
> reposition the jobs to optimise starting the smoke test.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 12/12] CI: Refresh and upgrade the GCC-IBT container

[Anthony PERARD]

On Thu, Jul 11, 2024 at 12:15:17PM +0100, Andrew Cooper wrote:
> Upgrade from Debian buster to bookworm, GCC 11.3 to 11.4 and to be a non-root
> container.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 08/12] CI: Use debian:12-ppc64le for both build and test

[Andrew Cooper]

On 11/07/2024 5:22 pm, Anthony PERARD wrote:
> On Thu, Jul 11, 2024 at 12:15:13PM +0100, Andrew Cooper wrote:
>> qemu-system-ppc64/8.1.0-ppc64 was added because bullseye's Qemu didn't
> QEMU is normally spelled "QEMU" unless we are in camel case context, or
> lower case context.
>
>> understand the powernv9 machine.
>>
>> However, bookworm's Qemu does, and this is preforable to maintaining a random
> "preforable", that's an intended mispell right? :-)

Yes.  both fixed.

>
>> build of Qemu ourselves.
>>
>> Add bookworm builds, and retain bullseye to keep our lower-bound toolchain
>> testing.
>>
>> Rename the jobs to follow to sort coherently ($DISTRO-$VERSION-$ARCH-*) and
> There's no renamed jobs in this patch, but the new jobs does follow the
> new naming schema.

Oh - good point.  I'll tweak the wording.


>  Did you intend to rename both
> "debian-bullseye-gcc-ppc64le*" jobs?

No - that's done in the following patch.

>
>> reposition the jobs to optimise starting the smoke test.
>>
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Patch is mostly ok, so:
> Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks.

~Andrew

[PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container

[Andrew Cooper]

Existing as suse:opensuse-tumbleweed is a historical quirk, and adjusted for
consistency with all the other containers.

Make it non-root, use heredocs for legibility, and use the zypper long names
for the benefit of those wondering what was being referenced or duplicated.

Trim the dependencies substantially.  Testing docs isn't very interesting and
saves a lot of space (pandoc pulls in all of Haskel).  Other savings come from
removing a huge pile of optional QEMU dependencies (QEMU just needs to build
the Xen parts to be useful here, not have a full GUI environment).

Finally, there where some packages such as bc, libssh2-devel, libtasn1-devel
and nasm that I'm not aware of any reason to have had, even historically.

This shaves 1G off the size of the container:

  registry.gitlab.com/xen-project/xen/opensuse  tumbleweed-x86_64      1.39GB
  registry.gitlab.com/xen-project/xen/suse      opensuse-tumbleweed    2.35GB

Furthermore, identify which components of the build use which dependencies,
which will help managing them in the future.

Thanks to Olaf Hering for dependency fixes that have been subsumed into this
total overhaul.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
CC: Olaf Hering <olaf@aepfle.de>

This includes both of Olaf's recent changes, python3-packaging (now required
for a rebuilt container), and findutils (not required yet; still gets pulled
in transitively), but added for good measure.

In terms of package delta, newly included are:

  findutils            # As suggested
  ocaml-findlib        # Replaces ocaml-findlib-devel
  perl                 # Transitive previously
  python3              # Transitive previously
  python3-packaging    # As suggested

while removed are:

  bc                   # ?
  cmake                # vtpm stubdom, optional
  discount             # docs
  ghostscript          # docs
  glibc-devel          # Not really optional, given a toolchain
  hostname             # Optional, not interesting
  libaio-devel         # Xen < 4.13, perhaps Qemu still but optional
  libext2fs-devel      # Libfsimage, off by default, optional
  libgnutls-devel      # Qemu ?
  libjpeg62-devel      # Qemu ?
  libnl3-devel         # Libxl COLO(?), option
  libnuma-devel        # ?
  libpng16-devel       # Qemu ?
  libssh2-devel        # ?
  libtasn1-devel       # ?
  nasm                 # ?
  ocaml-findlib-devel  # Swapped for non-devel
  ocaml-ocamlbuild     # ?
  ocaml-ocamldoc       # ?
  pandoc               # docs
  'pkgconfig(libpci)'  # Qemu
  'pkgconfig(sdl)'     # Qemu
  'pkgconfig(sdl2)'    # Qemu
  systemd-devel        # Xen < 4.19, optional
  transfig             # docs
  valgrind-devel       # really only for developers
  which                # Xen < 4.17

I've tested this on staging-4.17 and staging, both of which build fine.
staging-4.16 does not, because of SeaBIOS which we've decided not to fix,
hence it's safe to drop `which`.
---
 .../opensuse/tumbleweed-x86_64.dockerfile     | 76 +++++++++++++++++++
 .../build/suse/opensuse-tumbleweed.dockerfile | 74 ------------------
 automation/gitlab-ci/build.yaml               |  8 +-
 automation/scripts/containerize               |  2 +-
 4 files changed, 81 insertions(+), 79 deletions(-)
 create mode 100644 automation/build/opensuse/tumbleweed-x86_64.dockerfile
 delete mode 100644 automation/build/suse/opensuse-tumbleweed.dockerfile

diff --git a/automation/build/opensuse/tumbleweed-x86_64.dockerfile b/automation/build/opensuse/tumbleweed-x86_64.dockerfile
new file mode 100644
index 000000000000..8c5ff56cdc88
--- /dev/null
+++ b/automation/build/opensuse/tumbleweed-x86_64.dockerfile
@@ -0,0 +1,76 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 opensuse/tumbleweed
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV XEN_TARGET_ARCH=x86_64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    zypper refresh
+    zypper dist-upgrade -y --no-recommends
+
+    DEPS=(
+        # Xen
+        bison
+        checkpolicy
+        clang
+        diffutils
+        findutils
+        flex
+        gawk
+        gcc
+        make
+        python3
+
+        # Tools (general)
+        git
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-devel
+        libzstd-devel
+        lzo-devel
+        xz-devel
+        zlib-devel
+        # libacpi
+        acpica
+        # libxl
+        libuuid-devel
+        libyajl-devel
+        # Header Check
+        gcc-c++
+        # xentop
+        ncurses-devel
+        # Python bindings
+        python3-devel
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml
+        ocaml-findlib
+
+        # Stubdom (download/extract)
+        bzip2
+        tar
+
+        # Qemu build
+        glib2-devel
+        libpixman-1-0-devel
+        meson
+        ninja
+        python3-packaging
+    )
+
+    zypper install -y --no-recommends "${DEPS[@]}"
+    zypper clean -a
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/suse/opensuse-tumbleweed.dockerfile b/automation/build/suse/opensuse-tumbleweed.dockerfile
deleted file mode 100644
index 077fed22260c..000000000000
--- a/automation/build/suse/opensuse-tumbleweed.dockerfile
+++ /dev/null
@@ -1,74 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 opensuse/tumbleweed
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-RUN zypper ref && zypper dup -y --no-recommends
-RUN zypper install -y --no-recommends \
-        acpica \
-        bc \
-        bison \
-        bzip2 \
-        checkpolicy \
-        clang \
-        cmake \
-        diffutils \
-        discount \
-        flex \
-        gawk \
-        gcc \
-        gcc-c++ \
-        git \
-        ghostscript \
-        glib2-devel \
-        glibc-devel \
-        gzip \
-        hostname \
-        libaio-devel \
-        libbz2-devel \
-        libext2fs-devel \
-        libgnutls-devel \
-        libjpeg62-devel \
-        libnl3-devel \
-        libnuma-devel \
-        libpixman-1-0-devel \
-        libpng16-devel \
-        libssh2-devel \
-        libtasn1-devel \
-        libuuid-devel \
-        libyajl-devel \
-        libzstd-devel \
-        lzo-devel \
-        make \
-        meson \
-        nasm \
-        ncurses-devel \
-        ninja \
-        ocaml \
-        ocaml-findlib-devel \
-        ocaml-ocamlbuild \
-        ocaml-ocamldoc \
-        pandoc \
-        patch \
-        pkg-config \
-        'pkgconfig(libpci)' \
-        'pkgconfig(sdl)' \
-        'pkgconfig(sdl2)' \
-        python3-devel \
-        python3-setuptools \
-        # systemd-devel for Xen < 4.19
-        systemd-devel \
-        tar \
-        transfig \
-        valgrind-devel \
-        wget \
-        which \
-        xz-devel \
-        zlib-devel \
-        && \
-        zypper clean -a
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index b4139414bc8e..fd6ca1270e4e 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -668,25 +668,25 @@ opensuse-leap-gcc-debug:
 opensuse-tumbleweed-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 # PowerPC builds (x86 cross)
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 4d5669c5ad01..c4f013702122 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -43,7 +43,7 @@ case "_${CONTAINER}" in
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
-    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/suse:opensuse-tumbleweed" ;;
+    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
 esac
 
 # Use this variable to control whether root should be used
-- 
2.39.2

Re: [PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container

[Andrew Cooper]

On 11/07/2024 9:23 pm, Andrew Cooper wrote:
> Existing as suse:opensuse-tumbleweed is a historical quirk, and adjusted for
> consistency with all the other containers.
>
> Make it non-root, use heredocs for legibility, and use the zypper long names
> for the benefit of those wondering what was being referenced or duplicated.
>
> Trim the dependencies substantially.  Testing docs isn't very interesting and
> saves a lot of space (pandoc pulls in all of Haskel).  Other savings come from
> removing a huge pile of optional QEMU dependencies (QEMU just needs to build
> the Xen parts to be useful here, not have a full GUI environment).
>
> Finally, there where some packages such as bc, libssh2-devel, libtasn1-devel
> and nasm that I'm not aware of any reason to have had, even historically.
>
> This shaves 1G off the size of the container:
>
>   registry.gitlab.com/xen-project/xen/opensuse  tumbleweed-x86_64      1.39GB
>   registry.gitlab.com/xen-project/xen/suse      opensuse-tumbleweed    2.35GB
>
> Furthermore, identify which components of the build use which dependencies,
> which will help managing them in the future.
>
> Thanks to Olaf Hering for dependency fixes that have been subsumed into this
> total overhaul.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> CC: Anthony PERARD <anthony.perard@vates.tech>
> CC: Juergen Gross <jgross@suse.com>
> CC: Roger Pau Monné <roger.pau@citrix.com>
> CC: Jan Beulich <JBeulich@suse.com>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Julien Grall <julien@xen.org>
> CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> CC: Shawn Anastasio <sanastasio@raptorengineering.com>
> CC: Olaf Hering <olaf@aepfle.de>
>
> This includes both of Olaf's recent changes, python3-packaging (now required
> for a rebuilt container), and findutils (not required yet; still gets pulled
> in transitively), but added for good measure.
>
> In terms of package delta, newly included are:
>
>   findutils            # As suggested
>   ocaml-findlib        # Replaces ocaml-findlib-devel
>   perl                 # Transitive previously
>   python3              # Transitive previously
>   python3-packaging    # As suggested
>
> while removed are:
>
>   bc                   # ?
>   cmake                # vtpm stubdom, optional
>   discount             # docs
>   ghostscript          # docs
>   glibc-devel          # Not really optional, given a toolchain
>   hostname             # Optional, not interesting
>   libaio-devel         # Xen < 4.13, perhaps Qemu still but optional
>   libext2fs-devel      # Libfsimage, off by default, optional
>   libgnutls-devel      # Qemu ?
>   libjpeg62-devel      # Qemu ?
>   libnl3-devel         # Libxl COLO(?), option
>   libnuma-devel        # ?
>   libpng16-devel       # Qemu ?
>   libssh2-devel        # ?
>   libtasn1-devel       # ?
>   nasm                 # ?
>   ocaml-findlib-devel  # Swapped for non-devel
>   ocaml-ocamlbuild     # ?
>   ocaml-ocamldoc       # ?
>   pandoc               # docs
>   'pkgconfig(libpci)'  # Qemu
>   'pkgconfig(sdl)'     # Qemu
>   'pkgconfig(sdl2)'    # Qemu
>   systemd-devel        # Xen < 4.19, optional
>   transfig             # docs
>   valgrind-devel       # really only for developers
>   which                # Xen < 4.17
>
> I've tested this on staging-4.17 and staging, both of which build fine.
> staging-4.16 does not, because of SeaBIOS which we've decided not to fix,
> hence it's safe to drop `which`.

For good measure, runs with this container:

  4.17: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322567620
  4.19: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322493465

and the expected failure:

  4.16: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322499885

~Andrew

[PATCH 14/12] CI: Refresh OpenSUSE Leap container

[Andrew Cooper]

See prior patch for most discussion.

Despite appearing to a fixed release (and therefore not marked as permitted
failure), the dockerfile references the `leap` tag which is rolling in
practice.  Switch to 15.6 explicitly, for better test stability.

Vs tumbleweed, use `zypper update` rather than dist-upgrade, and retain the
RomBIOS dependencies; bin86 and dev86.

In terms of size, this saves ~700M:

  registry.gitlab.com/xen-project/xen/opensuse  leap-15.6-x86_64       1.33GB
  registry.gitlab.com/xen-project/xen/suse      opensuse-leap          2.05GB

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
CC: Olaf Hering <olaf@aepfle.de>

Runs with this new container:

 4.19:  https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322493460
 4.16:  https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322499881
---
 .../opensuse/leap-15.6-x86_64.dockerfile      | 79 +++++++++++++++++++
 .../build/suse/opensuse-leap.dockerfile       | 75 ------------------
 automation/gitlab-ci/build.yaml               | 16 ++--
 automation/scripts/containerize               |  2 +-
 4 files changed, 88 insertions(+), 84 deletions(-)
 create mode 100644 automation/build/opensuse/leap-15.6-x86_64.dockerfile
 delete mode 100644 automation/build/suse/opensuse-leap.dockerfile

diff --git a/automation/build/opensuse/leap-15.6-x86_64.dockerfile b/automation/build/opensuse/leap-15.6-x86_64.dockerfile
new file mode 100644
index 000000000000..562006672a4e
--- /dev/null
+++ b/automation/build/opensuse/leap-15.6-x86_64.dockerfile
@@ -0,0 +1,79 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 opensuse/leap:15.6
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV XEN_TARGET_ARCH=x86_64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    zypper refresh
+    zypper update -y --no-recommends
+
+    DEPS=(
+        # Xen
+        bison
+        checkpolicy
+        clang
+        diffutils
+        findutils
+        flex
+        gawk
+        gcc
+        make
+        python3
+
+        # Tools (general)
+        git
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-devel
+        libzstd-devel
+        lzo-devel
+        xz-devel
+        zlib-devel
+        # libacpi
+        acpica
+        # libxl
+        libuuid-devel
+        libyajl-devel
+        # RomBIOS
+        bin86
+        dev86
+        # Header Check
+        gcc-c++
+        # xentop
+        ncurses-devel
+        # Python bindings
+        python3-devel
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml
+        ocaml-findlib
+
+        # Stubdom (download/extract)
+        bzip2
+        tar
+
+        # Qemu build
+        glib2-devel
+        libpixman-1-0-devel
+        meson
+        ninja
+        python3-packaging
+    )
+
+    zypper install -y --no-recommends "${DEPS[@]}"
+    zypper clean -a
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/suse/opensuse-leap.dockerfile b/automation/build/suse/opensuse-leap.dockerfile
deleted file mode 100644
index 28d333881233..000000000000
--- a/automation/build/suse/opensuse-leap.dockerfile
+++ /dev/null
@@ -1,75 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 opensuse/leap
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-RUN zypper ref && zypper up -y --no-recommends
-RUN zypper install -y --no-recommends \
-        acpica \
-        bc \
-        bin86 \
-        bison \
-        bzip2 \
-        checkpolicy \
-        clang \
-        cmake \
-        dev86 \
-        diffutils \
-        discount \
-        flex \
-        gawk \
-        gcc \
-        gcc-c++ \
-        git \
-        ghostscript \
-        glib2-devel \
-        glibc-devel \
-        gzip \
-        hostname \
-        libaio-devel \
-        libbz2-devel \
-        libext2fs-devel \
-        libgnutls-devel \
-        libjpeg62-devel \
-        libnl3-devel \
-        libnuma-devel \
-        libpixman-1-0-devel \
-        libpng16-devel \
-        libssh2-devel \
-        libtasn1-devel \
-        libuuid-devel \
-        libyajl-devel \
-        lzo-devel \
-        make \
-        nasm \
-        ncurses-devel \
-        ocaml \
-        ocaml-findlib-devel \
-        ocaml-ocamlbuild \
-        ocaml-ocamldoc \
-        pandoc \
-        patch \
-        pkg-config \
-        'pkgconfig(libpci)' \
-        'pkgconfig(sdl)' \
-        'pkgconfig(sdl2)' \
-        python3-devel \
-        python3-setuptools \
-        # systemd-devel for Xen < 4.19
-        systemd-devel \
-        tar \
-        transfig \
-        valgrind-devel \
-        wget \
-        which \
-        xz-devel \
-        zlib-devel \
-        # QEMU
-        ninja \
-        && \
-        zypper clean -a
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index fd6ca1270e4e..f9a9275ff293 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -645,25 +645,25 @@ ubuntu-focal-clang-debug:
   variables:
     CONTAINER: ubuntu:focal
 
-opensuse-leap-clang:
+opensuse-leap-15.6-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
-opensuse-leap-clang-debug:
+opensuse-leap-15.6-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
-opensuse-leap-gcc:
+opensuse-leap-15.6-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
-opensuse-leap-gcc-debug:
+opensuse-leap-15.6-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
 opensuse-tumbleweed-clang:
   extends: .clang-x86-64-build
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index c4f013702122..d8110b125333 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -42,7 +42,7 @@ case "_${CONTAINER}" in
     _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
-    _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
+    _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
 esac
 
-- 
2.39.2

Re: [PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 1709 bytes --]

Thu, 11 Jul 2024 21:23:37 +0100 Andrew Cooper <andrew.cooper3@citrix.com>:

> use the zypper long names for the benefit of those wondering what was being referenced or duplicated.

Names of rpm packages do change over time, configure already searches for "something" where it is available, instead of some library or header.
If it is touched anyway, I would use more instead of less 'pkgconfig(something)' references:

libbz2-devel
    pkgconfig(bzip2) = 1.0.8
libzstd-devel
    pkgconfig(libzstd) = 1.5.6
lzo-devel
    pkgconfig(lzo2) = 2.10
xz-devel
    pkgconfig(liblzma) = 5.6.2
zlib-devel
    pkgconfig(zlib) = 1.3.1
libuuid-devel
    pkgconfig(uuid) = 2.40.1
libyajl-devel
    pkgconfig(yajl) = 2.1.0
ncurses-devel
    pkgconfig(form) = 6.5.20240629
    pkgconfig(formw) = 6.5.20240629
    pkgconfig(menu) = 6.5.20240629
    pkgconfig(menuw) = 6.5.20240629
    pkgconfig(ncurses) = 6.5.20240629
    pkgconfig(ncurses++) = 6.5.20240629
    pkgconfig(ncurses++w) = 6.5.20240629
    pkgconfig(ncursesw) = 6.5.20240629
    pkgconfig(panel) = 6.5.20240629
    pkgconfig(panelw) = 6.5.20240629
    pkgconfig(tic) = 6.5.20240629
    pkgconfig(tinfo) = 6.5.20240629
glib2-devel
    pkgconfig(gio-2.0) = 2.80.3
    pkgconfig(gio-unix-2.0) = 2.80.3
    pkgconfig(girepository-2.0) = 2.80.3
    pkgconfig(glib-2.0) = 2.80.3
    pkgconfig(gmodule-2.0) = 2.80.3
    pkgconfig(gmodule-export-2.0) = 2.80.3
    pkgconfig(gmodule-no-export-2.0) = 2.80.3
    pkgconfig(gobject-2.0) = 2.80.3
    pkgconfig(gthread-2.0) = 2.80.3
libpixman-1-0-devel
    pkgconfig(pixman-1) = 0.43.4

python3-devel is "special", can be used as is.

'git' could be just 'git-core'


Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container

[Andrew Cooper]

On 11/07/2024 10:49 pm, Olaf Hering wrote:
> Thu, 11 Jul 2024 21:23:37 +0100 Andrew Cooper <andrew.cooper3@citrix.com>:
>
>> use the zypper long names for the benefit of those wondering what was being referenced or duplicated.
> Names of rpm packages do change over time, configure already searches for "something" where it is available,

configure has to cope with distros taking different naming decisions,
and while I appreciate that packages may change over time even within a
distro ...

>  instead of some library or header.
> If it is touched anyway, I would use more instead of less 'pkgconfig(something)' references:
>
> libbz2-devel
>     pkgconfig(bzip2) = 1.0.8
> libzstd-devel
>     pkgconfig(libzstd) = 1.5.6
> lzo-devel
>     pkgconfig(lzo2) = 2.10
> xz-devel
>     pkgconfig(liblzma) = 5.6.2
> zlib-devel
>     pkgconfig(zlib) = 1.3.1
> libuuid-devel
>     pkgconfig(uuid) = 2.40.1
> libyajl-devel
>     pkgconfig(yajl) = 2.1.0
> ncurses-devel
>     pkgconfig(form) = 6.5.20240629
>     pkgconfig(formw) = 6.5.20240629
>     pkgconfig(menu) = 6.5.20240629
>     pkgconfig(menuw) = 6.5.20240629
>     pkgconfig(ncurses) = 6.5.20240629
>     pkgconfig(ncurses++) = 6.5.20240629
>     pkgconfig(ncurses++w) = 6.5.20240629
>     pkgconfig(ncursesw) = 6.5.20240629
>     pkgconfig(panel) = 6.5.20240629
>     pkgconfig(panelw) = 6.5.20240629
>     pkgconfig(tic) = 6.5.20240629
>     pkgconfig(tinfo) = 6.5.20240629
> glib2-devel
>     pkgconfig(gio-2.0) = 2.80.3
>     pkgconfig(gio-unix-2.0) = 2.80.3
>     pkgconfig(girepository-2.0) = 2.80.3
>     pkgconfig(glib-2.0) = 2.80.3
>     pkgconfig(gmodule-2.0) = 2.80.3
>     pkgconfig(gmodule-export-2.0) = 2.80.3
>     pkgconfig(gmodule-no-export-2.0) = 2.80.3
>     pkgconfig(gobject-2.0) = 2.80.3
>     pkgconfig(gthread-2.0) = 2.80.3
> libpixman-1-0-devel
>     pkgconfig(pixman-1) = 0.43.4

... this is just a differently-inconsistent list of names, with some
very dubious aliases such as `form` and `menu`.

What makes pkgconfig() any more stable than the main package name?

Why do we want to be treating libraries different to other packages?

> python3-devel is "special", can be used as is.

As an aside, I looked for a python3-minimal or equivalent, and couldn't
find one.  I take it there isn't one?

> 'git' could be just 'git-core'

Thanks.  Will fix locally, for Tumbleweed and Leap.

~Andrew

Re: [PATCH 13/12] CI: Refresh OpenSUSE Tumbleweed container

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 1471 bytes --]

Thu, 11 Jul 2024 23:39:42 +0100 Andrew Cooper <andrew.cooper3@citrix.com>:

> configure has to cope with distros taking different naming decisions,
> and while I appreciate that packages may change over time even within a
> distro ...

configure does "pkgconfig --exists something" to probe for some API.
I'm sure noone will rename the pkgconfig files, "something.pc" in this case.

> ... this is just a differently-inconsistent list of names, with some
> very dubious aliases such as `form` and `menu`.

I should have truncated that list of names, 'pkgconfig(ncurses)' and
'pkgconfig(glib-2.0)' are relevant.

> What makes pkgconfig() any more stable than the main package name?

They are supposed to describe some API, and in some cases some functionality.
It does not matter much which package provides the API.

> Why do we want to be treating libraries different to other packages?

Because there is no other namespace to describe functionality.
It was implemented for pkg-config, it exists for some cmake and perl
modules.

> > python3-devel is "special", can be used as is.  
> As an aside, I looked for a python3-minimal or equivalent, and couldn't
> find one.  I take it there isn't one?

I think there are libraries compiled. There is indeed a 'pkgconfig(python3)',
my search just failed to find it. Otherwise, if there are really just .py
files in the tree, 'python3-base' is the string provided by 'python3N-base'.


Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64

[Andrew Cooper]

Rework the container to be non-root, use heredocs for legibility, and use use
apt-get --no-install-recommends to keep the size down.

Ubuntu Xenial has no ninja-build package, so can't build the QEMU referenced
by Xen 4.16.  Therefore, drop the dependencies too.

This saves ~500M:

  registry.gitlab.com/xen-project/xen/ubuntu    16.04-x86_64           700MB
  registry.gitlab.com/xen-project/xen/ubuntu    xenial                 1.21GB

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>

Runs with the new container:

  4.16: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323548805
  4.19: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323548805
---
 .../build/ubuntu/16.04-x86_64.dockerfile      | 65 +++++++++++++++++++
 automation/build/ubuntu/xenial.dockerfile     | 46 -------------
 automation/gitlab-ci/build.yaml               | 16 ++---
 automation/scripts/containerize               |  2 +-
 4 files changed, 74 insertions(+), 55 deletions(-)
 create mode 100644 automation/build/ubuntu/16.04-x86_64.dockerfile
 delete mode 100644 automation/build/ubuntu/xenial.dockerfile

diff --git a/automation/build/ubuntu/16.04-x86_64.dockerfile b/automation/build/ubuntu/16.04-x86_64.dockerfile
new file mode 100644
index 000000000000..11e878050b83
--- /dev/null
+++ b/automation/build/ubuntu/16.04-x86_64.dockerfile
@@ -0,0 +1,65 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 ubuntu:16.04
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        clang
+        flex
+        python3-minimal
+
+        # Tools (general)
+        ca-certificates
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-dev
+        libzstd-dev
+        liblzo2-dev
+        liblzma-dev
+        zlib1g-dev
+        # libacpi
+        acpica-tools
+        # libxl
+        uuid-dev
+        libyajl-dev
+        # RomBIOS
+        bcc
+        bin86
+        # xentop
+        libncurses5-dev
+        # Python bindings
+        python3-dev
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml-nox
+        ocaml-findlib
+
+        # Stubdom download/extract
+        bzip2
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/ubuntu/xenial.dockerfile b/automation/build/ubuntu/xenial.dockerfile
deleted file mode 100644
index aa8f560286b6..000000000000
--- a/automation/build/ubuntu/xenial.dockerfile
+++ /dev/null
@@ -1,46 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 ubuntu:16.04
-LABEL maintainer.name="The Xen Project " \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        zlib1g-dev \
-        libncurses5-dev \
-        python-dev \
-        python3-dev \
-        uuid-dev \
-        libyajl-dev \
-        libaio-dev \
-        libglib2.0-dev \
-        clang \
-        libpixman-1-dev \
-        pkg-config \
-        flex \
-        bison \
-        acpica-tools \
-        bin86 \
-        bcc \
-        liblzma-dev \
-        libnl-3-dev \
-        ocaml-nox \
-        libfindlib-ocaml-dev \
-        markdown \
-        transfig \
-        pandoc \
-        checkpolicy \
-        wget \
-        git \
-        nasm \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 2216392a8414..a2e6815f30b3 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -588,25 +588,25 @@ fedora-gcc-debug:
   variables:
     CONTAINER: fedora:29
 
-ubuntu-xenial-clang:
+ubuntu-16.04-x86_64-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: ubuntu:xenial
+    CONTAINER: ubuntu:16.04-x86_64
 
-ubuntu-xenial-clang-debug:
+ubuntu-16.04-x86_64-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:xenial
+    CONTAINER: ubuntu:16.04-x86_64
 
-ubuntu-xenial-gcc:
+ubuntu-16.04-x86_64-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: ubuntu:xenial
+    CONTAINER: ubuntu:16.04-x86_64
 
-ubuntu-xenial-gcc-debug:
+ubuntu-16.04-x86_64-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:xenial
+    CONTAINER: ubuntu:16.04-x86_64
 
 ubuntu-bionic-clang:
   extends: .clang-x86-64-build
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 0ff53b6fe4e1..5fc41c36a27c 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -52,9 +52,9 @@ case "_${CONTAINER}" in
     _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
     _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
-    _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
+    _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
 
     *) guess_container ;;
 esac
-- 
2.39.2

[PATCH 16/12] CI: Refresh Ubuntu Bionic container as 18.04-x86_64

[Andrew Cooper]

As with 16.04 (Focal), except that ninja-build is available so QEMU can be
built.

This halves the size of the container:

  registry.gitlab.com/xen-project/xen/ubuntu    18.04-x86_64           860MB
  registry.gitlab.com/xen-project/xen/ubuntu    bionic                 1.44GB

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>

Runs with the new container:

  4.16: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323548809
  4.19: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323516970
---
 .../build/ubuntu/18.04-x86_64.dockerfile      | 72 +++++++++++++++++++
 automation/build/ubuntu/bionic.dockerfile     | 48 -------------
 automation/gitlab-ci/build.yaml               | 16 ++---
 automation/scripts/containerize               |  2 +-
 4 files changed, 81 insertions(+), 57 deletions(-)
 create mode 100644 automation/build/ubuntu/18.04-x86_64.dockerfile
 delete mode 100644 automation/build/ubuntu/bionic.dockerfile

diff --git a/automation/build/ubuntu/18.04-x86_64.dockerfile b/automation/build/ubuntu/18.04-x86_64.dockerfile
new file mode 100644
index 000000000000..24047e8f6ab2
--- /dev/null
+++ b/automation/build/ubuntu/18.04-x86_64.dockerfile
@@ -0,0 +1,72 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 ubuntu:18.04
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        clang
+        flex
+        python3-minimal
+
+        # Tools (general)
+        ca-certificates
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-dev
+        libzstd-dev
+        liblzo2-dev
+        liblzma-dev
+        zlib1g-dev
+        # libacpi
+        acpica-tools
+        # libxl
+        uuid-dev
+        libyajl-dev
+        # RomBIOS
+        bcc
+        bin86
+        # xentop
+        libncurses5-dev
+        # Python bindings
+        python3-dev
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml-nox
+        ocaml-findlib
+
+        # Stubdom download/extract
+        bzip2
+
+        # Qemu build
+        libglib2.0-dev
+        libpixman-1-dev
+        meson
+        ninja-build
+        python3-packaging
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/ubuntu/bionic.dockerfile b/automation/build/ubuntu/bionic.dockerfile
deleted file mode 100644
index c1effc725e17..000000000000
--- a/automation/build/ubuntu/bionic.dockerfile
+++ /dev/null
@@ -1,48 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 ubuntu:18.04
-LABEL maintainer.name="The Xen Project " \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        zlib1g-dev \
-        libncurses5-dev \
-        python-dev \
-        python3-dev \
-        uuid-dev \
-        libyajl-dev \
-        libaio-dev \
-        libglib2.0-dev \
-        clang \
-        libpixman-1-dev \
-        pkg-config \
-        flex \
-        bison \
-        acpica-tools \
-        bin86 \
-        bcc \
-        liblzma-dev \
-        libnl-3-dev \
-        ocaml-nox \
-        libfindlib-ocaml-dev \
-        markdown \
-        transfig \
-        pandoc \
-        checkpolicy \
-        wget \
-        git \
-        nasm \
-        # QEMU
-        ninja-build \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index a2e6815f30b3..c6f6a1ced7dc 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -608,25 +608,25 @@ ubuntu-16.04-x86_64-gcc-debug:
   variables:
     CONTAINER: ubuntu:16.04-x86_64
 
-ubuntu-bionic-clang:
+ubuntu-18.04-x86_64-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: ubuntu:bionic
+    CONTAINER: ubuntu:18.04-x86_64
 
-ubuntu-bionic-clang-debug:
+ubuntu-18.04-x86_64-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:bionic
+    CONTAINER: ubuntu:18.04-x86_64
 
-ubuntu-bionic-gcc:
+ubuntu-18.04-x86_64-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: ubuntu:bionic
+    CONTAINER: ubuntu:18.04-x86_64
 
-ubuntu-bionic-gcc-debug:
+ubuntu-18.04-x86_64-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:bionic
+    CONTAINER: ubuntu:18.04-x86_64
 
 ubuntu-focal-gcc:
   extends: .gcc-x86-64-build
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 5fc41c36a27c..d6132b4d22af 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -51,10 +51,10 @@ case "_${CONTAINER}" in
     _bookworm-arm64v8-arm32-gcc) CONTAINER="${BASE}/debian:bookworm-arm64v8-arm32-gcc" ;;
     _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
     _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
-    _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
+    _bionic) CONTAINER="${BASE}/ubuntu:18.04-x86_64" ;;
 
     *) guess_container ;;
 esac
-- 
2.39.2

[PATCH 17/12] CI: Refresh Ubuntu Focal container as 20.04-x86_64

[Andrew Cooper]

Exactly as per 18.04 (Bionic).  This saves ~500M:

  registry.gitlab.com/xen-project/xen/ubuntu    20.04-x86_64           1.06GB
  registry.gitlab.com/xen-project/xen/ubuntu    focal                  1.57GB

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>

Runs with the new container:

  4.16: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323548811
  4.19: https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7323517116
---
 .../build/ubuntu/20.04-x86_64.dockerfile      | 72 +++++++++++++++++++
 automation/build/ubuntu/focal.dockerfile      | 49 -------------
 automation/gitlab-ci/build.yaml               | 16 ++---
 automation/scripts/containerize               |  2 +-
 4 files changed, 81 insertions(+), 58 deletions(-)
 create mode 100644 automation/build/ubuntu/20.04-x86_64.dockerfile
 delete mode 100644 automation/build/ubuntu/focal.dockerfile

diff --git a/automation/build/ubuntu/20.04-x86_64.dockerfile b/automation/build/ubuntu/20.04-x86_64.dockerfile
new file mode 100644
index 000000000000..39fb297519ee
--- /dev/null
+++ b/automation/build/ubuntu/20.04-x86_64.dockerfile
@@ -0,0 +1,72 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 ubuntu:20.04
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        clang
+        flex
+        python3-minimal
+
+        # Tools (general)
+        ca-certificates
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-dev
+        libzstd-dev
+        liblzo2-dev
+        liblzma-dev
+        zlib1g-dev
+        # libacpi
+        acpica-tools
+        # libxl
+        uuid-dev
+        libyajl-dev
+        # RomBIOS
+        bcc
+        bin86
+        # xentop
+        libncurses5-dev
+        # Python bindings
+        python3-dev
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml-nox
+        ocaml-findlib
+
+        # Stubdom download/extract
+        bzip2
+
+        # Qemu build
+        libglib2.0-dev
+        libpixman-1-dev
+        meson
+        ninja-build
+        python3-packaging
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/ubuntu/focal.dockerfile b/automation/build/ubuntu/focal.dockerfile
deleted file mode 100644
index b49e53d185b4..000000000000
--- a/automation/build/ubuntu/focal.dockerfile
+++ /dev/null
@@ -1,49 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 ubuntu:20.04
-LABEL maintainer.name="The Xen Project " \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV DEBIAN_FRONTEND=noninteractive
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-# build depends
-RUN apt-get update && \
-    apt-get --quiet --yes install \
-        build-essential \
-        zlib1g-dev \
-        libncurses5-dev \
-        python3-dev \
-        python3-setuptools \
-        uuid-dev \
-        libyajl-dev \
-        libaio-dev \
-        libglib2.0-dev \
-        clang \
-        libpixman-1-dev \
-        pkg-config \
-        flex \
-        bison \
-        acpica-tools \
-        bin86 \
-        bcc \
-        liblzma-dev \
-        libnl-3-dev \
-        ocaml-nox \
-        libfindlib-ocaml-dev \
-        # libsystemd-dev for Xen < 4.19
-        libsystemd-dev \
-        transfig \
-        pandoc \
-        checkpolicy \
-        wget \
-        git \
-        nasm \
-        # QEMU
-        ninja-build \
-        && \
-        apt-get autoremove -y && \
-        apt-get clean && \
-        rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index c6f6a1ced7dc..6a2d796959a1 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -628,25 +628,25 @@ ubuntu-18.04-x86_64-gcc-debug:
   variables:
     CONTAINER: ubuntu:18.04-x86_64
 
-ubuntu-focal-gcc:
+ubuntu-20.04-x86_64-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: ubuntu:focal
+    CONTAINER: ubuntu:20.04-x86_64
 
-ubuntu-focal-gcc-debug:
+ubuntu-20.04-x86_64-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:focal
+    CONTAINER: ubuntu:20.04-x86_64
 
-ubuntu-focal-clang:
+ubuntu-20.04-x86_64-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: ubuntu:focal
+    CONTAINER: ubuntu:20.04-x86_64
 
-ubuntu-focal-clang-debug:
+ubuntu-20.04-x86_64-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:focal
+    CONTAINER: ubuntu:20.04-x86_64
 
 opensuse-leap-15.6-clang:
   extends: .clang-x86-64-build
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index d6132b4d22af..f27932dd4561 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -40,7 +40,6 @@ case "_${CONTAINER}" in
     _archlinux|_arch) CONTAINER="${BASE}/archlinux:current" ;;
     _centos7) CONTAINER="${BASE}/centos:7" ;;
     _fedora) CONTAINER="${BASE}/fedora:29";;
-    _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
     _bullseye-ppc64le) CONTAINER="${BASE}/debian:11-ppc64le" ;;
     _bookworm-ppc64le) CONTAINER="${BASE}/debian:12-ppc64le" ;;
     _bullseye-riscv64) CONTAINER="${BASE}/debian:11-riscv64" ;;
@@ -55,6 +54,7 @@ case "_${CONTAINER}" in
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:18.04-x86_64" ;;
+    _focal)  CONTAINER="${BASE}/ubuntu:20.04-x86_64" ;;
 
     *) guess_container ;;
 esac
-- 
2.39.2

[PATCH 18/12] CI: Add Ubuntu 22.04 (Jammy) and 24.04 (Noble) testing

[Andrew Cooper]

The containers are exactly as per 20.04 (Focal).  However, this now brings us
to 5 releases * 4 build jobs worth of Ubuntu testing, which is overkill.

The oldest and newest toolchains are the most likely to find problems with new
code, so reduce the middle 3 releases (18/20/22) to just a single smoke test
each.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>

Pipeline run showing all Ubuntu testing on staging:

  https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1371106799

There are 11 jobs in total.
---
 .../build/ubuntu/22.04-x86_64.dockerfile      | 72 +++++++++++++++++++
 .../build/ubuntu/24.04-x86_64.dockerfile      | 72 +++++++++++++++++++
 automation/gitlab-ci/build.yaml               | 39 +++++-----
 automation/scripts/containerize               |  2 +
 4 files changed, 163 insertions(+), 22 deletions(-)
 create mode 100644 automation/build/ubuntu/22.04-x86_64.dockerfile
 create mode 100644 automation/build/ubuntu/24.04-x86_64.dockerfile

diff --git a/automation/build/ubuntu/22.04-x86_64.dockerfile b/automation/build/ubuntu/22.04-x86_64.dockerfile
new file mode 100644
index 000000000000..230903f624d9
--- /dev/null
+++ b/automation/build/ubuntu/22.04-x86_64.dockerfile
@@ -0,0 +1,72 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 ubuntu:22.04
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        clang
+        flex
+        python3-minimal
+
+        # Tools (general)
+        ca-certificates
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-dev
+        libzstd-dev
+        liblzo2-dev
+        liblzma-dev
+        zlib1g-dev
+        # libacpi
+        acpica-tools
+        # libxl
+        uuid-dev
+        libyajl-dev
+        # RomBIOS
+        bcc
+        bin86
+        # xentop
+        libncurses5-dev
+        # Python bindings
+        python3-dev
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml-nox
+        ocaml-findlib
+
+        # Stubdom download/extract
+        bzip2
+
+        # Qemu build
+        libglib2.0-dev
+        libpixman-1-dev
+        meson
+        ninja-build
+        python3-packaging
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/ubuntu/24.04-x86_64.dockerfile b/automation/build/ubuntu/24.04-x86_64.dockerfile
new file mode 100644
index 000000000000..277f92facfd9
--- /dev/null
+++ b/automation/build/ubuntu/24.04-x86_64.dockerfile
@@ -0,0 +1,72 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 ubuntu:24.04
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    apt-get -y update
+
+    DEPS=(
+        # Xen
+        bison
+        build-essential
+        checkpolicy
+        clang
+        flex
+        python3-minimal
+
+        # Tools (general)
+        ca-certificates
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        libbz2-dev
+        libzstd-dev
+        liblzo2-dev
+        liblzma-dev
+        zlib1g-dev
+        # libacpi
+        acpica-tools
+        # libxl
+        uuid-dev
+        libyajl-dev
+        # RomBIOS
+        bcc
+        bin86
+        # xentop
+        libncurses5-dev
+        # Python bindings
+        python3-dev
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml-nox
+        ocaml-findlib
+
+        # Stubdom download/extract
+        bzip2
+
+        # Qemu build
+        libglib2.0-dev
+        libpixman-1-dev
+        meson
+        ninja-build
+        python3-packaging
+    )
+
+    apt-get -y --no-install-recommends install "${DEPS[@]}"
+    rm -rf /var/lib/apt/lists/*
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 6a2d796959a1..379cf17d65cc 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -608,45 +608,40 @@ ubuntu-16.04-x86_64-gcc-debug:
   variables:
     CONTAINER: ubuntu:16.04-x86_64
 
-ubuntu-18.04-x86_64-clang:
-  extends: .clang-x86-64-build
-  variables:
-    CONTAINER: ubuntu:18.04-x86_64
-
-ubuntu-18.04-x86_64-clang-debug:
-  extends: .clang-x86-64-build-debug
-  variables:
-    CONTAINER: ubuntu:18.04-x86_64
-
 ubuntu-18.04-x86_64-gcc:
   extends: .gcc-x86-64-build
   variables:
     CONTAINER: ubuntu:18.04-x86_64
 
-ubuntu-18.04-x86_64-gcc-debug:
-  extends: .gcc-x86-64-build-debug
-  variables:
-    CONTAINER: ubuntu:18.04-x86_64
-
 ubuntu-20.04-x86_64-gcc:
   extends: .gcc-x86-64-build
   variables:
     CONTAINER: ubuntu:20.04-x86_64
 
-ubuntu-20.04-x86_64-gcc-debug:
-  extends: .gcc-x86-64-build-debug
+ubuntu-22.04-x86_64-gcc:
+  extends: .gcc-x86-64-build
   variables:
-    CONTAINER: ubuntu:20.04-x86_64
+    CONTAINER: ubuntu:22.04-x86_64
 
-ubuntu-20.04-x86_64-clang:
+ubuntu-24.04-x86_64-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: ubuntu:20.04-x86_64
+    CONTAINER: ubuntu:24.04-x86_64
 
-ubuntu-20.04-x86_64-clang-debug:
+ubuntu-24.04-x86_64-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: ubuntu:20.04-x86_64
+    CONTAINER: ubuntu:24.04-x86_64
+
+ubuntu-24.04-x86_64-gcc:
+  extends: .gcc-x86-64-build
+  variables:
+    CONTAINER: ubuntu:24.04-x86_64
+
+ubuntu-24.04-x86_64-gcc-debug:
+  extends: .gcc-x86-64-build-debug
+  variables:
+    CONTAINER: ubuntu:24.04-x86_64
 
 opensuse-leap-15.6-clang:
   extends: .clang-x86-64-build
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index f27932dd4561..9844133aff63 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -55,6 +55,8 @@ case "_${CONTAINER}" in
     _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:18.04-x86_64" ;;
     _focal)  CONTAINER="${BASE}/ubuntu:20.04-x86_64" ;;
+    _jammy)  CONTAINER="${BASE}/ubuntu:22.04-x86_64" ;;
+    _noble)  CONTAINER="${BASE}/ubuntu:24.04-x86_64" ;;
 
     *) guess_container ;;
 esac

base-commit: d46e4264c3fa68974fbb9b1bce514ea44af6d945
prerequisite-patch-id: b6875ad0bbd279067fdef31b9adf67e2a5abdf8a
prerequisite-patch-id: 5d4a01a02e7694939cdbf620a2e7509fee49dd92
prerequisite-patch-id: 65ecea93b1a320219deb8670264c3d464e00384e
prerequisite-patch-id: eb7042acce82d2bc324f8560fed3ffd57025d248
prerequisite-patch-id: 91c6663df2cfa139497e2d8b4dc17f1697f0c9f6
prerequisite-patch-id: 84c3345d580461986a3a9560735279baff21234f
prerequisite-patch-id: 5d732fc5f039fb25f7177279f63019f59acae4b1
prerequisite-patch-id: dd5006da276abc0e4da1ecb2c84d699c5bd98124
prerequisite-patch-id: 7e816655cd1066b42be5fa57a832c777c9867fc9
prerequisite-patch-id: fc25cefbee4cb69703df5c1f5b3eb317f03952af
prerequisite-patch-id: 531b3db898f54c05bff5fbcb69e0f7f62811c744
prerequisite-patch-id: ddb51b78aeaa2a6efe8035c39a128810c01aa822
prerequisite-patch-id: b4894b8120682429e1a74369c6b36bbbdc09d749
prerequisite-patch-id: c002530cb86a08b38980779a3d06cf491fdb9fae
-- 
2.39.2

[PATCH v2 13/12] CI: Refresh OpenSUSE Tumbleweed container

[Andrew Cooper]

Existing as suse:opensuse-tumbleweed is a historical quirk, and adjusted for
consistency with all the other containers.

Make it non-root, use heredocs for legibility, and use the zypper long names
for the benefit of those wondering what was being referenced or duplicated.

Trim the dependencies substantially.  Testing docs isn't very interesting and
saves a lot of space.  Other savings come from removing a huge pile of
optional QEMU dependencies (QEMU just needs to build the Xen parts to be
useful here, not have a full GUI environment).

Finally, there where some packages such as bc, libssh2-devel, libtasn1-devel
and nasm that I'm not aware of any reason to have had, even historically.

This shaves 1G off the size of the container:

  registry.gitlab.com/xen-project/xen/opensuse  tumbleweed-x86_64      1.39GB
  registry.gitlab.com/xen-project/xen/suse      opensuse-tumbleweed    2.35GB

Furthermore, identify which components of the build use which dependencies,
which will help managing them in the future.

Thanks to Olaf Hering for dependency fixes that have been subsumed into this
total overhaul.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
CC: Olaf Hering <olaf@aepfle.de>

This includes both of Olaf's recent changes, python3-packaging (now required
for a rebuilt container), and findutils (not required yet; still gets pulled
in transitively), but added for good measure.

In terms of package delta, newly included are:

  findutils            # As suggested
  ocaml-findlib        # Replaces ocaml-findlib-devel
  perl                 # Transitive previously
  python3              # Transitive previously
  python3-packaging    # As suggested

while removed are:

  bc                   # ?
  cmake                # vtpm stubdom, optional
  discount             # docs
  ghostscript          # docs
  glibc-devel          # Not really optional, given a toolchain
  hostname             # Optional, not interesting
  libaio-devel         # Xen < 4.13, perhaps Qemu still but optional
  libext2fs-devel      # Libfsimage, off by default, optional
  libgnutls-devel      # Qemu ?
  libjpeg62-devel      # Qemu ?
  libnl3-devel         # Libxl COLO(?), option
  libnuma-devel        # ?
  libpng16-devel       # Qemu ?
  libssh2-devel        # ?
  libtasn1-devel       # ?
  nasm                 # ?
  ocaml-findlib-devel  # Swapped for non-devel
  ocaml-ocamlbuild     # ?
  ocaml-ocamldoc       # ?
  pandoc               # docs
  'pkgconfig(libpci)'  # Qemu
  'pkgconfig(sdl)'     # Qemu
  'pkgconfig(sdl2)'    # Qemu
  systemd-devel        # Xen < 4.19, optional
  transfig             # docs
  valgrind-devel       # really only for developers
  which                # Xen < 4.17

I've tested this on staging-4.17 and staging, both of which build fine.
staging-4.16 does not, because of SeaBIOS which we've decided not to fix,
hence it's safe to drop `which`.

v2:
 * git -> git-core
 * Use 'pkgconfig()' references
---
 .../opensuse/tumbleweed-x86_64.dockerfile     | 76 +++++++++++++++++++
 .../build/suse/opensuse-tumbleweed.dockerfile | 74 ------------------
 automation/gitlab-ci/build.yaml               |  8 +-
 automation/scripts/containerize               |  2 +-
 4 files changed, 81 insertions(+), 79 deletions(-)
 create mode 100644 automation/build/opensuse/tumbleweed-x86_64.dockerfile
 delete mode 100644 automation/build/suse/opensuse-tumbleweed.dockerfile

diff --git a/automation/build/opensuse/tumbleweed-x86_64.dockerfile b/automation/build/opensuse/tumbleweed-x86_64.dockerfile
new file mode 100644
index 000000000000..61e840fc67da
--- /dev/null
+++ b/automation/build/opensuse/tumbleweed-x86_64.dockerfile
@@ -0,0 +1,76 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 opensuse/tumbleweed
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV XEN_TARGET_ARCH=x86_64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    zypper refresh
+    zypper dist-upgrade -y --no-recommends
+
+    DEPS=(
+        # Xen
+        bison
+        checkpolicy
+        clang
+        diffutils
+        findutils
+        flex
+        gawk
+        gcc
+        make
+        python3
+
+        # Tools (general)
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        'pkgconfig(bzip2)'
+        'pkgconfig(libzstd)'
+        'pkgconfig(lzo2)'
+        'pkgconfig(liblzma)'
+        'pkgconfig(zlib)'
+        # libacpi
+        acpica
+        # libxl
+        'pkgconfig(uuid)'
+        'pkgconfig(yajl)'
+        # Header Check
+        gcc-c++
+        # xentop
+        'pkgconfig(ncurses)'
+        # Python bindings
+        python3-devel
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml
+        ocaml-findlib
+
+        # Stubdom (download/extract)
+        bzip2
+        tar
+
+        # Qemu build
+        meson
+        ninja
+        'pkgconfig(glib-2.0)'
+        'pkgconfig(pixman-1)'
+        python3-packaging
+    )
+
+    zypper install -y --no-recommends "${DEPS[@]}"
+    zypper clean -a
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/suse/opensuse-tumbleweed.dockerfile b/automation/build/suse/opensuse-tumbleweed.dockerfile
deleted file mode 100644
index 077fed22260c..000000000000
--- a/automation/build/suse/opensuse-tumbleweed.dockerfile
+++ /dev/null
@@ -1,74 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 opensuse/tumbleweed
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-RUN zypper ref && zypper dup -y --no-recommends
-RUN zypper install -y --no-recommends \
-        acpica \
-        bc \
-        bison \
-        bzip2 \
-        checkpolicy \
-        clang \
-        cmake \
-        diffutils \
-        discount \
-        flex \
-        gawk \
-        gcc \
-        gcc-c++ \
-        git \
-        ghostscript \
-        glib2-devel \
-        glibc-devel \
-        gzip \
-        hostname \
-        libaio-devel \
-        libbz2-devel \
-        libext2fs-devel \
-        libgnutls-devel \
-        libjpeg62-devel \
-        libnl3-devel \
-        libnuma-devel \
-        libpixman-1-0-devel \
-        libpng16-devel \
-        libssh2-devel \
-        libtasn1-devel \
-        libuuid-devel \
-        libyajl-devel \
-        libzstd-devel \
-        lzo-devel \
-        make \
-        meson \
-        nasm \
-        ncurses-devel \
-        ninja \
-        ocaml \
-        ocaml-findlib-devel \
-        ocaml-ocamlbuild \
-        ocaml-ocamldoc \
-        pandoc \
-        patch \
-        pkg-config \
-        'pkgconfig(libpci)' \
-        'pkgconfig(sdl)' \
-        'pkgconfig(sdl2)' \
-        python3-devel \
-        python3-setuptools \
-        # systemd-devel for Xen < 4.19
-        systemd-devel \
-        tar \
-        transfig \
-        valgrind-devel \
-        wget \
-        which \
-        xz-devel \
-        zlib-devel \
-        && \
-        zypper clean -a
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index b4139414bc8e..fd6ca1270e4e 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -668,25 +668,25 @@ opensuse-leap-gcc-debug:
 opensuse-tumbleweed-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 opensuse-tumbleweed-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-tumbleweed
+    CONTAINER: opensuse:tumbleweed-x86_64
   allow_failure: true
 
 # PowerPC builds (x86 cross)
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index 4d5669c5ad01..c4f013702122 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -43,7 +43,7 @@ case "_${CONTAINER}" in
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
     _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
-    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/suse:opensuse-tumbleweed" ;;
+    _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
 esac
 
 # Use this variable to control whether root should be used
-- 
2.39.2

Re: [PATCH 10/12] CI: Introduce debian:11/12-riscv64 containers

[Oleksii]

On Thu, 2024-07-11 at 12:15 +0100, Andrew Cooper wrote:
> For starters, they're slightly smaller:
> 
>   $ docker image list <snip>
>   registry.gitlab.com/xen-project/xen/debian      12-riscv64        
> 772MB
>   registry.gitlab.com/xen-project/xen/debian      11-riscv64        
> 422MB
Do we really need both 11-riscv64 and 12-riscv64?

Generally this patch LGTM: Reviewed-by: Oleksii Kurochko
<oleksii.kurochko@gmail.com>

~ Oleksii
 

>   registry.gitlab.com/xen-project/xen/archlinux   current-riscv64   
> 2.32GB
> 
> They also not rolling distros, so will be more predicatable testing
> for the
> stable trees in the future.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> CC: Anthony PERARD <anthony.perard@vates.tech>
> CC: Juergen Gross <jgross@suse.com>
> CC: Roger Pau Monné <roger.pau@citrix.com>
> CC: Jan Beulich <JBeulich@suse.com>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Julien Grall <julien@xen.org>
> CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> CC: Shawn Anastasio <sanastasio@raptorengineering.com>
> ---
>  automation/build/debian/11-riscv64.dockerfile | 33 +++++++++++++++++
>  automation/build/debian/12-riscv64.dockerfile | 36
> +++++++++++++++++++
>  2 files changed, 69 insertions(+)
>  create mode 100644 automation/build/debian/11-riscv64.dockerfile
>  create mode 100644 automation/build/debian/12-riscv64.dockerfile
> 
> diff --git a/automation/build/debian/11-riscv64.dockerfile
> b/automation/build/debian/11-riscv64.dockerfile
> new file mode 100644
> index 000000000000..1c99bc89ea1a
> --- /dev/null
> +++ b/automation/build/debian/11-riscv64.dockerfile
> @@ -0,0 +1,33 @@
> +# syntax=docker/dockerfile:1
> +FROM --platform=linux/amd64 debian:bullseye-slim
> +LABEL maintainer.name="The Xen Project"
> +LABEL maintainer.email="xen-devel@lists.xenproject.org"
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +ENV CROSS_COMPILE=riscv64-linux-gnu-
> +ENV XEN_TARGET_ARCH=riscv64
> +
> +RUN <<EOF
> +#!/bin/bash
> +    set -e
> +
> +    useradd --create-home user
> +
> +    apt-get -y update
> +
> +    DEPS=(
> +        # Xen
> +        bison
> +        build-essential
> +        checkpolicy
> +        flex
> +        gcc-riscv64-linux-gnu
> +        python3-minimal
> +    )
> +
> +    apt-get -y --no-install-recommends install "${DEPS[@]}"
> +    rm -rf /var/lib/apt/lists/*
> +EOF
> +
> +USER user
> +WORKDIR /build
> diff --git a/automation/build/debian/12-riscv64.dockerfile
> b/automation/build/debian/12-riscv64.dockerfile
> new file mode 100644
> index 000000000000..03fd4b03059b
> --- /dev/null
> +++ b/automation/build/debian/12-riscv64.dockerfile
> @@ -0,0 +1,36 @@
> +# syntax=docker/dockerfile:1
> +FROM --platform=linux/amd64 debian:bookworm-slim
> +LABEL maintainer.name="The Xen Project"
> +LABEL maintainer.email="xen-devel@lists.xenproject.org"
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +ENV CROSS_COMPILE=riscv64-linux-gnu-
> +ENV XEN_TARGET_ARCH=riscv64
> +
> +RUN <<EOF
> +#!/bin/bash
> +    set -e
> +
> +    useradd --create-home user
> +
> +    apt-get -y update
> +
> +    DEPS=(
> +        # Xen
> +        bison
> +        build-essential
> +        checkpolicy
> +        flex
> +        gcc-riscv64-linux-gnu
> +        python3-minimal
> +
> +        # Qemu for test phase
> +        qemu-system-riscv64
> +    )
> +
> +    apt-get -y --no-install-recommends install "${DEPS[@]}"
> +    rm -rf /var/lib/apt/lists/*
> +EOF
> +
> +USER user
> +WORKDIR /build

Re: [PATCH 11/12] CI: Swap to debian for riscv64 build and test

[Oleksii]

On Thu, 2024-07-11 at 12:15 +0100, Andrew Cooper wrote:
> The containers are both much smaller, with stable toolchains over
> time, and
> this at least means we're not doing all testing with a single
> compiler.
> 
> Rename the jobs to follow to sort coherently ($DISTRO-$VERSION-$ARCH-
> *) and
> reposition the jobs to optimise starting the smoke test.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
LGTM: Oleksii Kurochko <oleksii.kurochko@gmail.com>

~ Oleksii

> ---
> CC: Anthony PERARD <anthony.perard@vates.tech>
> CC: Juergen Gross <jgross@suse.com>
> CC: Roger Pau Monné <roger.pau@citrix.com>
> CC: Jan Beulich <JBeulich@suse.com>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Julien Grall <julien@xen.org>
> CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> CC: Shawn Anastasio <sanastasio@raptorengineering.com>
> ---
>  .../archlinux/current-riscv64.dockerfile      | 22 -----
>  automation/gitlab-ci/build.yaml               | 83 +++++++++++------
> --
>  automation/gitlab-ci/test.yaml                |  4 +-
>  automation/scripts/containerize               |  3 +-
>  4 files changed, 52 insertions(+), 60 deletions(-)
>  delete mode 100644 automation/build/archlinux/current-
> riscv64.dockerfile
> 
> diff --git a/automation/build/archlinux/current-riscv64.dockerfile
> b/automation/build/archlinux/current-riscv64.dockerfile
> deleted file mode 100644
> index f7770bf82a78..000000000000
> --- a/automation/build/archlinux/current-riscv64.dockerfile
> +++ /dev/null
> @@ -1,22 +0,0 @@
> -# syntax=docker/dockerfile:1
> -FROM --platform=linux/amd64 archlinux
> -LABEL maintainer.name="The Xen Project" \
> -      maintainer.email="xen-devel@lists.xenproject.org"
> -
> -# Packages needed for the build
> -RUN pacman --noconfirm --needed -Syu \
> -    base-devel \
> -    git \
> -    inetutils \
> -    riscv64-linux-gnu-binutils \
> -    riscv64-linux-gnu-gcc \
> -    riscv64-linux-gnu-glibc \
> -    # For test phase
> -    qemu-system-riscv
> -
> -# Add compiler path
> -ENV CROSS_COMPILE=riscv64-linux-gnu-
> -
> -RUN useradd --create-home user
> -USER user
> -WORKDIR /build
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-
> ci/build.yaml
> index e081664c4e95..4b9d80cc5632 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -357,6 +357,13 @@ debian-12-ppc64le-gcc-debug:
>      KBUILD_DEFCONFIG: ppc64_defconfig
>      HYPERVISOR_ONLY: y
>  
> +debian-12-riscv64-gcc-debug:
> +  extends: .gcc-riscv64-cross-build-debug
> +  variables:
> +    CONTAINER: debian:12-riscv64
> +    KBUILD_DEFCONFIG: tiny64_defconfig
> +    HYPERVISOR_ONLY: y
> +
>  # Arm32 cross-build
>  
>  debian-bookworm-gcc-arm32:
> @@ -458,41 +465,6 @@ alpine-3.18-gcc-debug-arm64-earlyprintk:
>        CONFIG_EARLY_UART_CHOICE_PL011=y
>        CONFIG_EARLY_UART_BASE_ADDRESS=0x9000000
>  
> -# RISC-V 64 cross-build
> -.riscv-fixed-randconfig:
> -  variables: &riscv-fixed-randconfig
> -    EXTRA_FIXED_RANDCONFIG: |
> -      CONFIG_BOOT_TIME_CPUPOOLS=n
> -      CONFIG_COVERAGE=n
> -      CONFIG_EXPERT=y
> -      CONFIG_GRANT_TABLE=n
> -      CONFIG_MEM_ACCESS=n
> -      CONFIG_PERF_COUNTERS=n
> -      CONFIG_LIVEPATCH=n
> -      CONFIG_XSM=n
> -
> -archlinux-current-gcc-riscv64:
> -  extends: .gcc-riscv64-cross-build
> -  variables:
> -    CONTAINER: archlinux:current-riscv64
> -    KBUILD_DEFCONFIG: tiny64_defconfig
> -    HYPERVISOR_ONLY: y
> -
> -archlinux-current-gcc-riscv64-debug:
> -  extends: .gcc-riscv64-cross-build-debug
> -  variables:
> -    CONTAINER: archlinux:current-riscv64
> -    KBUILD_DEFCONFIG: tiny64_defconfig
> -    HYPERVISOR_ONLY: y
> -
> -archlinux-current-gcc-riscv64-randconfig:
> -  extends: .gcc-riscv64-cross-build
> -  variables:
> -    CONTAINER: archlinux:current-riscv64
> -    KBUILD_DEFCONFIG: tiny64_defconfig
> -    RANDCONFIG: y
> -    <<: *riscv-fixed-randconfig
> -
>  # Yocto test jobs
>  yocto-qemuarm64:
>    extends: .yocto-test-arm64
> @@ -739,3 +711,44 @@ debian-12-ppc64le-gcc:
>      KBUILD_DEFCONFIG: ppc64_defconfig
>      HYPERVISOR_ONLY: y
>  
> +# RISC-V 64 cross-build
> +debian-11-riscv64-gcc:
> +  extends: .gcc-riscv64-cross-build
> +  variables:
> +    CONTAINER: debian:11-riscv64
> +    KBUILD_DEFCONFIG: tiny64_defconfig
> +    HYPERVISOR_ONLY: y
> +
> +debian-11-riscv64-gcc-debug:
> +  extends: .gcc-riscv64-cross-build-debug
> +  variables:
> +    CONTAINER: debian:11-riscv64
> +    KBUILD_DEFCONFIG: tiny64_defconfig
> +    HYPERVISOR_ONLY: y
> +
> +debian-12-riscv64-gcc:
> +  extends: .gcc-riscv64-cross-build
> +  variables:
> +    CONTAINER: debian:12-riscv64
> +    KBUILD_DEFCONFIG: tiny64_defconfig
> +    HYPERVISOR_ONLY: y
> +
> +.riscv-fixed-randconfig:
> +  variables: &riscv-fixed-randconfig
> +    EXTRA_FIXED_RANDCONFIG: |
> +      CONFIG_BOOT_TIME_CPUPOOLS=n
> +      CONFIG_COVERAGE=n
> +      CONFIG_EXPERT=y
> +      CONFIG_GRANT_TABLE=n
> +      CONFIG_MEM_ACCESS=n
> +      CONFIG_PERF_COUNTERS=n
> +      CONFIG_LIVEPATCH=n
> +      CONFIG_XSM=n
> +
> +debian-12-riscv64-gcc-randconfig:
> +  extends: .gcc-riscv64-cross-build
> +  variables:
> +    CONTAINER: debian:12-riscv64
> +    KBUILD_DEFCONFIG: tiny64_defconfig
> +    RANDCONFIG: y
> +    <<: *riscv-fixed-randconfig
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-
> ci/test.yaml
> index 70c946f87799..f7e1753ead34 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -56,7 +56,7 @@
>  .qemu-riscv64:
>    extends: .test-jobs-common
>    variables:
> -    CONTAINER: archlinux:current-riscv64
> +    CONTAINER: debian:12
>      LOGFILE: qemu-smoke-riscv64.log
>    artifacts:
>      paths:
> @@ -440,7 +440,7 @@ qemu-smoke-riscv64-gcc:
>    script:
>      - ./automation/scripts/qemu-smoke-riscv64.sh 2>&1 | tee
> ${LOGFILE}
>    needs:
> -    - archlinux-current-gcc-riscv64-debug
> +    - debian-12-riscv64-gcc-debug
>  
>  qemu-smoke-ppc64le-powernv9-gcc:
>    extends: .qemu-ppc64le
> diff --git a/automation/scripts/containerize
> b/automation/scripts/containerize
> index 5c3f6782d9cd..6dbf55c8f89a 100755
> --- a/automation/scripts/containerize
> +++ b/automation/scripts/containerize
> @@ -27,12 +27,13 @@ case "_${CONTAINER}" in
>      _alpine) CONTAINER="${BASE}/alpine:3.18" ;;
>      _alpine-arm64v8) CONTAINER="${BASE}/alpine:3.18-arm64v8" ;;
>      _archlinux|_arch) CONTAINER="${BASE}/archlinux:current" ;;
> -    _riscv64) CONTAINER="${BASE}/archlinux:current-riscv64" ;;
>      _centos7) CONTAINER="${BASE}/centos:7" ;;
>      _fedora) CONTAINER="${BASE}/fedora:29";;
>      _focal) CONTAINER="${BASE}/ubuntu:focal" ;;
>      _bullseye-ppc64le) CONTAINER="${BASE}/debian:11-ppc64le" ;;
>      _bookworm-ppc64le) CONTAINER="${BASE}/debian:12-ppc64le" ;;
> +    _bullseye-riscv64) CONTAINER="${BASE}/debian:11-riscv64" ;;
> +    _bookworm-riscv64) CONTAINER="${BASE}/debian:12-riscv64" ;;
>      _buster-gcc-ibt) CONTAINER="${BASE}/debian:buster-gcc-ibt" ;;
>      _bookworm|_) CONTAINER="${BASE}/debian:bookworm" ;;
>      _bookworm-i386) CONTAINER="${BASE}/debian:bookworm-i386" ;;

Re: [PATCH 10/12] CI: Introduce debian:11/12-riscv64 containers

[Andrew Cooper]

On 12/07/2024 12:39 pm, Oleksii wrote:
> On Thu, 2024-07-11 at 12:15 +0100, Andrew Cooper wrote:
>> For starters, they're slightly smaller:
>>
>>   $ docker image list <snip>
>>   registry.gitlab.com/xen-project/xen/debian      12-riscv64        
>> 772MB
>>   registry.gitlab.com/xen-project/xen/debian      11-riscv64        
>> 422MB
> Do we really need both 11-riscv64 and 12-riscv64?

Need? No, not strictly.

Want? Yes, absolutely.

You always want at least 2 different toolchains worth of testing, or
what you will find happens is that you end up accidentally depending on
a quirk of the single compiler your using, and that you discover this at
some point in the future, rather than now(ish) when CI says no.

At the moment, the RISC-V builds are very trivial and quick, so it makes
a lot of sense to have a second toolchain.  Before too much longer,
you'll want to get Clang working too.

> Generally this patch LGTM: Reviewed-by: Oleksii Kurochko
> <oleksii.kurochko@gmail.com>

Thanks.

~Andrew

Re: [PATCH v2 13/12] CI: Refresh OpenSUSE Tumbleweed container

[Anthony PERARD]

On Fri, Jul 12, 2024 at 12:14:26PM +0100, Andrew Cooper wrote:
> while removed are:
> 
>   bc                   # ?

I don't know why, but this one is also installed by osstest. The commit
adding it doesn't say why it was needed.

Also, looks like many QEMU containers are installing `bc`, but I can't
find a single use. Maybe historic.

>   libnl3-devel         # Libxl COLO(?), option

Yes.

>   libnuma-devel        # ?

There's some mention of it in QEMU.

>   libpng16-devel       # Qemu ?

It's added to a few QEMU containers, but I can't find any use.

>   libssh2-devel        # ?

It seems like it was used by QEMU at some point, but they switch to
'libssh'. Probably optional, like many QEMU deps.

>   libtasn1-devel       # ?

QEMU, and it seems to be only used in tests.

>   nasm                 # ?

'nasm' is used to build OVMF.


If we don't want to build OVMF in our GitLab CI, which I don't think
would be useful to do anyway (no dependency on xen.git), then:

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH v2 13/12] CI: Refresh OpenSUSE Tumbleweed container

[Andrew Cooper]

On 12/07/2024 1:33 pm, Anthony PERARD wrote:
> On Fri, Jul 12, 2024 at 12:14:26PM +0100, Andrew Cooper wrote:
>> while removed are:
>>
>>   bc                   # ?
> I don't know why, but this one is also installed by osstest. The commit
> adding it doesn't say why it was needed.
>
> Also, looks like many QEMU containers are installing `bc`, but I can't
> find a single use. Maybe historic.

Part of the reason why our containers are so massive is because
dependencies only ever accumulate...

But yes - this one is especially odd.
>>   nasm                 # ?
> 'nasm' is used to build OVMF.
>
>
> If we don't want to build OVMF in our GitLab CI, which I don't think
> would be useful to do anyway (no dependency on xen.git)

Oh, interesting.

I'd assumed (by virtue of the jobs succeeding) that we were building
OVMF by default, but we're clearly not.

./configure says:

  --enable-ovmf           Enable OVMF (default is DISABLED)

so it's really just a simple opt-in, which our build script does not.


In the longer term, we will want OVMF in Gitlab CI, but we'd want to
pull it in as a separate artefact and use
--with-system-ovmf=/path/to/artefact

So I agree we don't need to build OVMF inside this container.

> , then:
>
> Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks.

~Andrew

[PATCH v2 14/12] CI: Refresh OpenSUSE Leap container

[Andrew Cooper]

See prior patch for most discussion.

Despite appearing to be a fixed release (and therefore not marked as permitted
failure), the dockerfile references the `leap` tag which is rolling in
practice.  Switch to 15.6 explicitly, for better test stability.

Vs tumbleweed, use `zypper update` rather than dist-upgrade, and retain the
RomBIOS dependencies; bin86 and dev86.

In terms of size, this saves ~700M:

  registry.gitlab.com/xen-project/xen/opensuse  leap-15.6-x86_64       1.33GB
  registry.gitlab.com/xen-project/xen/suse      opensuse-leap          2.05GB

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
CC: Shawn Anastasio <sanastasio@raptorengineering.com>
CC: Olaf Hering <olaf@aepfle.de>

Runs with this new container:

 4.19:  https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322493460
 4.16:  https://gitlab.com/xen-project/people/andyhhp/xen/-/jobs/7322499881

v2:
 * git -> git-core
 * Use 'pkgconfig()' references
---
 .../opensuse/leap-15.6-x86_64.dockerfile      | 79 +++++++++++++++++++
 .../build/suse/opensuse-leap.dockerfile       | 75 ------------------
 automation/gitlab-ci/build.yaml               | 16 ++--
 automation/scripts/containerize               |  2 +-
 4 files changed, 88 insertions(+), 84 deletions(-)
 create mode 100644 automation/build/opensuse/leap-15.6-x86_64.dockerfile
 delete mode 100644 automation/build/suse/opensuse-leap.dockerfile

diff --git a/automation/build/opensuse/leap-15.6-x86_64.dockerfile b/automation/build/opensuse/leap-15.6-x86_64.dockerfile
new file mode 100644
index 000000000000..ea70dffe2afc
--- /dev/null
+++ b/automation/build/opensuse/leap-15.6-x86_64.dockerfile
@@ -0,0 +1,79 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 opensuse/leap:15.6
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV XEN_TARGET_ARCH=x86_64
+
+RUN <<EOF
+#!/bin/bash
+    set -e
+
+    useradd --create-home user
+
+    zypper refresh
+    zypper update -y --no-recommends
+
+    DEPS=(
+        # Xen
+        bison
+        checkpolicy
+        clang
+        diffutils
+        findutils
+        flex
+        gawk
+        gcc
+        make
+        python3
+
+        # Tools (general)
+        git-core
+        gzip
+        patch
+        perl
+        pkg-config
+        wget
+        # libxenguest dombuilder
+        'pkgconfig(bzip2)'
+        'pkgconfig(libzstd)'
+        'pkgconfig(lzo2)'
+        'pkgconfig(liblzma)'
+        'pkgconfig(zlib)'
+        # libacpi
+        acpica
+        # libxl
+        'pkgconfig(uuid)'
+        'pkgconfig(yajl)'
+        # RomBIOS
+        bin86
+        dev86
+        # Header Check
+        gcc-c++
+        # xentop
+        'pkgconfig(ncurses)'
+        # Python bindings
+        python3-devel
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml
+        ocaml-findlib
+
+        # Stubdom (download/extract)
+        bzip2
+        tar
+
+        # Qemu build
+        meson
+        ninja
+        'pkgconfig(glib-2.0)'
+        'pkgconfig(pixman-1)'
+        python3-packaging
+    )
+
+    zypper install -y --no-recommends "${DEPS[@]}"
+    zypper clean -a
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/build/suse/opensuse-leap.dockerfile b/automation/build/suse/opensuse-leap.dockerfile
deleted file mode 100644
index 28d333881233..000000000000
--- a/automation/build/suse/opensuse-leap.dockerfile
+++ /dev/null
@@ -1,75 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 opensuse/leap
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-ENV USER root
-
-RUN mkdir /build
-WORKDIR /build
-
-RUN zypper ref && zypper up -y --no-recommends
-RUN zypper install -y --no-recommends \
-        acpica \
-        bc \
-        bin86 \
-        bison \
-        bzip2 \
-        checkpolicy \
-        clang \
-        cmake \
-        dev86 \
-        diffutils \
-        discount \
-        flex \
-        gawk \
-        gcc \
-        gcc-c++ \
-        git \
-        ghostscript \
-        glib2-devel \
-        glibc-devel \
-        gzip \
-        hostname \
-        libaio-devel \
-        libbz2-devel \
-        libext2fs-devel \
-        libgnutls-devel \
-        libjpeg62-devel \
-        libnl3-devel \
-        libnuma-devel \
-        libpixman-1-0-devel \
-        libpng16-devel \
-        libssh2-devel \
-        libtasn1-devel \
-        libuuid-devel \
-        libyajl-devel \
-        lzo-devel \
-        make \
-        nasm \
-        ncurses-devel \
-        ocaml \
-        ocaml-findlib-devel \
-        ocaml-ocamlbuild \
-        ocaml-ocamldoc \
-        pandoc \
-        patch \
-        pkg-config \
-        'pkgconfig(libpci)' \
-        'pkgconfig(sdl)' \
-        'pkgconfig(sdl2)' \
-        python3-devel \
-        python3-setuptools \
-        # systemd-devel for Xen < 4.19
-        systemd-devel \
-        tar \
-        transfig \
-        valgrind-devel \
-        wget \
-        which \
-        xz-devel \
-        zlib-devel \
-        # QEMU
-        ninja \
-        && \
-        zypper clean -a
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index fd6ca1270e4e..f9a9275ff293 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -645,25 +645,25 @@ ubuntu-focal-clang-debug:
   variables:
     CONTAINER: ubuntu:focal
 
-opensuse-leap-clang:
+opensuse-leap-15.6-clang:
   extends: .clang-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
-opensuse-leap-clang-debug:
+opensuse-leap-15.6-clang-debug:
   extends: .clang-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
-opensuse-leap-gcc:
+opensuse-leap-15.6-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
-opensuse-leap-gcc-debug:
+opensuse-leap-15.6-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: suse:opensuse-leap
+    CONTAINER: opensuse:leap-15.6-x86_64
 
 opensuse-tumbleweed-clang:
   extends: .clang-x86-64-build
diff --git a/automation/scripts/containerize b/automation/scripts/containerize
index c4f013702122..d8110b125333 100755
--- a/automation/scripts/containerize
+++ b/automation/scripts/containerize
@@ -42,7 +42,7 @@ case "_${CONTAINER}" in
     _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
     _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
     _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
-    _opensuse-leap|_leap) CONTAINER="${BASE}/suse:opensuse-leap" ;;
+    _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
     _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
 esac
 
-- 
2.39.2

Re: [PATCH v2 14/12] CI: Refresh OpenSUSE Leap container

[Anthony PERARD]

On Fri, Jul 12, 2024 at 01:55:38PM +0100, Andrew Cooper wrote:
> See prior patch for most discussion.
> 
> Despite appearing to be a fixed release (and therefore not marked as permitted
> failure), the dockerfile references the `leap` tag which is rolling in
> practice.  Switch to 15.6 explicitly, for better test stability.
> 
> Vs tumbleweed, use `zypper update` rather than dist-upgrade, and retain the
> RomBIOS dependencies; bin86 and dev86.
> 
> In terms of size, this saves ~700M:
> 
>   registry.gitlab.com/xen-project/xen/opensuse  leap-15.6-x86_64       1.33GB
>   registry.gitlab.com/xen-project/xen/suse      opensuse-leap          2.05GB
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64

[Anthony PERARD]

On Fri, Jul 12, 2024 at 11:48:29AM +0100, Andrew Cooper wrote:
> Rework the container to be non-root, use heredocs for legibility, and use use
> apt-get --no-install-recommends to keep the size down.
> 
> Ubuntu Xenial has no ninja-build package, so can't build the QEMU referenced
> by Xen 4.16.  Therefore, drop the dependencies too.
> 
> This saves ~500M:
> 
>   registry.gitlab.com/xen-project/xen/ubuntu    16.04-x86_64           700MB
>   registry.gitlab.com/xen-project/xen/ubuntu    xenial                 1.21GB
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64

[Anthony PERARD]

On Fri, Jul 12, 2024 at 11:48:29AM +0100, Andrew Cooper wrote:
> diff --git a/automation/scripts/containerize b/automation/scripts/containerize
> index 0ff53b6fe4e1..5fc41c36a27c 100755
> --- a/automation/scripts/containerize
> +++ b/automation/scripts/containerize
> @@ -52,9 +52,9 @@ case "_${CONTAINER}" in
>      _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
>      _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
>      _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
> -    _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
>      _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
>      _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
> +    _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
>  
>      *) guess_container ;;

I've been wondering why I couldn't apply the patch (and the next one),
and there's these 2 lines of extra context ;-)

Cheers,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64

[Andrew Cooper]

On 12/07/2024 3:02 pm, Anthony PERARD wrote:
> On Fri, Jul 12, 2024 at 11:48:29AM +0100, Andrew Cooper wrote:
>> diff --git a/automation/scripts/containerize b/automation/scripts/containerize
>> index 0ff53b6fe4e1..5fc41c36a27c 100755
>> --- a/automation/scripts/containerize
>> +++ b/automation/scripts/containerize
>> @@ -52,9 +52,9 @@ case "_${CONTAINER}" in
>>      _bookworm-arm64v8) CONTAINER="${BASE}/debian:bookworm-arm64v8" ;;
>>      _bookworm-cppcheck) CONTAINER="${BASE}/debian:bookworm-cppcheck" ;;
>>      _bionic) CONTAINER="${BASE}/ubuntu:bionic" ;;
>> -    _xenial) CONTAINER="${BASE}/ubuntu:xenial" ;;
>>      _opensuse-leap|_leap) CONTAINER="${BASE}/opensuse:leap-15.6-x86_64" ;;
>>      _opensuse-tumbleweed|_tumbleweed) CONTAINER="${BASE}/opensuse:tumbleweed-x86_64" ;;
>> +    _xenial) CONTAINER="${BASE}/ubuntu:16.04-x86_64" ;;
>>  
>>      *) guess_container ;;
> I've been wondering why I couldn't apply the patch (and the next one),
> and there's these 2 lines of extra context ;-)

Yeah, sorry.  I found these when rebasing back into order.

https://xenbits.xen.org/gitweb/?p=people/andrewcoop/xen.git;a=shortlog;h=refs/heads/auto-4.19
(and its simlarly-named variants) are the branches I'm doing all of this
work on.

~Andrew

Re: [PATCH 16/12] CI: Refresh Ubuntu Bionic container as 18.04-x86_64

[Anthony PERARD]

On Fri, Jul 12, 2024 at 11:48:42AM +0100, Andrew Cooper wrote:
> As with 16.04 (Focal), except that ninja-build is available so QEMU can be

                 ^ Xenial

These names all mixed up after a while :-)

> built.
> 
> This halves the size of the container:
> 
>   registry.gitlab.com/xen-project/xen/ubuntu    18.04-x86_64           860MB
>   registry.gitlab.com/xen-project/xen/ubuntu    bionic                 1.44GB
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 16/12] CI: Refresh Ubuntu Bionic container as 18.04-x86_64

[Andrew Cooper]

On 12/07/2024 3:12 pm, Anthony PERARD wrote:
> On Fri, Jul 12, 2024 at 11:48:42AM +0100, Andrew Cooper wrote:
>> As with 16.04 (Focal), except that ninja-build is available so QEMU can be
>                  ^ Xenial
>
> These names all mixed up after a while :-)

Bah, yes.  This is one of the few cases where numbers are superior to
names.  Will fix.

>
>> built.
>>
>> This halves the size of the container:
>>
>>   registry.gitlab.com/xen-project/xen/ubuntu    18.04-x86_64           860MB
>>   registry.gitlab.com/xen-project/xen/ubuntu    bionic                 1.44GB
>>
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

~Andrew

Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64

[Anthony PERARD]

On Fri, Jul 12, 2024 at 11:48:29AM +0100, Andrew Cooper wrote:
> diff --git a/automation/build/ubuntu/16.04-x86_64.dockerfile b/automation/build/ubuntu/16.04-x86_64.dockerfile
> new file mode 100644
> +        python3-setuptools

Another thing,

I've just notice that "python3-setuptools" was originally only introduced
to focal container on purpose, by:
bbc72a7877d8 ("automation: Add python3's setuptools to some containers")

But now this patch series is introducing it to xenial and bionic. It
might be worth mentioning, or removing the deps.

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 15/12] CI: Refresh Ubuntu Xenial container as 16.04-x86_64

[Andrew Cooper]

On 12/07/2024 3:29 pm, Anthony PERARD wrote:
> On Fri, Jul 12, 2024 at 11:48:29AM +0100, Andrew Cooper wrote:
>> diff --git a/automation/build/ubuntu/16.04-x86_64.dockerfile b/automation/build/ubuntu/16.04-x86_64.dockerfile
>> new file mode 100644
>> +        python3-setuptools
> Another thing,
>
> I've just notice that "python3-setuptools" was originally only introduced
> to focal container on purpose, by:
> bbc72a7877d8 ("automation: Add python3's setuptools to some containers")
>
> But now this patch series is introducing it to xenial and bionic. It
> might be worth mentioning, or removing the deps.
>

Oh, good point.  I'll take this back out of Xenial/Bionic.

Stretch/Trusty have already been removed.

Thanks,

~Andrew

Re: [PATCH 17/12] CI: Refresh Ubuntu Focal container as 20.04-x86_64

[Anthony PERARD]

On Fri, Jul 12, 2024 at 11:48:55AM +0100, Andrew Cooper wrote:
> Exactly as per 18.04 (Bionic).  This saves ~500M:
> 
>   registry.gitlab.com/xen-project/xen/ubuntu    20.04-x86_64           1.06GB
>   registry.gitlab.com/xen-project/xen/ubuntu    focal                  1.57GB
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 18/12] CI: Add Ubuntu 22.04 (Jammy) and 24.04 (Noble) testing

[Anthony PERARD]

On Fri, Jul 12, 2024 at 11:49:07AM +0100, Andrew Cooper wrote:
> The containers are exactly as per 20.04 (Focal).  However, this now brings us
> to 5 releases * 4 build jobs worth of Ubuntu testing, which is overkill.
> 
> The oldest and newest toolchains are the most likely to find problems with new
> code, so reduce the middle 3 releases (18/20/22) to just a single smoke test
> each.

That would mean a bit less `clang` build-test, but I guess that would be
mostly covered by FreeBSD testing on GitHub.

I tried to find out which version of clang were been used with this
patch applied:
    Alpine clang version 16.0.6
    Debian clang version 14.0.6
    clang version 17.0.6
    clang version 18.1.8
    clang version 3.8.0-2ubuntu4 (tags/RELEASE_380/final)
    Ubuntu clang version 18.1.3 (1ubuntu1)
and the versions that won't be tested anymore:
    clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
    clang version 10.0.0-4ubuntu1
    Ubuntu clang version 14.0.0-1ubuntu1.1

> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Anyway:
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 18/12] CI: Add Ubuntu 22.04 (Jammy) and 24.04 (Noble) testing

[Andrew Cooper]

On 12/07/2024 5:44 pm, Anthony PERARD wrote:
> On Fri, Jul 12, 2024 at 11:49:07AM +0100, Andrew Cooper wrote:
>> The containers are exactly as per 20.04 (Focal).  However, this now brings us
>> to 5 releases * 4 build jobs worth of Ubuntu testing, which is overkill.
>>
>> The oldest and newest toolchains are the most likely to find problems with new
>> code, so reduce the middle 3 releases (18/20/22) to just a single smoke test
>> each.
> That would mean a bit less `clang` build-test, but I guess that would be
> mostly covered by FreeBSD testing on GitHub.
>
> I tried to find out which version of clang were been used with this
> patch applied:
>     Alpine clang version 16.0.6
>     Debian clang version 14.0.6
>     clang version 17.0.6
>     clang version 18.1.8
>     clang version 3.8.0-2ubuntu4 (tags/RELEASE_380/final)
>     Ubuntu clang version 18.1.3 (1ubuntu1)
> and the versions that won't be tested anymore:
>     clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
>     clang version 10.0.0-4ubuntu1
>     Ubuntu clang version 14.0.0-1ubuntu1.1
>
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Anyway:
> Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks.

We could intermix Clang/GCC if we really want, but tbh (still!) the only
thing you want to use Clang for is stricter/better diagnostics.  GCC is
still a better optimising compiler for C, particularly for kernel
activities.

Also, those clang versions are entirely obsolete.  If we wanted to do
sensible testing, we'd need to be using the nightly builds and keeping
within ~18mo of the release which I think is their support window.

Also also, this testing is CC=Clang, LD=ld, which is subtly different to
the FreeBSD testing which is full-stack LLVM (so LD=lld).

There is definitely room for improvement in our Clang/LLVM testing.

~Andrew

[PATCH 19/12] docs: Fix install-man$(1)-pages if no manpages are generated

[Andrew Cooper]

All tools to build manpages are optional, and if none of them happen to be
present, the intermediate working directory may not even be created.

Treat this as non-fatal, bringing the behaviour in line with install-html.
Like the html side, it needs to be not-or to avoid Make thinking the rule has
failed.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
 docs/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/Makefile b/docs/Makefile
index 966a104490ac..b30cc619f8dd 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -151,7 +151,7 @@ man$(1)-pages: $$(DOC_MAN$(1))
 .PHONY: install-man$(1)-pages
 install-man$(1)-pages: man$(1)-pages
 	$(INSTALL_DIR) $(DESTDIR)$(mandir)
-	cp -r man$(1) $(DESTDIR)$(mandir)
+	[ ! -d man$(1) ] || cp -r man$(1) $(DESTDIR)$(mandir)
 
 # Clean
 .PHONY: clean-man$(1)-pages
-- 
2.39.2

[PATCH 20/12] CI: Swap from perl to perl-base in build containers

[Andrew Cooper]

We only need a basic perl environment, not the things that a primarily-perl
project would want.

Discovered during the Fedora refresh where the difference is ~40M, but it's
more modest for OpenSUSE and Ubuntu.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>

https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1372507213
---
 automation/build/opensuse/leap-15.6-x86_64.dockerfile  | 2 +-
 automation/build/opensuse/tumbleweed-x86_64.dockerfile | 2 +-
 automation/build/ubuntu/16.04-x86_64.dockerfile        | 2 +-
 automation/build/ubuntu/18.04-x86_64.dockerfile        | 2 +-
 automation/build/ubuntu/20.04-x86_64.dockerfile        | 2 +-
 automation/build/ubuntu/22.04-x86_64.dockerfile        | 2 +-
 automation/build/ubuntu/24.04-x86_64.dockerfile        | 2 +-
 7 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/automation/build/opensuse/leap-15.6-x86_64.dockerfile b/automation/build/opensuse/leap-15.6-x86_64.dockerfile
index ea70dffe2afc..d9ee34ff240f 100644
--- a/automation/build/opensuse/leap-15.6-x86_64.dockerfile
+++ b/automation/build/opensuse/leap-15.6-x86_64.dockerfile
@@ -31,7 +31,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
diff --git a/automation/build/opensuse/tumbleweed-x86_64.dockerfile b/automation/build/opensuse/tumbleweed-x86_64.dockerfile
index 61e840fc67da..cd90f7357a50 100644
--- a/automation/build/opensuse/tumbleweed-x86_64.dockerfile
+++ b/automation/build/opensuse/tumbleweed-x86_64.dockerfile
@@ -31,7 +31,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
diff --git a/automation/build/ubuntu/16.04-x86_64.dockerfile b/automation/build/ubuntu/16.04-x86_64.dockerfile
index d8617bfade5e..7dc2ad8c3ef5 100644
--- a/automation/build/ubuntu/16.04-x86_64.dockerfile
+++ b/automation/build/ubuntu/16.04-x86_64.dockerfile
@@ -27,7 +27,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
diff --git a/automation/build/ubuntu/18.04-x86_64.dockerfile b/automation/build/ubuntu/18.04-x86_64.dockerfile
index 336697ad954b..12522cf48045 100644
--- a/automation/build/ubuntu/18.04-x86_64.dockerfile
+++ b/automation/build/ubuntu/18.04-x86_64.dockerfile
@@ -27,7 +27,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
diff --git a/automation/build/ubuntu/20.04-x86_64.dockerfile b/automation/build/ubuntu/20.04-x86_64.dockerfile
index 39fb297519ee..d071949bfe5c 100644
--- a/automation/build/ubuntu/20.04-x86_64.dockerfile
+++ b/automation/build/ubuntu/20.04-x86_64.dockerfile
@@ -27,7 +27,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
diff --git a/automation/build/ubuntu/22.04-x86_64.dockerfile b/automation/build/ubuntu/22.04-x86_64.dockerfile
index 230903f624d9..4f52b9d475ad 100644
--- a/automation/build/ubuntu/22.04-x86_64.dockerfile
+++ b/automation/build/ubuntu/22.04-x86_64.dockerfile
@@ -27,7 +27,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
diff --git a/automation/build/ubuntu/24.04-x86_64.dockerfile b/automation/build/ubuntu/24.04-x86_64.dockerfile
index 277f92facfd9..d125b9f739fa 100644
--- a/automation/build/ubuntu/24.04-x86_64.dockerfile
+++ b/automation/build/ubuntu/24.04-x86_64.dockerfile
@@ -27,7 +27,7 @@ RUN <<EOF
         git-core
         gzip
         patch
-        perl
+        perl-base
         pkg-config
         wget
         # libxenguest dombuilder
-- 
2.39.2

[PATCH 21/12] CI: Refresh and upgrade the Fedora container

[Andrew Cooper]

Fedora 29 is long out of date.  Move forward 5 years to Fedora 39.

Inlcude all the usual improvements.  Rework the container to be non-root, use
heredocs for legibility, and switch to the new naming scheme.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>

I know Fedora 40 the latest release, but it suffers from the same stubdom
failure as Archlinux currently does.  Fedora 39 builds fine.

Fedora is (correctly) blocking, so adding 40 would be bad move at this point
in the 4.19 release.

https://gitlab.com/xen-project/people/andyhhp/xen/-/pipelines/1372042761 is a
run with both 39 and 40, showing the failure.
---
 automation/build/fedora/29.dockerfile        | 46 ------------
 automation/build/fedora/39-x86_64.dockerfile | 74 ++++++++++++++++++++
 automation/gitlab-ci/build.yaml              |  8 +--
 3 files changed, 78 insertions(+), 50 deletions(-)
 delete mode 100644 automation/build/fedora/29.dockerfile
 create mode 100644 automation/build/fedora/39-x86_64.dockerfile

diff --git a/automation/build/fedora/29.dockerfile b/automation/build/fedora/29.dockerfile
deleted file mode 100644
index d5f017ac723d..000000000000
--- a/automation/build/fedora/29.dockerfile
+++ /dev/null
@@ -1,46 +0,0 @@
-# syntax=docker/dockerfile:1
-FROM --platform=linux/amd64 fedora:29
-LABEL maintainer.name="The Xen Project" \
-      maintainer.email="xen-devel@lists.xenproject.org"
-
-# install Xen depends
-RUN dnf -y install \
-        clang \
-        gcc \
-        gcc-c++ \
-        ncurses-devel \
-        zlib-devel \
-        python-devel \
-        python3-devel \
-        libuuid-devel \
-        pkgconfig \
-        flex \
-        bison \
-        libaio-devel \
-        glib2-devel \
-        yajl-devel \
-        pixman-devel \
-        glibc-devel \
-        make \
-        binutils \
-        git \
-        wget \
-        acpica-tools \
-        python-markdown \
-        patch \
-        checkpolicy \
-        dev86 \
-        xz-devel \
-        bzip2 \
-        nasm \
-        ocaml \
-        ocaml-findlib \
-        golang \
-        # QEMU
-        ninja-build \
-    && dnf clean all && \
-    rm -rf /var/cache/dnf
-
-RUN useradd --create-home user
-USER user
-WORKDIR /build
diff --git a/automation/build/fedora/39-x86_64.dockerfile b/automation/build/fedora/39-x86_64.dockerfile
new file mode 100644
index 000000000000..e2048a437581
--- /dev/null
+++ b/automation/build/fedora/39-x86_64.dockerfile
@@ -0,0 +1,74 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/amd64 fedora:39
+LABEL maintainer.name="The Xen Project"
+LABEL maintainer.email="xen-devel@lists.xenproject.org"
+
+RUN <<EOF
+    set -e
+
+    useradd --create-home user
+
+    dnf -y update
+
+    DEPS=(
+        # Xen
+        binutils
+        gcc
+        make
+        python3
+        # Kconfig
+        bison
+        flex
+        # Flask
+        checkpolicy
+
+        # Tools (general)
+        git-core
+        gzip
+        patch
+        perl-interpreter
+        pkgconfig
+        wget
+        # libxenguest dombuilder
+        bzip2-devel
+        libzstd-devel
+        lzo-devel
+        xz-devel
+        zlib-devel
+        # libacpi
+        acpica-tools
+        # libxl
+        libuuid-devel
+        yajl-devel
+        # RomBIOS
+        dev86
+        # Header Check
+        gcc-c++
+        # xentop
+        ncurses-devel
+        # Python bindings
+        python3-devel
+        python3-setuptools
+        # Ocaml bindings/oxenstored
+        ocaml
+        ocaml-findlib
+        # Golang bindings
+        golang
+
+        # Stubdom download/extract
+        bzip2
+
+        # Qemu build
+        glib2-devel
+        pixman-devel
+        ninja-build
+    )
+
+    dnf -y install "${DEPS[@]}"
+
+    dnf clean all
+    rm -rf /var/cache/dnf
+EOF
+
+USER user
+WORKDIR /build
diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 7ce88d38e7c7..3d35871d64f5 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -575,15 +575,15 @@ debian-bookworm-32-gcc-debug:
   variables:
     CONTAINER: debian:bookworm-i386
 
-fedora-gcc:
+fedora-39-x86_64-gcc:
   extends: .gcc-x86-64-build
   variables:
-    CONTAINER: fedora:29
+    CONTAINER: fedora:39-x86_64
 
-fedora-gcc-debug:
+fedora-39-x86_64-gcc-debug:
   extends: .gcc-x86-64-build-debug
   variables:
-    CONTAINER: fedora:29
+    CONTAINER: fedora:39-x86_64
 
 ubuntu-16.04-x86_64-clang:
   extends: .clang-x86-64-build
-- 
2.39.2

Re: [PATCH 19/12] docs: Fix install-man$(1)-pages if no manpages are generated

[Anthony PERARD]

On Sat, Jul 13, 2024 at 07:09:39PM +0100, Andrew Cooper wrote:
> All tools to build manpages are optional, and if none of them happen to be
> present, the intermediate working directory may not even be created.
> 
> Treat this as non-fatal, bringing the behaviour in line with install-html.
> Like the html side, it needs to be not-or to avoid Make thinking the rule has
> failed.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 20/12] CI: Swap from perl to perl-base in build containers

[Anthony PERARD]

On Sat, Jul 13, 2024 at 07:09:52PM +0100, Andrew Cooper wrote:
> We only need a basic perl environment, not the things that a primarily-perl
> project would want.
> 
> Discovered during the Fedora refresh where the difference is ~40M, but it's
> more modest for OpenSUSE and Ubuntu.

"perl-base" on Ubuntu is quite different from "perl-base" on OpenSUSE
Leap. I haven't check Tumbleweed but is probably the same as Leap.
So, I think these would deserve to be in separated patches.

On Ubuntu, installing "perl" or "perl-base" or none of them makes no
difference. We install "build-essentials" which pulls "dpkg-dev" which
pulls "perl", so we don't test properly if "perl-base" is enough or if
we need more.

I tried to find out which Perl module we would need, and I've got this
list:

- get_maintainer
    Getopt::Long;
- add_maintainer
    Getopt::Long;
    File::Basename;
    List::MoreUtils;
    IO::Handle;
- stubdom/vtpmmgr:
    Digest::SHA qw(sha1);
    Math::BigInt only => 'GMP';
- kconfig
    Getopt::Long;
- ocaml/lib/xc/abi-check
    Data::Dumper;
- docs/gen-html-index
    Getopt::Long;
    IO::File;
    File::Basename;
- docs/xen-headers
    Getopt::Long;
    File::Find;
    IO::File;
(there's also tools/examples/xeninfo.pl, I dont't if that can still work
or if we need to purge it)

Then I've got whatever module is in "perl-base" pkg.

For Leap's perl-base:
    getopt:long
    file:basename
    io:handle
    io:file
    file:find
    data:dumper
    digest:sha
So, add_maintainer.pl and the script in stubdom/vtpmmgr won't work. So
that's probably fine for Leap.

For Ubuntu 22.04 (not check others, but likely about the same)
there's more packages:
    perl-base
        getopt::long
        file:basename
        io:handle
        io:file
    perl-modules-5.34
        file:find
        math:bigint
    libperl5.34
        data:dumper
        digest:sha
    liblist-moreutils-perl
        list:MoreUtils
    librpc-xml-perl
        rpc:xml
("perl" pkg pulls "perl-modules-5.34" and "libperl5.34")

So with just "perl-base", the ocaml's abi-check wouldn't work, as well
as the docs's script (probably ok for "docs" if we don't use containers
to build them). Also add_maintainer and stubdom/vtpmmgr like for Leap.


So I would suggest to not touch the Ubuntu containers, make the change
to the OpenSUSE one and maybe add the limitation to the commit message.

Cheers,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 20/12] CI: Swap from perl to perl-base in build containers

[Andrew Cooper]

On 15/07/2024 1:46 pm, Anthony PERARD wrote:
> On Sat, Jul 13, 2024 at 07:09:52PM +0100, Andrew Cooper wrote:
>> We only need a basic perl environment, not the things that a primarily-perl
>> project would want.
>>
>> Discovered during the Fedora refresh where the difference is ~40M, but it's
>> more modest for OpenSUSE and Ubuntu.
> "perl-base" on Ubuntu is quite different from "perl-base" on OpenSUSE
> Leap. I haven't check Tumbleweed but is probably the same as Leap.

/sigh - because that would have been too easy.

> So, I think these would deserve to be in separated patches.
>
> On Ubuntu, installing "perl" or "perl-base" or none of them makes no
> difference. We install "build-essentials" which pulls "dpkg-dev" which
> pulls "perl", so we don't test properly if "perl-base" is enough or if
> we need more.

Fine - lets leave the Ubuntu's alone.

>
> I tried to find out which Perl module we would need, and I've got this
> list:
>
> - get_maintainer
>     Getopt::Long;
> - add_maintainer
>     Getopt::Long;
>     File::Basename;
>     List::MoreUtils;
>     IO::Handle;
> - stubdom/vtpmmgr:
>     Digest::SHA qw(sha1);
>     Math::BigInt only => 'GMP';
> - kconfig
>     Getopt::Long;
> - ocaml/lib/xc/abi-check
>     Data::Dumper;
> - docs/gen-html-index
>     Getopt::Long;
>     IO::File;
>     File::Basename;
> - docs/xen-headers
>     Getopt::Long;
>     File::Find;
>     IO::File;
> (there's also tools/examples/xeninfo.pl, I dont't if that can still work
> or if we need to purge it)

Ewww.

my %xenhosts = ("192.0.2.10" => {"port" => "9363"}, ...

is enough to demonstrate it's a special purpose script.

That file isn't referenced by the Makefile, nor is anything in the vnc/
dir, so we should purge the lot.

>
> Then I've got whatever module is in "perl-base" pkg.
>
> For Leap's perl-base:
>     getopt:long
>     file:basename
>     io:handle
>     io:file
>     file:find
>     data:dumper
>     digest:sha
> So, add_maintainer.pl and the script in stubdom/vtpmmgr won't work. So
> that's probably fine for Leap.

vtpmmgr isn't built by default, so gets 0 testing in CI.  I don't expect
people to be using add_maintainer inside the containers either.

>
> For Ubuntu 22.04 (not check others, but likely about the same)
> there's more packages:
>     perl-base
>         getopt::long
>         file:basename
>         io:handle
>         io:file
>     perl-modules-5.34
>         file:find
>         math:bigint
>     libperl5.34
>         data:dumper
>         digest:sha
>     liblist-moreutils-perl
>         list:MoreUtils
>     librpc-xml-perl
>         rpc:xml
> ("perl" pkg pulls "perl-modules-5.34" and "libperl5.34")
>
> So with just "perl-base", the ocaml's abi-check wouldn't work, as well
> as the docs's script (probably ok for "docs" if we don't use containers
> to build them).

Hmm yeah - we wouldn't want to break that.

>  Also add_maintainer and stubdom/vtpmmgr like for Leap.
>
>
> So I would suggest to not touch the Ubuntu containers, make the change
> to the OpenSUSE one and maybe add the limitation to the commit message.

Ok - I'll do this.

~Andrew

Re: [PATCH 10/12] CI: Introduce debian:11/12-riscv64 containers

[Oleksii]

On Fri, 2024-07-12 at 13:30 +0100, Andrew Cooper wrote:
> On 12/07/2024 12:39 pm, Oleksii wrote:
> > On Thu, 2024-07-11 at 12:15 +0100, Andrew Cooper wrote:
> > > For starters, they're slightly smaller:
> > > 
> > >   $ docker image list <snip>
> > >   registry.gitlab.com/xen-project/xen/debian      12-
> > > riscv64        
> > > 772MB
> > >   registry.gitlab.com/xen-project/xen/debian      11-
> > > riscv64        
> > > 422MB
> > Do we really need both 11-riscv64 and 12-riscv64?
> 
> Need? No, not strictly.
> 
> Want? Yes, absolutely.
> 
> You always want at least 2 different toolchains worth of testing, or
> what you will find happens is that you end up accidentally depending
> on
> a quirk of the single compiler your using, and that you discover this
> at
> some point in the future, rather than now(ish) when CI says no.
> 
> At the moment, the RISC-V builds are very trivial and quick, so it
> makes
> a lot of sense to have a second toolchain.  Before too much longer,
> you'll want to get Clang working too.
Agree, I should consider to add Clang. I will do that in the one of my
nearest patch series.

~ Oleksii

Re: [PATCH 21/12] CI: Refresh and upgrade the Fedora container

[Anthony PERARD]

On Sat, Jul 13, 2024 at 07:10:05PM +0100, Andrew Cooper wrote:
> Fedora 29 is long out of date.  Move forward 5 years to Fedora 39.
> 
> Inlcude all the usual improvements.  Rework the container to be non-root, use
> heredocs for legibility, and switch to the new naming scheme.

"clang" as been removed, but it's not mention in the commit message. The
other changes are kind of "the usual improvement" yes, as been done to
other containers (libaio, nasm, ...).

The patch is missing an update of the "containerize" script.

> diff --git a/automation/build/fedora/39-x86_64.dockerfile b/automation/build/fedora/39-x86_64.dockerfile
> new file mode 100644
> index 000000000000..e2048a437581
> --- /dev/null
> +++ b/automation/build/fedora/39-x86_64.dockerfile
...
> +
> +    dnf -y install "${DEPS[@]}"

You might want to add --setopt=install_weak_deps=False to avoid
installing "git" for example.
When running the original command by hand, `dnf` want to install those weak deps:
    Installing weak dependencies:
     apr-util-bdb
     apr-util-openssl
     cryptsetup-libs
     diffutils
     git
     libbpf
     libxkbcommon
     mercurial
     perl-NDBM_File
     python3-fb-re2
     python3-pip
     qrencode-libs
     subversion
     systemd-networkd
     systemd-resolved
And comparring the list of deps, these extra pkgs are installed because
of weak deps:
    apr
    apr-util
    dbus
    dbus-broker
    dbus-common
    device-mapper
    device-mapper-libs
    git-core-doc
    kmod-libs
    libargon2
    libseccomp
    libserf
    perl-Error
    perl-File-Find
    perl-Git
    perl-TermReadKey
    perl-lib
    python3-zombie-imp
    re2
    subversion-libs
    systemd
    systemd-pam
    utf8proc
    xkeyboard-config

So, probably only "perl-File-Find" (can be written "perl(File::Find)" I
think as well) might be needed, but only for docs/ I think, from the
previous email.

It seems I go from 1.26GB to 1.18GB without those weak deps.

Cheers,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 21/12] CI: Refresh and upgrade the Fedora container

[Andrew Cooper]

On 15/07/2024 2:39 pm, Anthony PERARD wrote:
> On Sat, Jul 13, 2024 at 07:10:05PM +0100, Andrew Cooper wrote:
>> Fedora 29 is long out of date.  Move forward 5 years to Fedora 39.
>>
>> Inlcude all the usual improvements.  Rework the container to be non-root, use
>> heredocs for legibility, and switch to the new naming scheme.
> "clang" as been removed, but it's not mention in the commit message.

Oops - I meant to mention that.  We don't do any clang builds with this
container, so I though I'd keep it slimer until we change that.

>  The
> other changes are kind of "the usual improvement" yes, as been done to
> other containers (libaio, nasm, ...).
>
> The patch is missing an update of the "containerize" script.

Ah - I found the hunk in a different patch.  I'll fix.

>
>> diff --git a/automation/build/fedora/39-x86_64.dockerfile b/automation/build/fedora/39-x86_64.dockerfile
>> new file mode 100644
>> index 000000000000..e2048a437581
>> --- /dev/null
>> +++ b/automation/build/fedora/39-x86_64.dockerfile
> ...
>> +
>> +    dnf -y install "${DEPS[@]}"
> You might want to add --setopt=install_weak_deps=False to avoid
> installing "git" for example.

Well, git-core is installed explicitly, but point taken.

> When running the original command by hand, `dnf` want to install those weak deps:
>     Installing weak dependencies:
>      apr-util-bdb
>      apr-util-openssl
>      cryptsetup-libs
>      diffutils
>      git
>      libbpf
>      libxkbcommon
>      mercurial
>      perl-NDBM_File
>      python3-fb-re2
>      python3-pip
>      qrencode-libs
>      subversion
>      systemd-networkd
>      systemd-resolved
> And comparring the list of deps, these extra pkgs are installed because
> of weak deps:
>     apr
>     apr-util
>     dbus
>     dbus-broker
>     dbus-common
>     device-mapper
>     device-mapper-libs
>     git-core-doc
>     kmod-libs
>     libargon2
>     libseccomp
>     libserf
>     perl-Error
>     perl-File-Find
>     perl-Git
>     perl-TermReadKey
>     perl-lib
>     python3-zombie-imp
>     re2
>     subversion-libs
>     systemd
>     systemd-pam
>     utf8proc
>     xkeyboard-config
>
> So, probably only "perl-File-Find" (can be written "perl(File::Find)" I
> think as well) might be needed, but only for docs/ I think, from the
> previous email.
>
> It seems I go from 1.26GB to 1.18GB without those weak deps.

Thanks.  I'll rework.

I have a specific plan for testing docs and it's going to be a dedicated
container and job, as well as things being a bit more -Werror-like.

~Andrew

[PATCH 22/12] tools/examples: Remove more obsolete content

[Andrew Cooper]

xeninfo.pl was introduced in commit 1b0a8bb57e3e ("Added xeninfo.pl, a script
for collecting statistics from Xen hosts using the Xen-API") and has been
touched exactly twice since to remove hardcoded IP addresses and paths.

The configuration files in vnc/* date from when we had a vendered version of
Qemu living in the tree.

These have never (AFAICT) been wired into the `make install` rule.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Anthony PERARD <anthony.perard@vates.tech>
CC: Juergen Gross <jgross@suse.com>
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Jan Beulich <JBeulich@suse.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Julien Grall <julien@xen.org>
CC: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
 tools/examples/vnc/Xservers |   5 -
 tools/examples/vnc/Xvnc-xen |  53 -------
 tools/examples/xeninfo.pl   | 284 ------------------------------------
 3 files changed, 342 deletions(-)
 delete mode 100644 tools/examples/vnc/Xservers
 delete mode 100755 tools/examples/vnc/Xvnc-xen
 delete mode 100644 tools/examples/xeninfo.pl

diff --git a/tools/examples/vnc/Xservers b/tools/examples/vnc/Xservers
deleted file mode 100644
index adc5748be6ba..000000000000
--- a/tools/examples/vnc/Xservers
+++ /dev/null
@@ -1,5 +0,0 @@
-# Configuration lines to go in /etc/X11/xdm/Xservers to
-# start Xvnc and connect back to a vncviewer in domain-0.
-# See 'man xdm' under 'LOCAL SERVER SPECIFICATION' for format details.
-
-:1 Xvnc local /usr/X11R6/bin/Xvnc-xen :1
\ No newline at end of file
diff --git a/tools/examples/vnc/Xvnc-xen b/tools/examples/vnc/Xvnc-xen
deleted file mode 100755
index 15058e622d7f..000000000000
--- a/tools/examples/vnc/Xvnc-xen
+++ /dev/null
@@ -1,53 +0,0 @@
-#!/bin/bash
-#============================================================================
-# This script should be installed in /usr/X11R6/bin/Xvnc-xen.
-#============================================================================
-#
-# Start Xvnc and use vncconnect to connect back to a vncviewer listening in
-# domain 0. The host and port to connect to are given by
-#
-#    VNC_VIEWER=<host>:<port>
-#
-# in the kernel command line (/proc/cmdline). 
-#
-# The '--vnc' option to 'xm create' will start a vncviewer and
-# pass its address in VNC_VIEWER for this script to find.
-#
-# Usage:
-#        Xvnc-xen [args]
-#
-# Any arguments are passed to Xvnc.
-#
-#============================================================================
-
-# Prefix for messages.
-M="[$(basename $0)]"
-
-# Usage: vnc_addr
-# Print <host>:<port> for the vncviewer given in
-# the kernel command line.
-vnc_addr () {
-    sed -n -e "s/.*VNC_VIEWER=\([^ ]*\).*/\1/p" /proc/cmdline
-}
-
-# Usage: vnc_connect
-# If a vncviewer address was given on the kernel command line,
-# run vncconnect for it.
-vnc_connect () {
-    local addr=$(vnc_addr)
-
-    if [ -n "${addr}" ] ; then
-        echo "$M Connecting to ${addr}."
-        vncconnect ${addr}
-    else
-        echo "$M No VNC_VIEWER in kernel command line."
-        echo "$M Create the domain with 'xm create --vnc <display>'."
-        return 1
-    fi
-}
-
-# Start the vnc server.
-Xvnc "$@" >/dev/null 2>&1 &
-
-# Connect back to the viewer in domain-0.
-vnc_connect
diff --git a/tools/examples/xeninfo.pl b/tools/examples/xeninfo.pl
deleted file mode 100644
index 95a15d9595c0..000000000000
--- a/tools/examples/xeninfo.pl
+++ /dev/null
@@ -1,284 +0,0 @@
-#!/usr/bin/perl -w
-
-#############################################################################################################
-#                                                                                                           #
-#  Developed by Ingard Mevåg @ Oslo University College, spring 2007                                         #
-#  ingard [at] mevaag  [dot] no                                                                             #
-#                                                                                                           #
-#  This work is licensed under the Creative Commons Attribution-Noncommercial-Share Alike 3.0 License.      #
-#  To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/ or send a letter #
-#  to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.                #
-#                                                                                                           #
-#############################################################################################################
-
-use strict;
-# http://search.cpan.org/~rjray/RPC-XML-0.59/lib/RPC/XML/Client.pm
-require RPC::XML;
-require RPC::XML::Client;
-
-# for debug purposes
-#use Data::Dumper;
-
-##### CONFIG ######
-
-my %xenhosts = ("192.0.2.10" => {"port" => "9363"}, 
-					 "192.0.2.11" => {"port" => "9363"}, 
-					 "192.0.2.12" => {"port" => "9363"}, 
-					 "192.0.2.13" => {"port" => "9363"});
-
-##### CONFIG END ###
-
-##### STATIC VARS #####
-my %host_info;
-
-#######################
-sub apiconnect
-{
-	foreach my $xenhost (keys %xenhosts)
-	{
-		my $xen = RPC::XML::Client->new("http://$xenhost:$xenhosts{$xenhost}{'port'}");
-		my $session = $xen->simple_request("session.login_with_password", "user","");
-		if (! $session)
-		{
-			print "Can't connect to $xenhost :(\n";
-			$xenhosts{$xenhost} = {'xen' => $xen, 'session' => ""};
-		}
-		else
-		{
-			$xenhosts{$xenhost} = {'xen' => $xen, 'session' => $session->{'Value'}};
-			print "Connected successfully to $xenhost..\n";
-		}
-	}
-}
-
-sub validate_response
-{
-	my ($result_ref) = @_;
-	if ($result_ref->{'Status'} eq "Success")
-	{
-		return $result_ref->{'Value'};
-	}
-	else
-	{
-		# status = Failure !
-#		die ("xmlrpc failed! ErrorDescription: $result_ref->{'ErrorDescription'}[1] -> $result_ref->{'ErrorDescription'}[0]");
-		print "xmlrpc failed! ErrorDescription: $result_ref->{'ErrorDescription'}[1] -> $result_ref->{'ErrorDescription'}[0]\n";
-	}
-}
-
-sub get_host_cpu_utilisation
-{
-	my ($xen, $session, $host_name, $host_ref) = @_;
-	my $host_cpu_ref = validate_response($xen->simple_request("host.get_host_CPUs", $session, $host_ref));
-	foreach (@$host_cpu_ref)
-	{
-		my $host_cpu_utilisation = validate_response($xen->simple_request("host_cpu.get_utilisation", $session, $_));
-		$host_info{$host_name}{'cpus'}{$_} = $host_cpu_utilisation;
-		print "     CPUiNFO: $host_cpu_utilisation\n";
-	}
-}
-
-sub get_host_pif_utilisation
-{
-	my ($xen, $session, $host_name, $host_ref) = @_;
-
-# This method isnt implemented yet it seems so using PIF.get_all for now.. 
-# This will break when xen is made cluster aware..
-#	my $host_pif_ref = validate_response($xen->simple_request("host.get_PIFs", $session, $host_ref)); 
-	my $host_pif_ref = validate_response($xen->simple_request("PIF.get_all", $session));
-	foreach (@$host_pif_ref)
-	{
-		my $host_pif_device = validate_response($xen->simple_request("PIF.get_device", $session, $_));
-		my $host_pif_metrics_ref = validate_response($xen->simple_request("PIF.get_metrics", $session, $_));
-
-# Whats the best solution performancewise?
-# Collecting stats from get_records, or pulling individually?
-
-#		my $host_pif_record = validate_response($xen->simple_request("PIF_metrics.get_record", $session, $host_pif_metrics_ref));
-#		my $host_pif_io_read = $host_pif_record->{'io_read_kbs'};
-#		my $host_pif_io_write = $host_pif_record->{'io_write_kbs'};
-		my $host_pif_io_read = validate_response($xen->simple_request("PIF_metrics.get_io_read_kbs", $session, $host_pif_metrics_ref));
-		my $host_pif_io_write = validate_response($xen->simple_request("PIF_metrics.get_io_write_kbs", $session, $host_pif_metrics_ref));
-
-		$host_info{$host_name}{'pifs'}{$host_pif_device} = {'read' => $host_pif_io_read, 'write' => $host_pif_io_write};
-		print "     PiFiNFO: $host_pif_device READ: $host_pif_io_read - WRITE: $host_pif_io_write\n";
-#		$host_info{$host_name}{'pifs'}{$host_pif_device}{'read'} = $host_pif_io_read;
-#		$host_info{$host_name}{'pifs'}{$host_pif_device}{'write'} = $host_pif_io_write;
-	}
-}
-
-sub get_host_mem_utilisation
-{
-	my ($xen, $session, $host_name, $host_ref) = @_;
-	my $host_metrics_ref = validate_response($xen->simple_request("host.get_metrics", $session, $host_ref)); 
-	my $host_mem_total =  validate_response($xen->simple_request("host_metrics.get_memory_total", $session, $host_metrics_ref)) / 1024 / 1024;
-	my $host_mem_free =  validate_response($xen->simple_request("host_metrics.get_memory_free", $session, $host_metrics_ref)) / 1024 / 1024;
-	$host_info{$host_name}{'memory'} = {'total' => $host_mem_total, 'free' => $host_mem_free};
-	print "     MEMiNFO: Total: $host_mem_total MB - Free: $host_mem_free MB\n";
-}
-
-sub get_vm_mem_info
-{
-	my ($xen, $session, $host_name, $vm_ref, $vm_name_label) = @_;
-	my $vm_mem_stat_max = validate_response($xen->simple_request("VM.get_memory_static_max",$session,$vm_ref));
-	my $vm_mem_stat_min = validate_response($xen->simple_request("VM.get_memory_static_min",$session,$vm_ref));
-	my $vm_mem_dyn_max = validate_response($xen->simple_request("VM.get_memory_dynamic_max",$session,$vm_ref));
-	my $vm_mem_dyn_min = validate_response($xen->simple_request("VM.get_memory_dynamic_min",$session,$vm_ref));
-
-	# not implemented yet.. We'll do this at the same time as getting cpu utilisation
-	# in the get_vm_metrics sub instead..
-	#my $vm_metrics_ref = validate_response($xen->simple_request("VM.get_metrics",$session,$vm_ref));
-	#my $vm_mem_actual = validate_response($xen->simple_request("VM_metrics.get_memory_actual",$session,$vm_metrics_ref));
-
-	$host_info{$host_name}{'vms'}{$vm_name_label}{'memory'} = {'static_max' => $vm_mem_stat_max,
-								  'static_min' => $vm_mem_stat_min,
-								  'dynamic_max' => $vm_mem_dyn_max,
-								  'dynamic_min' => $vm_mem_dyn_min};
-
-	# xm list uses the dynamic min var as far as i can tell.. or?
-	# Lets print the memactual info instead of this... I'll do that in the get_vm_metrics sub instead..
-	# print "  |- MEMiNFO: Dynamic Min: $vm_mem_dyn_min - Actually in use: $vm_mem_actual\n";
-}
-
-sub get_vm_metrics
-{
-	my ($xen, $session, $host_name, $vm_ref, $vm_name_label) = @_;
-	my $vm_metrics_ref = validate_response($xen->simple_request("VM.get_metrics",$session,$vm_ref));
-	
-	my %vm_vcpu_utilisation = %{validate_response($xen->simple_request("VM_metrics.get_vcpus_utilisation",$session,$vm_metrics_ref))};
-	for my $tempcpu (keys %vm_vcpu_utilisation)
-	{
-		print "  |- CPUiNFO: $tempcpu - $vm_vcpu_utilisation{$tempcpu}\n";
-		$host_info{$host_name}{'vms'}{$vm_name_label}{'vcpus'} = {$tempcpu => $vm_vcpu_utilisation{$tempcpu}};
-	}
-	my $vm_mem_actual = validate_response($xen->simple_request("VM_metrics.get_memory_actual",$session,$vm_metrics_ref)) / 1024 / 1024;
-	$host_info{$host_name}{'vms'}{$vm_name_label}{'memory'}{'actual'} = "$vm_mem_actual";
-	print "  |- MEMiNFO: Actually in use: $vm_mem_actual MB\n";
-}
-
-sub get_vm_vif_utilisation
-{
-	my ($xen, $session, $host_name, $vm_ref, $vm_name_label) = @_;
-	my $vm_vifs = validate_response($xen->simple_request("VM.get_VIFs",$session,$vm_ref));
-	foreach (@$vm_vifs)
-	{
-		my $vif_device = validate_response($xen->simple_request("VIF.get_device",$session,$_));
-		my $vif_io_read = validate_response($xen->simple_request("VIF_metrics.get_io_read_kbs", $session, $_));
-		my $vif_io_write = validate_response($xen->simple_request("VIF_metrics.get_io_write_kbs", $session, $_));
-		$host_info{$host_name}{'vms'}{$vm_name_label}{'vifs'}{$vif_device} = {'read' => $vif_io_read, 'write' => $vif_io_write};
-		print "  |- ViFiNFO: $vif_device READ: $vif_io_read - WRITE: $vif_io_write\n";
-	}
-}
-
-sub get_vm_vbd_utilisation
-{
-	my ($xen, $session, $host_name, $vm_ref, $vm_name_label) = @_;
-	my $vm_vbds = validate_response($xen->simple_request("VM.get_VBDs",$session,$vm_ref));
-	foreach (@$vm_vbds)
-	{
-		my $vbd_device = validate_response($xen->simple_request("VBD.get_device",$session,$_));
-		my $vbd_io_read = validate_response($xen->simple_request("VBD_metrics.get_io_read_kbs", $session, $_));
-		my $vbd_io_write = validate_response($xen->simple_request("VBD_metrics.get_io_write_kbs", $session, $_));
-		$host_info{$host_name}{'vms'}{$vm_name_label}{'vbds'}{$vbd_device} = {'read' => $vbd_io_read, 'write' => $vbd_io_write};
-		print "  |- VBDiNFO: $vbd_device READ: $vbd_io_read - WRITE: $vbd_io_write\n";
-	}
-}
-
-
-sub get_vm_type
-{
-	my ($xen, $session, $host_name, $vm_ref, $vm_name_label) = @_;
-	# not running response through validate_response() here to stop it from crashing..
-	#
-	# api docs says if this (following) field is set, its a HVM domain.
-	my $vm_bootloader_results = $xen->simple_request("VM.get_HVM_boot_policy",$session,$vm_ref);
-	if ("$vm_bootloader_results->{'Status'}" eq "Success")
-	{
-		if ("$vm_bootloader_results->{'Value'}" ne "")
-		{
-			$host_info{$host_name}{'vms'}{$vm_name_label}{'type'} = "HVM";
-		}
-		else
-		{
-			$host_info{$host_name}{'vms'}{$vm_name_label}{'type'} = "PV";
-		}
-	}
-	else
-	{
-		# However, xen 3.0.4 doest support this part of the api, so afaik I can get the difference with: 
-		my $vm_pv_kernel_results = $xen->simple_request("VM.get_PV_kernel",$session,$vm_ref);
-		# which is something like:
-		# 'PV_kernel': '/boot/vmlinuz-2.6.18-xen',
-		# or
-		# 'PV_kernel': 'hvmloader',
-		if ("$vm_pv_kernel_results->{'Value'}" =~ m/hvm/i)
-		{
-			$host_info{$host_name}{'vms'}{$vm_name_label}{'type'} = "HVM";
-		}
-		else
-		{
-			$host_info{$host_name}{'vms'}{$vm_name_label}{'type'} = "PV";
-		}
-	}
-}
-
-sub get_complete_info
-{	
-	my %all_vms;
-	foreach my $xenhost (sort keys %xenhosts)
-	{
-		next unless $xenhosts{$xenhost}{'session'};
-		my $xen = $xenhosts{$xenhost}{'xen'};
-		my $session = $xenhosts{$xenhost}{'session'};
-		print "_______________________\n## $xenhost ##\n-----------------------\n";
-	
-		my $host_ref = validate_response($xen->simple_request("session.get_this_host", $session));
-		
-		my $host_name = validate_response($xen->simple_request("host.get_name_label", $session, $host_ref));
-		$xenhosts{$xenhost}{'hostname'} = $host_name;
-		$host_info{$host_name}{'ip'} = $xenhost;
-		
-		get_host_cpu_utilisation($xen, $session, $host_name, $host_ref);
-
-		get_host_mem_utilisation($xen, $session, $host_name, $host_ref);
-	
-		get_host_pif_utilisation($xen, $session, $host_name, $host_ref);
-	
-	
-		my $all_vm_refs = validate_response($xen->simple_request("host.get_resident_VMs",$session, $host_ref));
-		
-		foreach my $vm_ref (@$all_vm_refs)
-		{
-			my $vm_name_label = validate_response($xen->simple_request("VM.get_name_label",$session,$vm_ref));
-			get_vm_type($xen,$session,$host_name,$vm_ref,$vm_name_label);
-			
-			my $vm_id = validate_response($xen->simple_request("VM.get_domid",$session,$vm_ref));
-
-			print "vm: $vm_id\t$vm_name_label\ttype: $host_info{$host_name}{'vms'}->{$vm_name_label}{'type'}\n";
-			
-			# vm_metrics includes both mem_actual & cpu utilisation
-			# So we'll add all stats found in that class in one go..
-			get_vm_metrics($xen,$session,$host_name,$vm_ref,$vm_name_label);
-#			get_vm_cpu_utilisation($xen,$session,$host_name,$vm_ref,$vm_name_label);
-
-			# all other mem stats are added seperately..
-			# This might not be needed at all as xen doesnt have functionality to
-			# resize mem for a VM atm (afaik)
-			get_vm_mem_info($xen,$session,$host_name,$vm_ref,$vm_name_label);
-	
-			get_vm_vif_utilisation($xen,$session,$host_name,$vm_ref,$vm_name_label);
-			
-			get_vm_vbd_utilisation($xen,$session,$host_name,$vm_ref,$vm_name_label);
-			
-			$all_vms{$vm_name_label} = "" unless ("$vm_name_label" eq "Domain-0");
-		}
-		print "\n";
-	}
-	# Debug: Uncomment to see the nested datastructure..
-	#print Dumper(%host_info);
-}
-
-
-
-apiconnect();
-get_complete_info();
-- 
2.39.2

Re: [PATCH 22/12] tools/examples: Remove more obsolete content

[Anthony PERARD]

On Mon, Jul 15, 2024 at 04:16:40PM +0100, Andrew Cooper wrote:
> xeninfo.pl was introduced in commit 1b0a8bb57e3e ("Added xeninfo.pl, a script
> for collecting statistics from Xen hosts using the Xen-API") and has been
> touched exactly twice since to remove hardcoded IP addresses and paths.

Xen-API, is that xapi?

> The configuration files in vnc/* date from when we had a vendered version of
> Qemu living in the tree.

I guess QEMU at that time didn't have VNC support? Because looks like
the vnc config was supposed to be put in a guest, and `xm` had support
for it.

> These have never (AFAICT) been wired into the `make install` rule.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Anyway, patch looks good:
Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 22/12] tools/examples: Remove more obsolete content

[Andrew Cooper]

On 15/07/2024 5:53 pm, Anthony PERARD wrote:
> On Mon, Jul 15, 2024 at 04:16:40PM +0100, Andrew Cooper wrote:
>> xeninfo.pl was introduced in commit 1b0a8bb57e3e ("Added xeninfo.pl, a script
>> for collecting statistics from Xen hosts using the Xen-API") and has been
>> touched exactly twice since to remove hardcoded IP addresses and paths.
> Xen-API, is that xapi?

Yes.  xapi is a contraction of Xen-API, but even the docs intermix the
two names.  https://xapi-project.github.io/xen-api/

>
>> The configuration files in vnc/* date from when we had a vendered version of
>> Qemu living in the tree.
> I guess QEMU at that time didn't have VNC support? Because looks like
> the vnc config was supposed to be put in a guest, and `xm` had support
> for it.

Thinking about it, this may have been for PV guests instead.  We used to
have a mode in XenServer to inject vnc configuration into a PV guest in
order to get a GUI installer rather than the TUI one.

Either way, xl doesn't have --vnc, although it does have -V which seems
to have subsumed this functionality.


>
>> These have never (AFAICT) been wired into the `make install` rule.
>>
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Anyway, patch looks good:
> Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

~Andrew