[PATCH TEST-ARTEFACTS 0/8] Cleanup and Linux ARM64 support

[PATCH TEST-ARTEFACTS 0/8] Cleanup and Linux ARM64 support

[Andrew Cooper]

Various bits of cleanup, and support for arm64 Linux builds.

Run using the new Linux 6.6.86 on (most) x86, and ARM64:
  https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1760667411

Still to go:
 - Merge argo into the linux build (as it builds a module), strip devel artefacts
 - Rootfs generation, both x86 and ARM64

The argo kernel module and userspace should be a CPIO fragment too, as it's
embedded into both dom0 and domU in the relevant test.

Switching from tar to cpio can happen when the artefact name changes; which
fixes the backwards comptibility concerns.  In hindsight, domU shouldn't be
automatically embedded in dom0, as several tests further customise it; the
test job can adjust, then wrap the whole lot in a CPIO and append it to
dom0's.

Xen's main build jobs should either build with --prefix=/usr, or the common
rootfs wants to set up /usr/local/, because right now it's done by all jobs.

Andrew Cooper (7):
  Port containerise
  Fix container user setup
  Clean up Gitlab yaml
  Adjust Linux build script to work with other major versions
  Factor our x86-isms in the linux build script
  Infrastructure for arm64 linux builds
  Linux 6.6.86 for x86 and arm64

Marek Marczykowski-Górecki (1):
  Consistently use DOCKER_CMD in makefiles

 .gitlab-ci.yml                                | 43 ++++++---
 containerize                                  | 95 +++++++++++++++++++
 images/Makefile                               |  5 +-
 ...dockerfile => 3.18-arm64-build.dockerfile} | 22 ++---
 images/alpine/x86_64-build.dockerfile         |  7 +-
 scripts/build-linux.sh                        | 54 +++++++++++
 scripts/x86_64-kernel-linux.sh                | 31 ------
 7 files changed, 197 insertions(+), 60 deletions(-)
 create mode 100755 containerize
 copy images/alpine/{x86_64-build.dockerfile => 3.18-arm64-build.dockerfile} (55%)
 create mode 100755 scripts/build-linux.sh
 delete mode 100755 scripts/x86_64-kernel-linux.sh

-- 
2.39.5

[PATCH 1/8] Consistently use DOCKER_CMD in makefiles

[Andrew Cooper]

From: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

This allows rebuilding containers using podman too.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
[Ported from Xen]
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
 images/Makefile | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/images/Makefile b/images/Makefile
index 42f231bcce27..b0b7a7cf63ee 100644
--- a/images/Makefile
+++ b/images/Makefile
@@ -1,3 +1,4 @@
+DOCKER_CMD ?= docker
 
 # The base of where these containers will appear
 REGISTRY := registry.gitlab.com/xen-project/hardware/test-artifacts
@@ -11,9 +12,9 @@ help:
 	@echo "To push container builds, set the environment variable PUSH"
 
 %: %.dockerfile ## Builds containers
-	docker build --pull -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
+	$(DOCKER_CMD) build --pull -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D)
 	@if [ ! -z $${PUSH+x} ]; then \
-		docker push $(REGISTRY)/$(@D):$(@F); \
+		$(DOCKER_CMD) push $(REGISTRY)/$(@D):$(@F); \
 	fi
 
 .PHONY: all
-- 
2.39.5

[PATCH 2/8] Port containerise

[Andrew Cooper]

While it pains me to keep the wrong spelling, do so for consistency.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
CC: Anthony PERARD <anthony.perard@vates.tech>
---
 containerize | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 94 insertions(+)
 create mode 100755 containerize

diff --git a/containerize b/containerize
new file mode 100755
index 000000000000..c32e40f5a6d9
--- /dev/null
+++ b/containerize
@@ -0,0 +1,94 @@
+#!/bin/bash
+
+#
+# DOCKER_CMD should be either `docker` or `podman`.
+#
+# if using (rootless) podman, remember to set /etc/subuid
+# and /etc/subgid.
+#
+docker_cmd=${DOCKER_CMD:-"docker"}
+[ "$DOCKER_CMD" = "podman" ] && userns_podman="--userns=keep-id" selinux=",z"
+
+einfo() {
+    echo "$*" >&2
+}
+
+die() {
+    echo "$*" >&2
+    exit 1
+}
+
+#
+# The caller is expected to override the CONTAINER environment
+# variable with the container they wish to launch.
+#
+BASE="registry.gitlab.com/xen-project/hardware/test-artifacts"
+case "_${CONTAINER}" in
+    _alpine-x86_64-base) CONTAINER="${BASE}/alpine:x86_64-base" ;;
+    _alpine-x86_64-build|_) CONTAINER="${BASE}/alpine:x86_64-build" ;;
+esac
+
+# Use this variable to control whether root should be used
+case "_${CONTAINER_UID0}" in
+    _1)   userarg="-u 0" ;;
+    _0|_) userarg="-u $(id -u) $userns_podman" ;;
+esac
+
+# Save the commands for future use
+cmd=("$@")
+
+# If no command was specified, just drop us into a shell if we're interactive
+[ $# -eq 0 ] && tty -s && cmd=("/bin/bash")
+
+# Are we in an interactive terminal?
+tty -s && termint=t
+
+#
+# Fetch the latest version of the container in hub.docker.com,
+# unless it's a newly created local copy.
+#
+if [[ "_${CONTAINER_NO_PULL}" != "_1" ]]; then
+    einfo "*** Ensuring ${CONTAINER} is up to date"
+    ${docker_cmd} pull ${CONTAINER} > /dev/null ||     \
+        die "Failed to update container"
+fi
+
+if hash greadlink > /dev/null 2>&1; then
+    READLINK=greadlink
+elif [[ $(uname -s) == "Darwin" ]]; then
+    echo "Unable to forward SSH agent without coreutils installed"
+    unset SSH_AUTH_SOCK
+else
+    READLINK=readlink
+fi
+
+# Ensure we've got what we need for SSH_AUTH_SOCK
+if [[ -n ${SSH_AUTH_SOCK} ]]; then
+    fullpath_sock=$(${READLINK} -f ${SSH_AUTH_SOCK} 2> /dev/null)
+    if [ $? -ne 0 ]; then
+        echo "Invalid SSH_AUTH_SOCK: ${SSH_AUTH_SOCK}"
+        unset SSH_AUTH_SOCK
+    else
+        SSH_AUTH_DIR=$(dirname ${fullpath_sock})
+        SSH_AUTH_NAME=$(basename ${fullpath_sock})
+    fi
+fi
+
+# Figure out the base of what we want as our sources
+# by using the top of the git repo
+if [[ -z ${CONTAINER_PATH} ]]; then
+    CONTAINER_PATH=$(git rev-parse --show-toplevel)
+fi
+
+# Kick off Docker
+einfo "*** Launching container ..."
+exec ${docker_cmd} run \
+    ${userarg} \
+    ${SSH_AUTH_SOCK:+-e SSH_AUTH_SOCK="/tmp/ssh-agent/${SSH_AUTH_NAME}"} \
+    -v "${CONTAINER_PATH}":/build:rw${selinux} \
+    -v "${HOME}/.ssh":/root/.ssh:ro \
+    ${SSH_AUTH_DIR:+-v "${SSH_AUTH_DIR}":/tmp/ssh-agent${selinux}} \
+    ${CONTAINER_ARGS} \
+    -${termint}i --rm -- \
+    ${CONTAINER} \
+    "${cmd[@]}"
-- 
2.39.5

[PATCH 3/8] Fix container user setup

[Andrew Cooper]

Use USER (and 'user' as normal), rather than specifying configuration from the
outside.  This also fixes the fact that root-needing containers need to
override the default.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
CC: Anthony PERARD <anthony.perard@vates.tech>
---
 .gitlab-ci.yml                        | 7 -------
 images/alpine/x86_64-build.dockerfile | 7 ++++---
 2 files changed, 4 insertions(+), 10 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 7e18a87b3a07..36ec6a7e1ee5 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -14,10 +14,6 @@ workflow:
 
 .x86_64-artifacts:
   extends: .artifacts
-  image:
-    docker:
-      platform: linux/amd64
-      user: xenproject
   tags:
     - x86_64
   variables:
@@ -32,9 +28,6 @@ x86_64-kernel-linux-6.6.56:
 
 x86_64-rootfs-alpine-3.18:
   extends: .x86_64-artifacts
-  image:
-    docker:
-      user: root
   script:
     - . scripts/x86_64-rootfs-alpine.sh
   variables:
diff --git a/images/alpine/x86_64-build.dockerfile b/images/alpine/x86_64-build.dockerfile
index 3bfd17248d0a..6bfdb6ff5a0c 100644
--- a/images/alpine/x86_64-build.dockerfile
+++ b/images/alpine/x86_64-build.dockerfile
@@ -3,12 +3,10 @@ FROM --platform=linux/amd64 alpine:3.18
 LABEL maintainer.name="The Xen Project" \
       maintainer.email="xen-devel@lists.xenproject.org"
 
-WORKDIR /build/
-
 RUN apk --no-cache add bash
 
 RUN bash -ex <<EOF
-      adduser -D xenproject --shell bash
+      adduser -D user --shell bash
 
       DEPS=(
             musl-dev
@@ -29,3 +27,6 @@ RUN bash -ex <<EOF
 
       apk add --no-cache "\${DEPS[@]}"
 EOF
+
+USER user
+WORKDIR /build
-- 
2.39.5

[PATCH 4/8] Clean up Gitlab yaml

[Andrew Cooper]

Factor out the registry into a common location.  Fix the worflow name.  List
all the stages.  Set a default expiry of 1 month.

Note all the current jobs as legacy.  Their naming scheme needs changing, and
we'll use this opportunity to switch formats too.  However, the artefacts need
to stay using the old name until the final staging-* branch using them is
phased out.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
CC: Anthony PERARD <anthony.perard@vates.tech>
---
 .gitlab-ci.yml | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 36ec6a7e1ee5..5a0a853e551d 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,12 +1,19 @@
+variables:
+  REGISTRY: registry.gitlab.com/xen-project/hardware/test-artifacts
+
 workflow:
-  name: "xen test artifacts"
+  name: "Xen test artifacts"
+
+stages:
+  - build
 
 .artifacts:
   stage: build
   image:
-    name: registry.gitlab.com/xen-project/hardware/test-artifacts/${CONTAINER}
+    name: ${REGISTRY}/${CONTAINER}
   artifacts:
     name: "${CI_JOB_NAME_SLUG}"
+    expire_in: 1 month
     paths:
       - binaries/
     exclude:
@@ -19,6 +26,9 @@ workflow:
   variables:
     CONTAINER: alpine:x86_64-build
 
+#
+# The jobs below here are legacy and being phased out.
+#
 x86_64-kernel-linux-6.6.56:
   extends: .x86_64-artifacts
   script:
-- 
2.39.5

[PATCH 5/8] Adjust Linux build script to work with other major versions

[Andrew Cooper]

Also use 'tar o' to decompress based on file name.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
CC: Anthony PERARD <anthony.perard@vates.tech>
---
 scripts/x86_64-kernel-linux.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/scripts/x86_64-kernel-linux.sh b/scripts/x86_64-kernel-linux.sh
index 2a816f0b3204..5a0160655bea 100755
--- a/scripts/x86_64-kernel-linux.sh
+++ b/scripts/x86_64-kernel-linux.sh
@@ -11,9 +11,10 @@ WORKDIR="${PWD}"
 COPYDIR="${WORKDIR}/binaries/"
 
 # Build Linux
+MAJOR=${LINUX_VERSION%%.*}
 curl -fsSLO \
-    https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-"${LINUX_VERSION}".tar.xz
-tar xJf linux-"${LINUX_VERSION}".tar.xz
+    https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
+tar oxf linux-"${LINUX_VERSION}".tar.xz
 cd linux-"${LINUX_VERSION}"
 make ARCH=x86 defconfig
 make ARCH=x86 xen.config
-- 
2.39.5

[PATCH 6/8] Factor our x86-isms in the linux build script

[Andrew Cooper]

... in preparation to use it for arm64 too.  Rename the script.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
CC: Anthony PERARD <anthony.perard@vates.tech>
---
 .gitlab-ci.yml                                |  3 +-
 ...{x86_64-kernel-linux.sh => build-linux.sh} | 34 +++++++++++++------
 2 files changed, 24 insertions(+), 13 deletions(-)
 rename scripts/{x86_64-kernel-linux.sh => build-linux.sh} (55%)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 5a0a853e551d..ff8dce7be05d 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -31,8 +31,7 @@ stages:
 #
 x86_64-kernel-linux-6.6.56:
   extends: .x86_64-artifacts
-  script:
-    - . scripts/x86_64-kernel-linux.sh
+  script: ./scripts/build-linux.sh
   variables:
     LINUX_VERSION: 6.6.56
 
diff --git a/scripts/x86_64-kernel-linux.sh b/scripts/build-linux.sh
similarity index 55%
rename from scripts/x86_64-kernel-linux.sh
rename to scripts/build-linux.sh
index 5a0160655bea..5e25e958c0f3 100755
--- a/scripts/x86_64-kernel-linux.sh
+++ b/scripts/build-linux.sh
@@ -9,6 +9,7 @@ set -ex -o pipefail
 
 WORKDIR="${PWD}"
 COPYDIR="${WORKDIR}/binaries/"
+UNAME=$(uname -m)
 
 # Build Linux
 MAJOR=${LINUX_VERSION%%.*}
@@ -16,17 +17,28 @@ curl -fsSLO \
     https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
 tar oxf linux-"${LINUX_VERSION}".tar.xz
 cd linux-"${LINUX_VERSION}"
-make ARCH=x86 defconfig
-make ARCH=x86 xen.config
+
+make defconfig
 ./scripts/config --enable BRIDGE
 ./scripts/config --enable IGC
 ./scripts/config --enable TUN
-cp .config .config.orig
-cat .config.orig \
-    | grep 'XEN' \
-    | grep '=m' \
-    | sed 's/=m/=y/g' \
-    >> .config
-make ARCH=x86 olddefconfig
-make -s -j "$(nproc)" ARCH=x86
-cp arch/x86/boot/bzImage "${COPYDIR}"
+
+case $UNAME in
+    x86_64)
+        make xen.config
+        cp .config .config.orig
+        cat .config.orig \
+            | grep 'XEN' \
+            | grep '=m' \
+            | sed 's/=m/=y/g' >> .config
+        ;;
+esac
+
+make olddefconfig
+
+case $UNAME in
+    x86_64)
+        make -j$(nproc) bzImage
+        cp arch/x86/boot/bzImage "${COPYDIR}"
+        ;;
+esac
-- 
2.39.5

[PATCH 7/8] Infrastructure for arm64 linux builds

[Andrew Cooper]

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
CC: Anthony PERARD <anthony.perard@vates.tech>
---
 .gitlab-ci.yml                            |  7 ++++++
 containerize                              |  1 +
 images/alpine/3.18-arm64-build.dockerfile | 27 +++++++++++++++++++++++
 scripts/build-linux.sh                    | 10 +++++++++
 4 files changed, 45 insertions(+)
 create mode 100644 images/alpine/3.18-arm64-build.dockerfile

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index ff8dce7be05d..6e38c2f2a108 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -19,6 +19,13 @@ stages:
     exclude:
       - binaries/.gitignore
 
+.arm64-artifacts:
+  extends: .artifacts
+  tags:
+    - arm64
+  variables:
+    CONTAINER: alpine:3.18-arm64-build
+
 .x86_64-artifacts:
   extends: .artifacts
   tags:
diff --git a/containerize b/containerize
index c32e40f5a6d9..6664ef9821d0 100755
--- a/containerize
+++ b/containerize
@@ -24,6 +24,7 @@ die() {
 #
 BASE="registry.gitlab.com/xen-project/hardware/test-artifacts"
 case "_${CONTAINER}" in
+    _alpine-3.18-arm64-build) CONTAINER="${BASE}/alpine:3.18-arm64-build" ;;
     _alpine-x86_64-base) CONTAINER="${BASE}/alpine:x86_64-base" ;;
     _alpine-x86_64-build|_) CONTAINER="${BASE}/alpine:x86_64-build" ;;
 esac
diff --git a/images/alpine/3.18-arm64-build.dockerfile b/images/alpine/3.18-arm64-build.dockerfile
new file mode 100644
index 000000000000..c76ac435e8dd
--- /dev/null
+++ b/images/alpine/3.18-arm64-build.dockerfile
@@ -0,0 +1,27 @@
+# syntax=docker/dockerfile:1
+FROM --platform=linux/arm64/v8 alpine:3.18
+LABEL maintainer.name="The Xen Project" \
+      maintainer.email="xen-devel@lists.xenproject.org"
+
+RUN apk --no-cache add bash
+
+RUN bash -ex <<EOF
+      adduser -D user --shell bash
+
+      DEPS=(# Base environment
+            musl-dev
+            build-base
+            curl
+
+            # Linux build deps
+            flex
+            bison
+            perl
+            openssl-dev
+      )
+
+      apk add --no-cache "\${DEPS[@]}"
+EOF
+
+USER user
+WORKDIR /build
diff --git a/scripts/build-linux.sh b/scripts/build-linux.sh
index 5e25e958c0f3..1be20a38071d 100755
--- a/scripts/build-linux.sh
+++ b/scripts/build-linux.sh
@@ -21,6 +21,7 @@ cd linux-"${LINUX_VERSION}"
 make defconfig
 ./scripts/config --enable BRIDGE
 ./scripts/config --enable IGC
+./scripts/config --enable IPV6
 ./scripts/config --enable TUN
 
 case $UNAME in
@@ -32,6 +33,10 @@ case $UNAME in
             | grep '=m' \
             | sed 's/=m/=y/g' >> .config
         ;;
+
+    aarch64)
+        ./scripts/config --enable XEN_NETDEV_BACKEND
+        ;;
 esac
 
 make olddefconfig
@@ -41,4 +46,9 @@ case $UNAME in
         make -j$(nproc) bzImage
         cp arch/x86/boot/bzImage "${COPYDIR}"
         ;;
+
+    aarch64)
+        make -j$(nproc) Image
+        cp arch/arm64/boot/Image "${COPYDIR}"
+        ;;
 esac
-- 
2.39.5

[PATCH 8/8] Linux 6.6.86 for x86 and arm64

[Andrew Cooper]

It's the latest 6.6 at the time of writing.

Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
---
CC: Roger Pau Monné <roger.pau@citrix.com>
CC: Stefano Stabellini <sstabellini@kernel.org>
CC: Michal Orzel <michal.orzel@amd.com>
CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
CC: Anthony PERARD <anthony.perard@vates.tech>
---
 .gitlab-ci.yml | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 6e38c2f2a108..c1ce3f38e8b8 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -33,6 +33,18 @@ stages:
   variables:
     CONTAINER: alpine:x86_64-build
 
+linux-6.6.86-arm64:
+  extends: .arm64-artifacts
+  script: ./scripts/build-linux.sh
+  variables:
+    LINUX_VERSION: 6.6.86
+
+linux-6.6.86-x86_64:
+  extends: .x86_64-artifacts
+  script: ./scripts/build-linux.sh
+  variables:
+    LINUX_VERSION: 6.6.86
+
 #
 # The jobs below here are legacy and being phased out.
 #
-- 
2.39.5

Re: [PATCH 5/8] Adjust Linux build script to work with other major versions

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1451 bytes --]

On Wed, Apr 09, 2025 at 05:36:59PM +0100, Andrew Cooper wrote:
> Also use 'tar o' to decompress based on file name.

Do you mean 'tar a'? -o is --no-same-owner. Anyway, just 'tar xf'
works too, even in alpine.

> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> CC: Roger Pau Monné <roger.pau@citrix.com>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Michal Orzel <michal.orzel@amd.com>
> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> CC: Anthony PERARD <anthony.perard@vates.tech>
> ---
>  scripts/x86_64-kernel-linux.sh | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/x86_64-kernel-linux.sh b/scripts/x86_64-kernel-linux.sh
> index 2a816f0b3204..5a0160655bea 100755
> --- a/scripts/x86_64-kernel-linux.sh
> +++ b/scripts/x86_64-kernel-linux.sh
> @@ -11,9 +11,10 @@ WORKDIR="${PWD}"
>  COPYDIR="${WORKDIR}/binaries/"
>  
>  # Build Linux
> +MAJOR=${LINUX_VERSION%%.*}
>  curl -fsSLO \
> -    https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-"${LINUX_VERSION}".tar.xz
> -tar xJf linux-"${LINUX_VERSION}".tar.xz
> +    https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
> +tar oxf linux-"${LINUX_VERSION}".tar.xz
>  cd linux-"${LINUX_VERSION}"
>  make ARCH=x86 defconfig
>  make ARCH=x86 xen.config
> -- 
> 2.39.5
> 

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH TEST-ARTEFACTS 0/8] Cleanup and Linux ARM64 support

[Andrew Cooper]

On 09/04/2025 5:36 pm, Andrew Cooper wrote:
> Various bits of cleanup, and support for arm64 Linux builds.
>
> Run using the new Linux 6.6.86 on (most) x86, and ARM64:
>   https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1760667411

Lovely, Linux 6.6.86 is broken for x86 PVH.  It triple faults very early on.

Sample log:
https://gitlab.com/xen-project/hardware/xen-staging/-/jobs/9673797450

I guess we'll have to stay on 6.6.56 for now.  (Only affects the final
patch.)

~Andrew

Re: [PATCH 6/8] Factor our x86-isms in the linux build script

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 2843 bytes --]

On Wed, Apr 09, 2025 at 05:37:00PM +0100, Andrew Cooper wrote:
> ... in preparation to use it for arm64 too.  Rename the script.
> 
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> ---
> CC: Roger Pau Monné <roger.pau@citrix.com>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Michal Orzel <michal.orzel@amd.com>
> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> CC: Anthony PERARD <anthony.perard@vates.tech>
> ---
>  .gitlab-ci.yml                                |  3 +-
>  ...{x86_64-kernel-linux.sh => build-linux.sh} | 34 +++++++++++++------
>  2 files changed, 24 insertions(+), 13 deletions(-)
>  rename scripts/{x86_64-kernel-linux.sh => build-linux.sh} (55%)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 5a0a853e551d..ff8dce7be05d 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -31,8 +31,7 @@ stages:
>  #
>  x86_64-kernel-linux-6.6.56:
>    extends: .x86_64-artifacts
> -  script:
> -    - . scripts/x86_64-kernel-linux.sh
> +  script: ./scripts/build-linux.sh
>    variables:
>      LINUX_VERSION: 6.6.56
>  
> diff --git a/scripts/x86_64-kernel-linux.sh b/scripts/build-linux.sh
> similarity index 55%
> rename from scripts/x86_64-kernel-linux.sh
> rename to scripts/build-linux.sh
> index 5a0160655bea..5e25e958c0f3 100755
> --- a/scripts/x86_64-kernel-linux.sh
> +++ b/scripts/build-linux.sh
> @@ -9,6 +9,7 @@ set -ex -o pipefail
>  
>  WORKDIR="${PWD}"
>  COPYDIR="${WORKDIR}/binaries/"
> +UNAME=$(uname -m)
>  
>  # Build Linux
>  MAJOR=${LINUX_VERSION%%.*}
> @@ -16,17 +17,28 @@ curl -fsSLO \
>      https://cdn.kernel.org/pub/linux/kernel/v"${MAJOR}".x/linux-"${LINUX_VERSION}".tar.xz
>  tar oxf linux-"${LINUX_VERSION}".tar.xz
>  cd linux-"${LINUX_VERSION}"
> -make ARCH=x86 defconfig
> -make ARCH=x86 xen.config
> +
> +make defconfig
>  ./scripts/config --enable BRIDGE
>  ./scripts/config --enable IGC
>  ./scripts/config --enable TUN
> -cp .config .config.orig
> -cat .config.orig \
> -    | grep 'XEN' \
> -    | grep '=m' \
> -    | sed 's/=m/=y/g' \
> -    >> .config
> -make ARCH=x86 olddefconfig
> -make -s -j "$(nproc)" ARCH=x86
> -cp arch/x86/boot/bzImage "${COPYDIR}"
> +
> +case $UNAME in
> +    x86_64)
> +        make xen.config
> +        cp .config .config.orig
> +        cat .config.orig \
> +            | grep 'XEN' \
> +            | grep '=m' \
> +            | sed 's/=m/=y/g' >> .config
> +        ;;
> +esac
> +
> +make olddefconfig
> +
> +case $UNAME in
> +    x86_64)
> +        make -j$(nproc) bzImage
> +        cp arch/x86/boot/bzImage "${COPYDIR}"
> +        ;;
> +esac
> -- 
> 2.39.5
> 

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 7/8] Infrastructure for arm64 linux builds

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 1478 bytes --]

On Wed, Apr 09, 2025 at 05:37:01PM +0100, Andrew Cooper wrote:
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> CC: Roger Pau Monné <roger.pau@citrix.com>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Michal Orzel <michal.orzel@amd.com>
> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> CC: Anthony PERARD <anthony.perard@vates.tech>
> ---
>  .gitlab-ci.yml                            |  7 ++++++
>  containerize                              |  1 +
>  images/alpine/3.18-arm64-build.dockerfile | 27 +++++++++++++++++++++++
>  scripts/build-linux.sh                    | 10 +++++++++
>  4 files changed, 45 insertions(+)
>  create mode 100644 images/alpine/3.18-arm64-build.dockerfile
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index ff8dce7be05d..6e38c2f2a108 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -19,6 +19,13 @@ stages:
>      exclude:
>        - binaries/.gitignore
>  
> +.arm64-artifacts:
> +  extends: .artifacts
> +  tags:
> +    - arm64
> +  variables:
> +    CONTAINER: alpine:3.18-arm64-build

Arm64 kernel used to be built in a Bookworm container. The build in alpine
has "find: unrecognized: -printf" in the middle of the build. It doesn't
fail outright, but something might be broken. I guess it's related to
initramfs - maybe some option can be disabled, to avoid this message?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 4/8] Clean up Gitlab yaml

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 2036 bytes --]

On Wed, Apr 09, 2025 at 05:36:58PM +0100, Andrew Cooper wrote:
> Factor out the registry into a common location.  Fix the worflow name.  List
> all the stages.  Set a default expiry of 1 month.

This is okay, since last artifacts for a branch are preserved even if
expired.

> Note all the current jobs as legacy.  Their naming scheme needs changing, and
> we'll use this opportunity to switch formats too.  However, the artefacts need
> to stay using the old name until the final staging-* branch using them is
> phased out.

This could use a comment what should the new naming be.

> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> CC: Roger Pau Monné <roger.pau@citrix.com>
> CC: Stefano Stabellini <sstabellini@kernel.org>
> CC: Michal Orzel <michal.orzel@amd.com>
> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> CC: Anthony PERARD <anthony.perard@vates.tech>
> ---
>  .gitlab-ci.yml | 14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 36ec6a7e1ee5..5a0a853e551d 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,12 +1,19 @@
> +variables:
> +  REGISTRY: registry.gitlab.com/xen-project/hardware/test-artifacts
> +
>  workflow:
> -  name: "xen test artifacts"
> +  name: "Xen test artifacts"
> +
> +stages:
> +  - build
>  
>  .artifacts:
>    stage: build
>    image:
> -    name: registry.gitlab.com/xen-project/hardware/test-artifacts/${CONTAINER}
> +    name: ${REGISTRY}/${CONTAINER}
>    artifacts:
>      name: "${CI_JOB_NAME_SLUG}"
> +    expire_in: 1 month
>      paths:
>        - binaries/
>      exclude:
> @@ -19,6 +26,9 @@ workflow:
>    variables:
>      CONTAINER: alpine:x86_64-build
>  
> +#
> +# The jobs below here are legacy and being phased out.
> +#
>  x86_64-kernel-linux-6.6.56:
>    extends: .x86_64-artifacts
>    script:
> -- 
> 2.39.5
> 

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 7/8] Infrastructure for arm64 linux builds

[Andrew Cooper]

On 09/04/2025 6:11 pm, Marek Marczykowski-Górecki wrote:
> On Wed, Apr 09, 2025 at 05:37:01PM +0100, Andrew Cooper wrote:
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
>> ---
>> CC: Roger Pau Monné <roger.pau@citrix.com>
>> CC: Stefano Stabellini <sstabellini@kernel.org>
>> CC: Michal Orzel <michal.orzel@amd.com>
>> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
>> CC: Anthony PERARD <anthony.perard@vates.tech>
>> ---
>>  .gitlab-ci.yml                            |  7 ++++++
>>  containerize                              |  1 +
>>  images/alpine/3.18-arm64-build.dockerfile | 27 +++++++++++++++++++++++
>>  scripts/build-linux.sh                    | 10 +++++++++
>>  4 files changed, 45 insertions(+)
>>  create mode 100644 images/alpine/3.18-arm64-build.dockerfile
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index ff8dce7be05d..6e38c2f2a108 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -19,6 +19,13 @@ stages:
>>      exclude:
>>        - binaries/.gitignore
>>  
>> +.arm64-artifacts:
>> +  extends: .artifacts
>> +  tags:
>> +    - arm64
>> +  variables:
>> +    CONTAINER: alpine:3.18-arm64-build
> Arm64 kernel used to be built in a Bookworm container.

So did x86 before they were moved across.

>  The build in alpine
> has "find: unrecognized: -printf" in the middle of the build. It doesn't
> fail outright, but something might be broken. I guess it's related to
> initramfs - maybe some option can be disabled, to avoid this message?

Nothing seems to break...

There's also a bad awk regex.  Alpine seem to fix this by using mawk,
except they then have to patch the Linux build system to take the
override properly.

I can't see anything obvious they do to fix this -printf warning.

~Andrew

Re: [PATCH 5/8] Adjust Linux build script to work with other major versions

[Andrew Cooper]

On 09/04/2025 5:57 pm, Marek Marczykowski-Górecki wrote:
> On Wed, Apr 09, 2025 at 05:36:59PM +0100, Andrew Cooper wrote:
>> Also use 'tar o' to decompress based on file name.
> Do you mean 'tar a'? -o is --no-same-owner. Anyway, just 'tar xf'
> works too, even in alpine.

Hmm...  Muscle memory from long enough ago that I don't recall why.

Lets go with just xf then.

~Andrew

Re: [PATCH 6/8] Factor our x86-isms in the linux build script

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 512 bytes --]

On Wed, Apr 09, 2025 at 07:05:21PM +0200, Marek Marczykowski-Górecki wrote:
> On Wed, Apr 09, 2025 at 05:37:00PM +0100, Andrew Cooper wrote:
> > ... in preparation to use it for arm64 too.  Rename the script.
> > 
> > Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> 
> Reviewed-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>

Replaying without dockerfile in context, so it actually hits the list...

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 6/8] Factor our x86-isms in the linux build script

[Andrew Cooper]

On 09/04/2025 11:59 pm, Marek Marczykowski-Górecki wrote:
> On Wed, Apr 09, 2025 at 07:05:21PM +0200, Marek Marczykowski-Górecki wrote:
>> On Wed, Apr 09, 2025 at 05:37:00PM +0100, Andrew Cooper wrote:
>>> ... in preparation to use it for arm64 too.  Rename the script.
>>>
>>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
>> Reviewed-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
> Replaying without dockerfile in context, so it actually hits the list...

It did hit the list.  (I'm paying attention via my shadow subscription)

Lore seems happy too. 
https://lore.kernel.org/xen-devel/828cabc9-512f-4c2f-a7d7-b9c8963c1300@citrix.com/T/#mf8fba1fa1372985c108752f5179e3f1d7a688323

At a guess, Inumbo have reverted whatever was causing the problems with
dockerfiles in emails.

~Andrew

Re: [PATCH TEST-ARTEFACTS 0/8] Cleanup and Linux ARM64 support

[Jason Andryuk]

On 2025-04-09 13:01, Andrew Cooper wrote:
> On 09/04/2025 5:36 pm, Andrew Cooper wrote:
>> Various bits of cleanup, and support for arm64 Linux builds.
>>
>> Run using the new Linux 6.6.86 on (most) x86, and ARM64:
>>    https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1760667411
> 
> Lovely, Linux 6.6.86 is broken for x86 PVH.  It triple faults very early on.
> 
> Sample log:
> https://gitlab.com/xen-project/hardware/xen-staging/-/jobs/9673797450
> 
> I guess we'll have to stay on 6.6.56 for now.  (Only affects the final
> patch.)

This is an AMD system:

(XEN) [    2.577549] d0v0 Triple fault - invoking HVM shutdown action 1
(XEN) [    2.577557] RIP:    0008:[<0000000001f851d4>]

The instruction:
ffffffff81f851d4:       0f 01 c1                vmcall

vmcall is the Intel instruction, and vmmcall is the AMD one, so CPU 
detection is malfunctioning.

(Early PVH is running identity mapped, so it's offset from ffffffff80000000)

There are no debug symbols in the vmlinux I extracted from the bzImage 
from gitlab, but I can repro locally with on 6.6.86.  It's unclear to me 
why it's failing.

Trying:
diff --git i/arch/x86/xen/enlighten.c w/arch/x86/xen/enlighten.c
index 0219f1c90202..fb4ad7fe3e34 100644
--- i/arch/x86/xen/enlighten.c
+++ w/arch/x86/xen/enlighten.c
@@ -123,11 +123,10 @@ noinstr void *__xen_hypercall_setfunc(void)
         if (!boot_cpu_has(X86_FEATURE_CPUID))
                 xen_get_vendor();

-       if ((boot_cpu_data.x86_vendor == X86_VENDOR_AMD ||
-            boot_cpu_data.x86_vendor == X86_VENDOR_HYGON))
-               func = xen_hypercall_amd;
-       else
+       if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL )
                 func = xen_hypercall_intel;
+       else
+               func = xen_hypercall_amd;

         static_call_update_early(xen_hypercall, func);

But it still calls xen_hypercall_intel().  So maybe x86_vendor isn't 
getting set and ends up as 0 (X86_VENDOR_INTEL)?

That's as far as I got here.

Different but related, on mainline master, I also get a fail in vmcall. 
There, I see in the disassembly that __xen_hypercall_setfunc()'s calls 
to xen_get_vendor() is gone.  xen_get_vendor() seems to have been 
DCE-ed.  There is some new code that hardcodes features - 
"x86/cpufeatures: Add {REQUIRED,DISABLED} feature configs" - which may 
be responsible.

Regards,
Jason

Re: [PATCH TEST-ARTEFACTS 0/8] Cleanup and Linux ARM64 support

[Andrew Cooper]

On 10/04/2025 1:09 am, Jason Andryuk wrote:
> On 2025-04-09 13:01, Andrew Cooper wrote:
>> On 09/04/2025 5:36 pm, Andrew Cooper wrote:
>>> Various bits of cleanup, and support for arm64 Linux builds.
>>>
>>> Run using the new Linux 6.6.86 on (most) x86, and ARM64:
>>>   
>>> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1760667411
>>
>> Lovely, Linux 6.6.86 is broken for x86 PVH.  It triple faults very
>> early on.
>>
>> Sample log:
>> https://gitlab.com/xen-project/hardware/xen-staging/-/jobs/9673797450
>>
>> I guess we'll have to stay on 6.6.56 for now.  (Only affects the final
>> patch.)
>
> This is an AMD system:
>
> (XEN) [    2.577549] d0v0 Triple fault - invoking HVM shutdown action 1
> (XEN) [    2.577557] RIP:    0008:[<0000000001f851d4>]
>
> The instruction:
> ffffffff81f851d4:       0f 01 c1                vmcall
>
> vmcall is the Intel instruction, and vmmcall is the AMD one, so CPU
> detection is malfunctioning.
>
> (Early PVH is running identity mapped, so it's offset from
> ffffffff80000000)
>
> There are no debug symbols in the vmlinux I extracted from the bzImage
> from gitlab, but I can repro locally with on 6.6.86.  It's unclear to
> me why it's failing.
>
> Trying:
> diff --git i/arch/x86/xen/enlighten.c w/arch/x86/xen/enlighten.c
> index 0219f1c90202..fb4ad7fe3e34 100644
> --- i/arch/x86/xen/enlighten.c
> +++ w/arch/x86/xen/enlighten.c
> @@ -123,11 +123,10 @@ noinstr void *__xen_hypercall_setfunc(void)
>         if (!boot_cpu_has(X86_FEATURE_CPUID))
>                 xen_get_vendor();
>
> -       if ((boot_cpu_data.x86_vendor == X86_VENDOR_AMD ||
> -            boot_cpu_data.x86_vendor == X86_VENDOR_HYGON))
> -               func = xen_hypercall_amd;
> -       else
> +       if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL )
>                 func = xen_hypercall_intel;
> +       else
> +               func = xen_hypercall_amd;
>
>         static_call_update_early(xen_hypercall, func);
>
> But it still calls xen_hypercall_intel().  So maybe x86_vendor isn't
> getting set and ends up as 0 (X86_VENDOR_INTEL)?
>
> That's as far as I got here.
>
> Different but related, on mainline master, I also get a fail in
> vmcall. There, I see in the disassembly that
> __xen_hypercall_setfunc()'s calls to xen_get_vendor() is gone. 
> xen_get_vendor() seems to have been DCE-ed.  There is some new code
> that hardcodes features - "x86/cpufeatures: Add {REQUIRED,DISABLED}
> feature configs" - which may be responsible.

6.6.74 is broken too.  (That's the revision that the ARM tests want). 
So it broke somewhere between .56 and .74 which narrows the bisect a little.

https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1761323774

In Gitlab, both AMD and Intel are failing in roughly the same way.

~Andrew

Re: [PATCH TEST-ARTEFACTS 0/8] Cleanup and Linux ARM64 support

[Jürgen Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 2868 bytes --]

On 10.04.25 02:09, Jason Andryuk wrote:
> On 2025-04-09 13:01, Andrew Cooper wrote:
>> On 09/04/2025 5:36 pm, Andrew Cooper wrote:
>>> Various bits of cleanup, and support for arm64 Linux builds.
>>>
>>> Run using the new Linux 6.6.86 on (most) x86, and ARM64:
>>>    https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1760667411
>>
>> Lovely, Linux 6.6.86 is broken for x86 PVH.  It triple faults very early on.
>>
>> Sample log:
>> https://gitlab.com/xen-project/hardware/xen-staging/-/jobs/9673797450
>>
>> I guess we'll have to stay on 6.6.56 for now.  (Only affects the final
>> patch.)
> 
> This is an AMD system:
> 
> (XEN) [    2.577549] d0v0 Triple fault - invoking HVM shutdown action 1
> (XEN) [    2.577557] RIP:    0008:[<0000000001f851d4>]
> 
> The instruction:
> ffffffff81f851d4:       0f 01 c1                vmcall
> 
> vmcall is the Intel instruction, and vmmcall is the AMD one, so CPU detection is 
> malfunctioning.
> 
> (Early PVH is running identity mapped, so it's offset from ffffffff80000000)
> 
> There are no debug symbols in the vmlinux I extracted from the bzImage from 
> gitlab, but I can repro locally with on 6.6.86.  It's unclear to me why it's 
> failing.
> 
> Trying:
> diff --git i/arch/x86/xen/enlighten.c w/arch/x86/xen/enlighten.c
> index 0219f1c90202..fb4ad7fe3e34 100644
> --- i/arch/x86/xen/enlighten.c
> +++ w/arch/x86/xen/enlighten.c
> @@ -123,11 +123,10 @@ noinstr void *__xen_hypercall_setfunc(void)
>          if (!boot_cpu_has(X86_FEATURE_CPUID))
>                  xen_get_vendor();
> 
> -       if ((boot_cpu_data.x86_vendor == X86_VENDOR_AMD ||
> -            boot_cpu_data.x86_vendor == X86_VENDOR_HYGON))
> -               func = xen_hypercall_amd;
> -       else
> +       if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL )
>                  func = xen_hypercall_intel;
> +       else
> +               func = xen_hypercall_amd;
> 
>          static_call_update_early(xen_hypercall, func);
> 
> But it still calls xen_hypercall_intel().  So maybe x86_vendor isn't getting set 
> and ends up as 0 (X86_VENDOR_INTEL)?
> 
> That's as far as I got here.
> 
> Different but related, on mainline master, I also get a fail in vmcall. There, I 
> see in the disassembly that __xen_hypercall_setfunc()'s calls to 
> xen_get_vendor() is gone.  xen_get_vendor() seems to have been DCE-ed.  There is 
> some new code that hardcodes features - "x86/cpufeatures: Add 
> {REQUIRED,DISABLED} feature configs" - which may be responsible.

The test for !X86_FEATURE_CPUID will probably never be true now.

I guess the most simple fix will be to just call xen_get_vendor()
unconditionally.


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3743 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [PATCH 2/8] Port containerise

[Anthony PERARD]

On Wed, Apr 09, 2025 at 05:36:56PM +0100, Andrew Cooper wrote:
> +BASE="registry.gitlab.com/xen-project/hardware/test-artifacts"
> +case "_${CONTAINER}" in
> +    _alpine-x86_64-base) CONTAINER="${BASE}/alpine:x86_64-base" ;;

There's no "images/alpine/x86_64-base.dockerfile" in the repo, do you
mean "alpine:x86_64-rootfs" instead?

Cheers,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 2/8] Port containerise

[Andrew Cooper]

On 10/04/2025 10:33 am, Anthony PERARD wrote:
> On Wed, Apr 09, 2025 at 05:36:56PM +0100, Andrew Cooper wrote:
>> +BASE="registry.gitlab.com/xen-project/hardware/test-artifacts"
>> +case "_${CONTAINER}" in
>> +    _alpine-x86_64-base) CONTAINER="${BASE}/alpine:x86_64-base" ;;
> There's no "images/alpine/x86_64-base.dockerfile" in the repo, do you
> mean "alpine:x86_64-rootfs" instead?

Yes and no.  I'm renaming the containers, but haven't posted that part
of the series yet.

This wants to go back to rootfs in the short term.

~Andrew

Re: [PATCH 4/8] Clean up Gitlab yaml

[Andrew Cooper]

On 09/04/2025 6:15 pm, Marek Marczykowski-Górecki wrote:
> On Wed, Apr 09, 2025 at 05:36:58PM +0100, Andrew Cooper wrote:
>> Factor out the registry into a common location.  Fix the worflow name.  List
>> all the stages.  Set a default expiry of 1 month.
> This is okay, since last artifacts for a branch are preserved even if
> expired.
>
>> Note all the current jobs as legacy.  Their naming scheme needs changing, and
>> we'll use this opportunity to switch formats too.  However, the artefacts need
>> to stay using the old name until the final staging-* branch using them is
>> phased out.
> This could use a comment what should the new naming be.

Same as we use in Xen.  linux-$VER-$ARCH, as shown later in the series,
and alpine-$VER-$ARCH-rootfs (currently there's no $VER, and this is
going to break horribly when moving off 3.18).

~Andrew

Re: [PATCH 3/8] Fix container user setup

[Anthony PERARD]

On Wed, Apr 09, 2025 at 05:36:57PM +0100, Andrew Cooper wrote:
> Use USER (and 'user' as normal), rather than specifying configuration from the
> outside.  This also fixes the fact that root-needing containers need to
> override the default.

This patch also removes one "docker:platform" which is redundant with
the containers image name.

> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>

Reviewed-by: Anthony PERARD <anthony.perard@vates.tech>

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 7/8] Infrastructure for arm64 linux builds

[Andrew Cooper]

On 09/04/2025 5:37 pm, Andrew Cooper wrote:
> diff --git a/images/alpine/3.18-arm64-build.dockerfile b/images/alpine/3.18-arm64-build.dockerfile
> new file mode 100644
> index 000000000000..c76ac435e8dd
> --- /dev/null
> +++ b/images/alpine/3.18-arm64-build.dockerfile
> @@ -0,0 +1,27 @@
> +# syntax=docker/dockerfile:1
> +FROM --platform=linux/arm64/v8 alpine:3.18
> +LABEL maintainer.name="The Xen Project" \
> +      maintainer.email="xen-devel@lists.xenproject.org"
> +
> +RUN apk --no-cache add bash
> +
> +RUN bash -ex <<EOF
> +      adduser -D user --shell bash
> +
> +      DEPS=(# Base environment
> +            musl-dev
> +            build-base
> +            curl

It turns out we don't need musl-dev.  build-base depends on libc-dev
which musl-dev provides.  This was a bad dependency brought across from
the form we have in xen.git right now.

~Andrew

Re: [PATCH 4/8] Clean up Gitlab yaml

[Anthony PERARD]

On Wed, Apr 09, 2025 at 05:36:58PM +0100, Andrew Cooper wrote:
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 36ec6a7e1ee5..5a0a853e551d 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,12 +1,19 @@
> +variables:
> +  REGISTRY: registry.gitlab.com/xen-project/hardware/test-artifacts

Did you consider naming that variable XEN_REGISTRY like in the xen.git
repo? REGISTRY is a fine name too.

> +
>  workflow:
> -  name: "xen test artifacts"
> +  name: "Xen test artifacts"

Isn't this a very useless workflow:name? Can I suggest to remove it
instead? I mean, currently, all pipeline appear to do the same thing:

    https://gitlab.com/xen-project/hardware/test-artifacts/-/pipelines

Without "workflow:name" set, the default name will be the subject of the
top commit.

Thanks,

-- 

Anthony Perard | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 4/8] Clean up Gitlab yaml

[Andrew Cooper]

On 10/04/2025 11:07 am, Anthony PERARD wrote:
> On Wed, Apr 09, 2025 at 05:36:58PM +0100, Andrew Cooper wrote:
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index 36ec6a7e1ee5..5a0a853e551d 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -1,12 +1,19 @@
>> +variables:
>> +  REGISTRY: registry.gitlab.com/xen-project/hardware/test-artifacts
> Did you consider naming that variable XEN_REGISTRY like in the xen.git
> repo? REGISTRY is a fine name too.

I did, except the makefile for rebuilding takes REGISTRY, so this
removes a REGISTERY=$XEN_REGISTRY from the container rebuild logic.

Which is another thing I haven't posted, because it turns out our
docker-in-docker only works for x86, not for ARM.

>
>> +
>>  workflow:
>> -  name: "xen test artifacts"
>> +  name: "Xen test artifacts"
> Isn't this a very useless workflow:name? Can I suggest to remove it
> instead? I mean, currently, all pipeline appear to do the same thing:
>
>     https://gitlab.com/xen-project/hardware/test-artifacts/-/pipelines
>
> Without "workflow:name" set, the default name will be the subject of the
> top commit.

I did wonder about the utility of it.  I'll drop.

~Andrew

Re: [PATCH 7/8] Infrastructure for arm64 linux builds

[Andrew Cooper]

On 09/04/2025 6:15 pm, Andrew Cooper wrote:
> On 09/04/2025 6:11 pm, Marek Marczykowski-Górecki wrote:
>> On Wed, Apr 09, 2025 at 05:37:01PM +0100, Andrew Cooper wrote:
>>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
>>> ---
>>> CC: Roger Pau Monné <roger.pau@citrix.com>
>>> CC: Stefano Stabellini <sstabellini@kernel.org>
>>> CC: Michal Orzel <michal.orzel@amd.com>
>>> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
>>> CC: Anthony PERARD <anthony.perard@vates.tech>
>>> ---
>>>  .gitlab-ci.yml                            |  7 ++++++
>>>  containerize                              |  1 +
>>>  images/alpine/3.18-arm64-build.dockerfile | 27 +++++++++++++++++++++++
>>>  scripts/build-linux.sh                    | 10 +++++++++
>>>  4 files changed, 45 insertions(+)
>>>  create mode 100644 images/alpine/3.18-arm64-build.dockerfile
>>>
>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>> index ff8dce7be05d..6e38c2f2a108 100644
>>> --- a/.gitlab-ci.yml
>>> +++ b/.gitlab-ci.yml
>>> @@ -19,6 +19,13 @@ stages:
>>>      exclude:
>>>        - binaries/.gitignore
>>>  
>>> +.arm64-artifacts:
>>> +  extends: .artifacts
>>> +  tags:
>>> +    - arm64
>>> +  variables:
>>> +    CONTAINER: alpine:3.18-arm64-build
>> Arm64 kernel used to be built in a Bookworm container.
> So did x86 before they were moved across.
>
>>  The build in alpine
>> has "find: unrecognized: -printf" in the middle of the build. It doesn't
>> fail outright, but something might be broken. I guess it's related to
>> initramfs - maybe some option can be disabled, to avoid this message?
> Nothing seems to break...
>
> There's also a bad awk regex.  Alpine seem to fix this by using mawk,
> except they then have to patch the Linux build system to take the
> override properly.
>
> I can't see anything obvious they do to fix this -printf warning.

Yes I can.  We want findutils too.

x86 also wants diffutils too. (as I'm fixing that side of things too)

~Andrew

Re: [PATCH 7/8] Infrastructure for arm64 linux builds

[Andrew Cooper]

On 10/04/2025 11:41 am, Andrew Cooper wrote:
> On 09/04/2025 6:15 pm, Andrew Cooper wrote:
>> On 09/04/2025 6:11 pm, Marek Marczykowski-Górecki wrote:
>>> On Wed, Apr 09, 2025 at 05:37:01PM +0100, Andrew Cooper wrote:
>>>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
>>>> ---
>>>> CC: Roger Pau Monné <roger.pau@citrix.com>
>>>> CC: Stefano Stabellini <sstabellini@kernel.org>
>>>> CC: Michal Orzel <michal.orzel@amd.com>
>>>> CC: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
>>>> CC: Anthony PERARD <anthony.perard@vates.tech>
>>>> ---
>>>>  .gitlab-ci.yml                            |  7 ++++++
>>>>  containerize                              |  1 +
>>>>  images/alpine/3.18-arm64-build.dockerfile | 27 +++++++++++++++++++++++
>>>>  scripts/build-linux.sh                    | 10 +++++++++
>>>>  4 files changed, 45 insertions(+)
>>>>  create mode 100644 images/alpine/3.18-arm64-build.dockerfile
>>>>
>>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>>> index ff8dce7be05d..6e38c2f2a108 100644
>>>> --- a/.gitlab-ci.yml
>>>> +++ b/.gitlab-ci.yml
>>>> @@ -19,6 +19,13 @@ stages:
>>>>      exclude:
>>>>        - binaries/.gitignore
>>>>  
>>>> +.arm64-artifacts:
>>>> +  extends: .artifacts
>>>> +  tags:
>>>> +    - arm64
>>>> +  variables:
>>>> +    CONTAINER: alpine:3.18-arm64-build
>>> Arm64 kernel used to be built in a Bookworm container.
>> So did x86 before they were moved across.
>>
>>>  The build in alpine
>>> has "find: unrecognized: -printf" in the middle of the build. It doesn't
>>> fail outright, but something might be broken. I guess it's related to
>>> initramfs - maybe some option can be disabled, to avoid this message?
>> Nothing seems to break...
>>
>> There's also a bad awk regex.  Alpine seem to fix this by using mawk,
>> except they then have to patch the Linux build system to take the
>> override properly.
>>
>> I can't see anything obvious they do to fix this -printf warning.
> Yes I can.  We want findutils too.
>
> x86 also wants diffutils too. (as I'm fixing that side of things too)

Excellent.  Both x86 and ARM are clean with these changes:

https://gitlab.com/xen-project/hardware/test-artifacts/-/jobs/9682944850
https://gitlab.com/xen-project/hardware/test-artifacts/-/jobs/9682944867

The awk complaint has gone away too, so I suppose that's a knock-on effect.

~Andrew

Re: [PATCH 7/8] Infrastructure for arm64 linux builds

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

On Thu, Apr 10, 2025 at 11:57:23AM +0100, Andrew Cooper wrote:
> The awk complaint has gone away too, so I suppose that's a knock-on effect.

It isn't? 
https://gitlab.com/xen-project/hardware/test-artifacts/-/jobs/9683454411#L106

Anyway, hopefully harmless.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH TEST-ARTEFACTS 0/8] Cleanup and Linux ARM64 support

[Alejandro Vallejo]

On Thu Apr 10, 2025 at 10:17 AM BST, Andrew Cooper wrote:
> On 10/04/2025 1:09 am, Jason Andryuk wrote:
>> On 2025-04-09 13:01, Andrew Cooper wrote:
>>> On 09/04/2025 5:36 pm, Andrew Cooper wrote:
>>>> Various bits of cleanup, and support for arm64 Linux builds.
>>>>
>>>> Run using the new Linux 6.6.86 on (most) x86, and ARM64:
>>>>   
>>>> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1760667411
>>>
>>> Lovely, Linux 6.6.86 is broken for x86 PVH.  It triple faults very
>>> early on.
>>>
>>> Sample log:
>>> https://gitlab.com/xen-project/hardware/xen-staging/-/jobs/9673797450
>>>
>>> I guess we'll have to stay on 6.6.56 for now.  (Only affects the final
>>> patch.)
>>
>> This is an AMD system:
>>
>> (XEN) [    2.577549] d0v0 Triple fault - invoking HVM shutdown action 1
>> (XEN) [    2.577557] RIP:    0008:[<0000000001f851d4>]
>>
>> The instruction:
>> ffffffff81f851d4:       0f 01 c1                vmcall
>>
>> vmcall is the Intel instruction, and vmmcall is the AMD one, so CPU
>> detection is malfunctioning.
>>
>> (Early PVH is running identity mapped, so it's offset from
>> ffffffff80000000)
>>
>> There are no debug symbols in the vmlinux I extracted from the bzImage
>> from gitlab, but I can repro locally with on 6.6.86.  It's unclear to
>> me why it's failing.
>>
>> Trying:
>> diff --git i/arch/x86/xen/enlighten.c w/arch/x86/xen/enlighten.c
>> index 0219f1c90202..fb4ad7fe3e34 100644
>> --- i/arch/x86/xen/enlighten.c
>> +++ w/arch/x86/xen/enlighten.c
>> @@ -123,11 +123,10 @@ noinstr void *__xen_hypercall_setfunc(void)
>>         if (!boot_cpu_has(X86_FEATURE_CPUID))
>>                 xen_get_vendor();
>>
>> -       if ((boot_cpu_data.x86_vendor == X86_VENDOR_AMD ||
>> -            boot_cpu_data.x86_vendor == X86_VENDOR_HYGON))
>> -               func = xen_hypercall_amd;
>> -       else
>> +       if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL )
>>                 func = xen_hypercall_intel;
>> +       else
>> +               func = xen_hypercall_amd;
>>
>>         static_call_update_early(xen_hypercall, func);
>>
>> But it still calls xen_hypercall_intel().  So maybe x86_vendor isn't
>> getting set and ends up as 0 (X86_VENDOR_INTEL)?
>>
>> That's as far as I got here.
>>
>> Different but related, on mainline master, I also get a fail in
>> vmcall. There, I see in the disassembly that
>> __xen_hypercall_setfunc()'s calls to xen_get_vendor() is gone. 
>> xen_get_vendor() seems to have been DCE-ed.  There is some new code
>> that hardcodes features - "x86/cpufeatures: Add {REQUIRED,DISABLED}
>> feature configs" - which may be responsible.
>
> 6.6.74 is broken too.  (That's the revision that the ARM tests want). 
> So it broke somewhere between .56 and .74 which narrows the bisect a little.
>
> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1761323774
>
> In Gitlab, both AMD and Intel are failing in roughly the same way.
>
> ~Andrew

I've bisected the tags and it was was introduced somewhere between the
v6.6.66 and the v6.6.67 tags.

The hypercall page was removed very shortly before v6.6.67 was tagged,
so I have a nagging suspicion...

Cheers,
Alejandro

Re: [PATCH TEST-ARTEFACTS 0/8] Cleanup and Linux ARM64 support

[Alejandro Vallejo]

On Thu Apr 10, 2025 at 7:20 PM BST, Alejandro Vallejo wrote:
> On Thu Apr 10, 2025 at 10:17 AM BST, Andrew Cooper wrote:
>> On 10/04/2025 1:09 am, Jason Andryuk wrote:
>>> On 2025-04-09 13:01, Andrew Cooper wrote:
>>>> On 09/04/2025 5:36 pm, Andrew Cooper wrote:
>>>>> Various bits of cleanup, and support for arm64 Linux builds.
>>>>>
>>>>> Run using the new Linux 6.6.86 on (most) x86, and ARM64:
>>>>>   
>>>>> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1760667411
>>>>
>>>> Lovely, Linux 6.6.86 is broken for x86 PVH.  It triple faults very
>>>> early on.
>>>>
>>>> Sample log:
>>>> https://gitlab.com/xen-project/hardware/xen-staging/-/jobs/9673797450
>>>>
>>>> I guess we'll have to stay on 6.6.56 for now.  (Only affects the final
>>>> patch.)
>>>
>>> This is an AMD system:
>>>
>>> (XEN) [    2.577549] d0v0 Triple fault - invoking HVM shutdown action 1
>>> (XEN) [    2.577557] RIP:    0008:[<0000000001f851d4>]
>>>
>>> The instruction:
>>> ffffffff81f851d4:       0f 01 c1                vmcall
>>>
>>> vmcall is the Intel instruction, and vmmcall is the AMD one, so CPU
>>> detection is malfunctioning.
>>>
>>> (Early PVH is running identity mapped, so it's offset from
>>> ffffffff80000000)
>>>
>>> There are no debug symbols in the vmlinux I extracted from the bzImage
>>> from gitlab, but I can repro locally with on 6.6.86.  It's unclear to
>>> me why it's failing.
>>>
>>> Trying:
>>> diff --git i/arch/x86/xen/enlighten.c w/arch/x86/xen/enlighten.c
>>> index 0219f1c90202..fb4ad7fe3e34 100644
>>> --- i/arch/x86/xen/enlighten.c
>>> +++ w/arch/x86/xen/enlighten.c
>>> @@ -123,11 +123,10 @@ noinstr void *__xen_hypercall_setfunc(void)
>>>         if (!boot_cpu_has(X86_FEATURE_CPUID))
>>>                 xen_get_vendor();
>>>
>>> -       if ((boot_cpu_data.x86_vendor == X86_VENDOR_AMD ||
>>> -            boot_cpu_data.x86_vendor == X86_VENDOR_HYGON))
>>> -               func = xen_hypercall_amd;
>>> -       else
>>> +       if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL )
>>>                 func = xen_hypercall_intel;
>>> +       else
>>> +               func = xen_hypercall_amd;
>>>
>>>         static_call_update_early(xen_hypercall, func);
>>>
>>> But it still calls xen_hypercall_intel().  So maybe x86_vendor isn't
>>> getting set and ends up as 0 (X86_VENDOR_INTEL)?
>>>
>>> That's as far as I got here.
>>>
>>> Different but related, on mainline master, I also get a fail in
>>> vmcall. There, I see in the disassembly that
>>> __xen_hypercall_setfunc()'s calls to xen_get_vendor() is gone. 
>>> xen_get_vendor() seems to have been DCE-ed.  There is some new code
>>> that hardcodes features - "x86/cpufeatures: Add {REQUIRED,DISABLED}
>>> feature configs" - which may be responsible.
>>
>> 6.6.74 is broken too.  (That's the revision that the ARM tests want). 
>> So it broke somewhere between .56 and .74 which narrows the bisect a little.
>>
>> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1761323774
>>
>> In Gitlab, both AMD and Intel are failing in roughly the same way.
>>
>> ~Andrew
>
> I've bisected the tags and it was was introduced somewhere between the
> v6.6.66 and the v6.6.67 tags.
>
> The hypercall page was removed very shortly before v6.6.67 was tagged,
> so I have a nagging suspicion...
>
> Cheers,
> Alejandro

The cutoff point is bcf0e2fda80c6("x86/xen: remove hypercall page").

Together with Jason's observation it would seem that Linux doesn't guess
the correct instruction (or not early enough) when running as PVH dom0.
On PV it's just "syscall", but on PVH it's a tad more complicated.

Cheers,
Alejandro

[RFC PATCH] x86/xen: Fix PVH dom0 xen_hypercall detection

[Jason Andryuk]

A Xen PVH dom0 on an AMD processor triple faults early in boot on
6.6.86.  CPU detection appears to fail, as the faulting instruction is
vmcall in xen_hypercall_intel() and not vmmcall in xen_hypercall_amd().

Detection fails because __xen_hypercall_setfunc() returns the full
kernel mapped address of xen_hypercall_amd() or xen_hypercall_intel() -
e.g. 0xffffffff815b93f0.  But this is compared against the rip-relative
xen_hypercall_amd(%rip), which when running from identity mapping, is
only 0x015b93f0.

Replace the rip-relative address with just loading the actual address to
restore the proper comparision.

This only seems to affect PVH dom0 boot.  This is probably because the
XENMEM_memory_map hypercall is issued early on from the identity
mappings.  With a domU, the memory map is provided via hvm_start_info
and the hypercall is skipped.  The domU is probably running from the
kernel high mapping when it issues hypercalls.

Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
---
I think this sort of address mismatch would be addresed by
e8fbc0d9cab6 ("x86/pvh: Call C code via the kernel virtual mapping")

That could be backported instead, but it depends on a fair number of
patches.

Not sure on how getting a patch just into 6.6 would work.  This patch
could go into upstream Linux though it's not strictly necessary when the
rip-relative address is a high address.
---
 arch/x86/xen/xen-head.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/xen/xen-head.S b/arch/x86/xen/xen-head.S
index 059f343da76d..71a0eda2da60 100644
--- a/arch/x86/xen/xen-head.S
+++ b/arch/x86/xen/xen-head.S
@@ -117,7 +117,7 @@ SYM_FUNC_START(xen_hypercall_hvm)
 	pop %ebx
 	pop %eax
 #else
-	lea xen_hypercall_amd(%rip), %rcx
+	mov $xen_hypercall_amd, %rcx
 	cmp %rax, %rcx
 #ifdef CONFIG_FRAME_POINTER
 	pop %rax	/* Dummy pop. */
-- 
2.49.0

Re: [PATCH TEST-ARTEFACTS 0/8] Cleanup and Linux ARM64 support

[Jason Andryuk]

[-- Attachment #1: Type: text/plain, Size: 803 bytes --]

On 2025-04-10 05:17, Andrew Cooper wrote:
> On 10/04/2025 1:09 am, Jason Andryuk wrote:
>> On 2025-04-09 13:01, Andrew Cooper wrote:
>>> On 09/04/2025 5:36 pm, Andrew Cooper wrote:
> 
> 6.6.74 is broken too.  (That's the revision that the ARM tests want).
> So it broke somewhere between .56 and .74 which narrows the bisect a little.
> 
> https://gitlab.com/xen-project/hardware/xen-staging/-/pipelines/1761323774
> 
> In Gitlab, both AMD and Intel are failing in roughly the same way.

Something else goes wrong in QEMU even with my patch for the hypercall, 
and Linux eventually crashes.  Lots of unhandled memory read/write in 
0x1bfffe000 - 0x1bfffeff8, which is marked unusable for dom0.  I trimmed 
lots of the consecutive "unhandled memory" from the attached log 
(313KB->22KB)

Regards,
Jason

[-- Attachment #2: qemu-pvh-dom0.log --]
[-- Type: text/x-log, Size: 22409 bytes --]

(XEN) Xen version 4.21-unstable (jandryuk@) (gcc (GCC) 14.2.1 20250110 (Red Hat 14.2.1-7)) debug=y Thu Apr 10 16:12:39 EDT 2025
(XEN) Latest ChangeSet: Mon Apr 7 12:26:17 2025 -0400 git:20df25d109-dirty
(XEN) build-id: 94b94e5392f061915babcc41c26730e9b4f264de
(XEN) Console output is synchronous.
(XEN) CPU Vendor: AMD, Family 15 (0xf), Model 107 (0x6b), Stepping 1 (raw 00060fb1)
(XEN) Microcode loading not available
(XEN) Bootloader: qemu
(XEN) Command line: noreboot console=com1 sync_console dom0_mem=1G dom0=pvh noreboot
(XEN) Xen image load base address: 0
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V1 V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 0 MBR signatures
(XEN)  Found 0 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  [0000000000000000, 000000000009fbff] (usable)
(XEN)  [000000000009fc00, 000000000009ffff] (reserved)
(XEN)  [00000000000f0000, 00000000000fffff] (reserved)
(XEN)  [0000000000100000, 000000007ffdefff] (usable)
(XEN)  [000000007ffdf000, 000000007fffffff] (reserved)
(XEN)  [00000000b0000000, 00000000bfffffff] (reserved)
(XEN)  [00000000fed1c000, 00000000fed1ffff] (reserved)
(XEN)  [00000000fffc0000, 00000000ffffffff] (reserved)
(XEN)  [0000000100000000, 00000001bfffffff] (usable)
(XEN)  [000000fd00000000, 000000ffffffffff] (reserved)
(XEN) New Xen image base address: 0x7f600000
(XEN) ACPI: RSDP 000F52D0, 0014 (r0 BOCHS )
(XEN) ACPI: RSDT 7FFE2432, 003C (r1 BOCHS  BXPC            1 BXPC        1)
(XEN) ACPI: FACP 7FFE216A, 00F4 (r3 BOCHS  BXPC            1 BXPC        1)
(XEN) ACPI: DSDT 7FFE0040, 212A (r1 BOCHS  BXPC            1 BXPC        1)
(XEN) ACPI: FACS 7FFE0000, 0040
(XEN) ACPI: APIC 7FFE225E, 0080 (r3 BOCHS  BXPC            1 BXPC        1)
(XEN) ACPI: HPET 7FFE22DE, 0038 (r1 BOCHS  BXPC            1 BXPC        1)
(XEN) ACPI: MCFG 7FFE2316, 003C (r1 BOCHS  BXPC            1 BXPC        1)
(XEN) ACPI: IVRS 7FFE2352, 00B8 (r1 BOCHS  BXPC            1 BXPC        1)
(XEN) ACPI: WAET 7FFE240A, 0028 (r1 BOCHS  BXPC            1 BXPC        1)
(XEN) System RAM: 5119MB (5242360kB)
(XEN) No NUMA configuration found
(XEN) Faking a node at 0000000000000000-00000001c0000000
(XEN) Domain heap initialised
(XEN) found SMP MP-table at 000f54a0
(XEN) DMI 2.8 present.
(XEN) XSM Framework v1.0.1 initialized
(XEN) Using APIC driver default
(XEN) ACPI: PM-Timer IO Port: 0x608 (24 bits)
(XEN) ACPI: SLEEP INFO: pm1x_cnt[1:604,1:0], pm1x_evt[1:600,1:0]
(XEN) ACPI: 32/64X FACS address mismatch in FADT - 7ffe0000/0000000000000000, using 32
(XEN) ACPI:             wakeup_vec[7ffe000c], vec_size[20]
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ5 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) ACPI: IRQ10 used by override.
(XEN) ACPI: IRQ11 used by override.
(XEN) ACPI: HPET id: 0x8086a201 base: 0xfed00000
(XEN) PCI: MCFG configuration 0: base b0000000 segment 0000 buses 00 - ff
(XEN) PCI: MCFG area at b0000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-ff
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 2 CPUs (0 hotplug CPUs)
(XEN) IRQ limits: 24 GSI, 392 MSI/MSI-X
(XEN) No SSBD controls available
(XEN) CPU0: AMD K8 machine check reporting enabled
(XEN) Speculative mitigation facilities:
(XEN)   Hardware hints:
(XEN)   Hardware features:
(XEN)   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING HARDEN_ARRAY HARDEN_BRANCH HARDEN_GUEST_ACCESS HARDEN_LOCK
(XEN)   Xen settings: BTI-Thunk: RETPOLINE, SPEC_CTRL: No, Other: BRANCH_HARDEN
(XEN)   Support for HVM VMs: RSB
(XEN)   Support for PV VMs: RSB
(XEN)   XPTI (64-bit PV only): Dom0 disabled, DomU disabled (without PCID)
(XEN)   PV L1TF shadowing: Dom0 disabled, DomU disabled
(XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2)
(XEN) Initializing Credit2 scheduler
(XEN)  load_precision_shift: 18
(XEN)  load_window_shift: 30
(XEN)  underload_balance_tolerance: 0
(XEN)  overload_balance_tolerance: -3
(XEN)  runqueues arrangement: socket
(XEN)  cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Platform timer is 100.000MHz HPET
(XEN) Detected 2694.956 MHz processor.
(XEN) Freed 1024kB unused BSS memory
(XEN) alt table ffff82d0404c53f8 -> ffff82d0404d8718
(XEN) AMD-Vi: IOMMU Extended Features:
(XEN) - Prefetch Pages Command
(XEN) - Peripheral Page Service Request
(XEN) - Invalidate All Command
(XEN) - Guest APIC
(XEN) - Hardware Error Registers
(XEN) - Host Address Translation Size: 0x2
(XEN) - Guest Address Translation Size: 0x2
(XEN) - Guest CR3 Root Table Level: 0
(XEN) - Maximum PASID: 0
(XEN) - SMI Filter Register: 0
(XEN) - SMI Filter Register Count: 0
(XEN) - Guest Virtual APIC Modes: 0
(XEN) - Dual PPR Log: 0
(XEN) - Dual Event Log: 0
(XEN) - Device Table Segmentation: 0
(XEN) - Memory Access Routing and Control: 0
(XEN) - VM Table Size: 0
(XEN) AMD-Vi: Disabled HAP memory map sharing with IOMMU
(XEN) AMD-Vi: IOMMU 0 Enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) nr_sockets: 1
(XEN) Enabling APIC mode.  Using 1 I/O APICs
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) TSC only partially writable
(XEN) Wallclock source: CMOS RTC
(XEN) Allocated console ring of 16 KiB.
(XEN) mwait-idle: does not run on family 15 model 107
(XEN) HVM: ASIDs enabled.
(XEN) SVM: Supported advanced features:
(XEN)  - none
(XEN) HVM: SVM enabled
(XEN) HVM: Hardware Assisted Paging (HAP) not detected
(XEN) alt table ffff82d0404c53f8 -> ffff82d0404d8718
(XEN) Brought up 2 CPUs
(XEN) Scheduling granularity: cpu, 1 CPU per sched-resource
(XEN) Initializing Credit2 scheduler
(XEN)  load_precision_shift: 18
(XEN)  load_window_shift: 30
(XEN)  underload_balance_tolerance: 0
(XEN)  overload_balance_tolerance: -3
(XEN)  runqueues arrangement: socket
(XEN)  cap enforcement granularity: 10ms
(XEN) load tracking window length 1073741824 ns
(XEN) Adding cpu 0 to runqueue 0
(XEN)  First cpu on runqueue, activating
(XEN) Adding cpu 1 to runqueue 0
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) mtrr: your CPUs had inconsistent fixed MTRR settings
(XEN) mtrr: your CPUs had inconsistent variable MTRR settings
(XEN) mtrr: your CPUs had inconsistent MTRRdefType settings
(XEN) mtrr: probably your BIOS does not setup all CPUs.
(XEN) mtrr: corrected configuration.
(XEN) MTRR default type: write-back
(XEN) MTRR fixed ranges enabled:
(XEN)   00000-9ffff write-back
(XEN)   a0000-bffff uncachable
(XEN)   c0000-fffff write-protect
(XEN) MTRR variable ranges enabled:
(XEN)   0 base 00c0000000 mask ffc0000000 uncachable
(XEN)   1 disabled
(XEN)   2 disabled
(XEN)   3 disabled
(XEN)   4 disabled
(XEN)   5 disabled
(XEN)   6 disabled
(XEN)   7 disabled
(XEN) Running stub recovery selftests...
(XEN) Fixup #UD[0000]: ffff82d07fffe044 [ffff82d07fffe044] -> ffff82d0403a6ecf
(XEN) Fixup #GP[0000]: ffff82d07fffe045 [ffff82d07fffe045] -> ffff82d0403a6ecf
(XEN) Fixup #GP[0000]: ffff82d07fffe044 [ffff82d07fffe044] -> ffff82d0403a6ecf
(XEN) Selftest 2 failed: Opc f3 0f 1e fa 02 04 04 c3 expected #SS[0000], got #GP[0000]
(XEN) Fixup #BP[0000]: ffff82d07fffe045 [ffff82d07fffe045] -> ffff82d0403a6ecf
(XEN) NX (Execute Disable) protection active
(XEN) d0 has maximum 416 PIRQs
(XEN) *** Building a PVH Dom0 ***
(XEN) Dom0 memory allocation stats:
(XEN) order  0 allocations: 2
(XEN) order  1 allocations: 1
(XEN) order  2 allocations: 1
(XEN) order  3 allocations: 1
(XEN) order  4 allocations: 1
(XEN) order  5 allocations: 1
(XEN) order  6 allocations: 1
(XEN) order  7 allocations: 1
(XEN) order  8 allocations: 1
(XEN) order  9 allocations: 1
(XEN) order 10 allocations: 1
(XEN) order 11 allocations: 1
(XEN) order 12 allocations: 1
(XEN) order 13 allocations: 1
(XEN) order 14 allocations: 1
(XEN) order 15 allocations: 1
(XEN) order 16 allocations: 1
(XEN) order 17 allocations: 1
(XEN) ELF: phdr: paddr=0x1000000 memsz=0x9d5e60
(XEN) ELF: phdr: paddr=0x1a00000 memsz=0x317000
(XEN) ELF: phdr: paddr=0x1d17000 memsz=0x319000
(XEN) ELF: memory: 0x1000000 -> 0x2030000
(XEN) ELF: note: PHYS32_ENTRY = 0x1000000
(XEN) ELF: note: GUEST_OS = "linux"
(XEN) ELF: note: GUEST_VERSION = "2.6"
(XEN) ELF: note: XEN_VERSION = "xen-3.0"
(XEN) ELF: note: SUPPORTED_FEATURES = 0x8800
(XEN) ELF: note: LOADER = "generic"
(XEN) ELF: note: SUSPEND_CANCEL = 0x1
(XEN) ELF: Found PVH image
(XEN) ELF: addresses:
(XEN)     virt_base        = 0x0
(XEN)     elf_paddr_offset = 0x0
(XEN)     virt_offset      = 0x0
(XEN)     virt_kstart      = 0x1000000
(XEN)     virt_kend        = 0x2030000
(XEN)     virt_entry       = 0x1000000
(XEN)     p2m_base         = 0xffffffffffffffff
(XEN) ELF: phdr 0 at 0x1000000 -> 0x19d5e60
(XEN) ELF: phdr 1 at 0x1a00000 -> 0x1d17000
(XEN) ELF: phdr 2 at 0x1d17000 -> 0x1e78423
(XEN) Dom0 memory map:
(XEN)  [0000000000000000, 000000000009efff] (usable)
(XEN)  [000000000009fc00, 000000000009ffff] (reserved)
(XEN)  [00000000000f0000, 00000000000fffff] (reserved)
(XEN)  [0000000000100000, 0000000040060f15] (usable)
(XEN)  [0000000040060f16, 0000000040060f9f] (ACPI data)
(XEN)  [0000000040061000, 000000007ffdefff] (unusable)
(XEN)  [000000007ffdf000, 000000007fffffff] (reserved)
(XEN)  [00000000b0000000, 00000000bfffffff] (reserved)
(XEN)  [00000000fed1c000, 00000000fed1ffff] (reserved)
(XEN)  [00000000fffc0000, 00000000ffffffff] (reserved)
(XEN)  [0000000100000000, 00000001bfffffff] (unusable)
(XEN)  [000000fd00000000, 000000ffffffffff] (reserved)
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) ***************************************************
(XEN) WARNING: CONSOLE OUTPUT IS SYNCHRONOUS
(XEN) This option is intended to aid debugging of Xen by ensuring
(XEN) that all output is synchronously delivered on the serial line.
(XEN) However it can introduce SIGNIFICANT latencies and affect
(XEN) timekeeping. It is NOT recommended for production use!
(XEN) ***************************************************
(XEN) SELFTEST FAILURE: CORRECT BEHAVIOR CANNOT BE GUARANTEED
(XEN) ***************************************************
(XEN) 3... 2... 1... 
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) Freed 676kB init memory
Linux version 6.6.87-00001-ga609dbde943f (jandryuk@fedora) (gcc (GCC) 14.2.1 20250110 (Red Hat 14.2.1-7), GNU ld version 2.43.1-5.fc41) #15 PREEMPT_DYNAMIC Thu Apr 10 14:27:44 EDT 2025
Command line: console=hvc0 earlyprintk=xen,keep debug verbose loglevel=9
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at mm/memblock.c:603 memblock_add_range.isra.0+0x16c/0x360
CPU: 0 PID: 0 Comm: swapper Not tainted 6.6.87-00001-ga609dbde943f #15
RIP: 0010:memblock_add_range.isra.0+0x16c/0x360
Code: 2e 4c 8b 6e 08 49 01 ed 4c 39 fd 73 4f 4d 39 ee 73 45 49 39 ee 0f 82 2c ff ff ff 49 8b 01 eb c0 48 83 f8 01 0f 84 97 01 00 00 <0f> 0b 48 89 37 48 8b 43 18 4c 89 68 08 48 8b 43 18 89 48 10 4c 89
RSP: 0000:ffffffff81a03e38 EFLAGS: 00010002 ORIG_RAX: 0000000000000000
RAX: 0000000000000002 RBX: ffffffff81acd218 RCX: 0000000000000000
RDX: 0000000001000000 RSI: 0000000001000000 RDI: ffffffff81eb4960
RBP: 0000000001000000 R08: 62247223ff787373 R09: ffffffff81eb4960
R10: 0000000000000010 R11: 0000000000000010 R12: ffffffff81a03f00
R13: 0000000001000000 R14: 0000000000000000 R15: 0000000001d6dda0
FS:  0000000000000000(0000) GS:ffffffff81a4f000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff888001a47ff8 CR3: 0000000001d7a000 CR4: 00000000000000a0
Call Trace:
 <TASK>
 ? memblock_reserve+0x46/0x90
 ? setup_arch+0x23b/0xb40
 ? start_kernel+0x51/0x5c0
 ? x86_64_start_reservations+0x18/0x30
 ? x86_64_start_kernel+0xc3/0xd0
 ? secondary_startup_64_no_verify+0x112/0x11b
 </TASK>
---[ end trace 0000000000000000 ]---
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009efff] usable
BIOS-e820: [mem 0x000000000009fc00-0x00000000000fffff] reserved
BIOS-e820: [mem 0x0000000000100000-0x0000000040060f15] usable
BIOS-e820: [mem 0x0000000040060f16-0x0000000040060f9f] ACPI data
BIOS-e820: [mem 0x0000000040061000-0x000000007ffdefff] usable
BIOS-e820: [mem 0x000000007ffdf000-0x000000007fffffff] reserved
BIOS-e820: [mem 0x00000000b0000000-0x00000000bfffffff] reserved
BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1ffff] reserved
BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
BIOS-e820: [mem 0x0000000100000000-0x00000001bfffffff] usable
BIOS-e820: [mem 0x000000fd00000000-0x000000ffffffffff] reserved
printk: console [xenboot0] enabled
NX (Execute Disable) protection: active
APIC: Static calls initialized
SMBIOS 2.8 present.
DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-3.fc41 04/01/2014
Hypervisor detected: Xen HVM
Xen version 4.21.
tsc: Fast TSC calibration failed
tsc: Detected 2694.956 MHz processor
e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
e820: remove [mem 0x000a0000-0x000fffff] usable
last_pfn = 0x1c0000 max_arch_pfn = 0x400000000
MTRR map: 4 entries (3 fixed + 1 variable; max 19), built from 8 variable MTRRs
x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
last_pfn = 0x7ffdf max_arch_pfn = 0x400000000
found SMP MP-table at [mem 0x000f54a0-0x000f54af]
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bffff000 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bffff008 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bffff010 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bffff018 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bffff020 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bffff028 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bffff030 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bffff038 size 8
...
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffefe0 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffefe8 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffeff0 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffeff8 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bfffe000 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bfffe008 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bfffe010 size 8
...
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bfffe2e8 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bfffe2f0 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bfffe2f8 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bffff000 size 8
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at arch/x86/include/asm/pgalloc.h:75 phys_pmd_init+0x333/0x380
CPU: 0 PID: 0 Comm: swapper Tainted: G        W          6.6.87-00001-ga609dbde943f #15
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-3.fc41 04/01/2014
RIP: 0010:phys_pmd_init+0x333/0x380
Code: ff ff 48 85 f6 75 0e 48 c7 c0 00 00 00 80 48 2b 05 8a ba 96 00 48 8b 33 48 01 d0 48 83 c8 67 40 f6 c6 80 74 1a 48 39 c6 74 91 <0f> 0b eb 8d f7 c2 81 01 00 00 0f 84 10 fe ff ff e9 04 fe ff ff f7
RSP: 0000:ffffffff81a03c50 EFLAGS: 00010082 ORIG_RAX: 0000000000000000
RAX: 00000001bfffe067 RBX: ffff8881bffff000 RCX: 0000000040060000
RDX: ffff88823fffe000 RSI: ffffffffffffffff RDI: ffff8881bfffe000
RBP: 0000000040200000 R08: 0000000000000001 R09: 000ffffffffff000
R10: 0000000040060000 R11: 8000000000000163 R12: 0000000040000000
R13: 0000000000000000 R14: ffff8881bffff000 R15: 0000000040060000
FS:  0000000000000000(0000) GS:ffffffff81a4f000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff8881bffff000 CR3: 0000000001d7a000 CR4: 00000000000000b0
Call Trace:
 <TASK>
 ? phys_pud_init+0x28a/0x390
 ? __kernel_physical_mapping_init+0xa9/0x390
 ? init_memory_mapping+0x172/0x420
 ? init_range_memory_mapping+0xd3/0x170
 ? init_mem_mapping+0x232/0x370
 ? setup_arch+0x6f8/0xb40
 ? start_kernel+0x51/0x5c0
 ? x86_64_start_reservations+0x18/0x30
 ? x86_64_start_kernel+0xc3/0xd0
 ? secondary_startup_64_no_verify+0x112/0x11b
 </TASK>
---[ end trace 0000000000000000 ]---
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bffff000 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bffff000 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bffff000 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bffff000 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bffff008 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bffff010 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bffff018 size 8
...
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bfffffd8 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bfffffe0 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bfffffe8 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bffffff0 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bffffff8 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bffffff8 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bffffff8 size 8
ACPI: Early table checksum verification disabled
ACPI: RSDP 0x0000000040060F16 000024 (v02 BOCHS )
ACPI: XSDT 0x0000000040060F3A 00003C (v01 BOCHS  BXPC     00000001 BXPC 00000001)
ACPI: APIC 0x0000000040060F76 00008A (v03 BOCHS  BXPC     00000001 BXPC 00000001)
ACPI: FACP 0x000000007FFE216A 0000F4 (v03 BOCHS  BXPC     00000001 BXPC 00000001)
ACPI: DSDT 0x000000007FFE0040 00212A (v01 BOCHS  BXPC     00000001 BXPC 00000001)
ACPI: FACS 0x000000007FFE0000 000040
ACPI: MCFG 0x000000007FFE2316 00003C (v01 BOCHS  BXPC     00000001 BXPC 00000001)
ACPI: Reserving APIC table memory at [mem 0x40060f76-0x40060fff]
ACPI: Reserving FACP table memory at [mem 0x7ffe216a-0x7ffe225d]
ACPI: Reserving DSDT table memory at [mem 0x7ffe0040-0x7ffe2169]
ACPI: Reserving FACS table memory at [mem 0x7ffe0000-0x7ffe003f]
ACPI: Reserving MCFG table memory at [mem 0x7ffe2316-0x7ffe2351]
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffa000 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffa008 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffa010 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffa018 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffa020 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffa028 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffa030 size 8
...
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffdfe8 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffdff0 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory write to 0x1bfffdff8 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bfffa000 size 8
(XEN) arch/x86/hvm/emulate.c:409:d0v0 unhandled memory read from 0x1bfffa000 size 8
BUG: unable to handle page fault for address: ffffffffffffffff
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 1a47067 P4D 1a47067 PUD 1a49067 PMD 0 
Oops: 0000 [#1] PREEMPT NOPTI
CPU: 0 PID: 0 Comm: swapper Tainted: G        W          6.6.87-00001-ga609dbde943f #15
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-3.fc41 04/01/2014
RIP: 0010:sparse_init+0xf7/0x470
Code: 00 49 39 c7 0f 83 a4 00 00 00 48 8b 05 aa cd 16 00 48 85 c0 74 14 4a 8b 04 f8 48 85 c0 74 0b 41 0f b6 d6 48 c1 e2 04 48 01 d0 <48> 83 38 00 75 20 4c 89 20 4c 39 35 79 cd 16 00 4c 89 e2 73 0a 4c
RSP: 0000:ffffffff81a03e68 EFLAGS: 00010086
RAX: ffffffffffffffff RBX: 0000000000000000 RCX: ffff8881bfffa000
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8881bfffe000
RBP: 0000000000000000 R08: ffffffff81eb4a20 R09: 0000000000000000
R10: ffff8881bfffa000 R11: 00000001bfffe000 R12: 0000000000000004
R13: 000000000000009f R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffffff81a4f000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffffffffff CR3: 0000000001a42000 CR4: 00000000000000b0
Call Trace:
 <TASK>
 ? paging_init+0x9/0x10
 ? setup_arch+0x809/0xb40
 ? start_kernel+0x51/0x5c0
 ? x86_64_start_reservations+0x18/0x30
 ? x86_64_start_kernel+0xc3/0xd0
 ? secondary_startup_64_no_verify+0x112/0x11b
 </TASK>
CR2: ffffffffffffffff
---[ end trace 0000000000000000 ]---
RIP: 0010:sparse_init+0xf7/0x470
Code: 00 49 39 c7 0f 83 a4 00 00 00 48 8b 05 aa cd 16 00 48 85 c0 74 14 4a 8b 04 f8 48 85 c0 74 0b 41 0f b6 d6 48 c1 e2 04 48 01 d0 <48> 83 38 00 75 20 4c 89 20 4c 39 35 79 cd 16 00 4c 89 e2 73 0a 4c
RSP: 0000:ffffffff81a03e68 EFLAGS: 00010086
RAX: ffffffffffffffff RBX: 0000000000000000 RCX: ffff8881bfffa000
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8881bfffe000
RBP: 0000000000000000 R08: ffffffff81eb4a20 R09: 0000000000000000
R10: ffff8881bfffa000 R11: 00000001bfffe000 R12: 0000000000000004
R13: 000000000000009f R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffffff81a4f000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffffffffff CR3: 0000000001a42000 CR4: 00000000000000b0
Kernel panic - not syncing: Attempted to kill the idle task!

Re: [RFC PATCH] x86/xen: Fix PVH dom0 xen_hypercall detection

[Andrew Cooper]

On 10/04/2025 8:50 pm, Jason Andryuk wrote:
> A Xen PVH dom0 on an AMD processor triple faults early in boot on
> 6.6.86.  CPU detection appears to fail, as the faulting instruction is
> vmcall in xen_hypercall_intel() and not vmmcall in xen_hypercall_amd().
>
> Detection fails because __xen_hypercall_setfunc() returns the full
> kernel mapped address of xen_hypercall_amd() or xen_hypercall_intel() -
> e.g. 0xffffffff815b93f0.  But this is compared against the rip-relative
> xen_hypercall_amd(%rip), which when running from identity mapping, is
> only 0x015b93f0.
>
> Replace the rip-relative address with just loading the actual address to
> restore the proper comparision.
>
> This only seems to affect PVH dom0 boot.  This is probably because the
> XENMEM_memory_map hypercall is issued early on from the identity
> mappings.  With a domU, the memory map is provided via hvm_start_info
> and the hypercall is skipped.  The domU is probably running from the
> kernel high mapping when it issues hypercalls.
>
> Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
> ---
> I think this sort of address mismatch would be addresed by
> e8fbc0d9cab6 ("x86/pvh: Call C code via the kernel virtual mapping")
>
> That could be backported instead, but it depends on a fair number of
> patches.

I've just spoken to Ard, and he thinks that it's standalone.  Should be
ok to backport as a fix.

> Not sure on how getting a patch just into 6.6 would work.  This patch
> could go into upstream Linux though it's not strictly necessary when the
> rip-relative address is a high address.

Do we know which other trees are broken?  I only found 6.6 because I was
messing around with other bits of CI that happen to use 6.6.

~Andrew

Re: [PATCH TEST-ARTEFACTS 0/8] Cleanup and Linux ARM64 support

[Jason Andryuk]

On 2025-04-10 17:16, Jason Andryuk wrote:
> On 2025-04-10 05:17, Andrew Cooper wrote:
>> On 10/04/2025 1:09 am, Jason Andryuk wrote:
>>> On 2025-04-09 13:01, Andrew Cooper wrote:
>>>> On 09/04/2025 5:36 pm, Andrew Cooper wrote:
>>
>> 6.6.74 is broken too.  (That's the revision that the ARM tests want).
>> So it broke somewhere between .56 and .74 which narrows the bisect a 
>> little.
>>
>> https://gitlab.com/xen-project/hardware/xen-staging/-/ 
>> pipelines/1761323774
>>
>> In Gitlab, both AMD and Intel are failing in roughly the same way.
> 
> Something else goes wrong in QEMU even with my patch for the hypercall, 
> and Linux eventually crashes.  Lots of unhandled memory read/write in 
> 0x1bfffe000 - 0x1bfffeff8, which is marked unusable for dom0.  I trimmed 
> lots of the consecutive "unhandled memory" from the attached log (313KB- 
>  >22KB)

Seems like Rogers patches need backporting too:

x86/xen: fix memblock_reserve() usage on PVH
x86/xen: move xen_reserve_extra_memory()

Regards,
Jason

Re: [PATCH 4/8] Clean up Gitlab yaml

[Andrew Cooper]

On 11/04/2025 10:35 am, Anthony PERARD wrote:
> On Thu, Apr 10, 2025 at 11:11:49AM +0100, Andrew Cooper wrote:
>> Which is another thing I haven't posted, because it turns out our
>> docker-in-docker only works for x86, not for ARM.
> Fixed! I've installed `qemu-user-static` on the runner
> which run jobs with tag `container-builder`.

Oh excellent.

https://gitlab.com/xen-project/hardware/test-artifacts/-/jobs/9691896141
was happy.

~Andrew

Re: [RFC PATCH] x86/xen: Fix PVH dom0 xen_hypercall detection

[Ard Biesheuvel]

On Thu, 10 Apr 2025 at 23:49, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>
> On 10/04/2025 8:50 pm, Jason Andryuk wrote:
> > A Xen PVH dom0 on an AMD processor triple faults early in boot on
> > 6.6.86.  CPU detection appears to fail, as the faulting instruction is
> > vmcall in xen_hypercall_intel() and not vmmcall in xen_hypercall_amd().
> >
> > Detection fails because __xen_hypercall_setfunc() returns the full
> > kernel mapped address of xen_hypercall_amd() or xen_hypercall_intel() -
> > e.g. 0xffffffff815b93f0.  But this is compared against the rip-relative
> > xen_hypercall_amd(%rip), which when running from identity mapping, is
> > only 0x015b93f0.
> >
> > Replace the rip-relative address with just loading the actual address to
> > restore the proper comparision.
> >
> > This only seems to affect PVH dom0 boot.  This is probably because the
> > XENMEM_memory_map hypercall is issued early on from the identity
> > mappings.  With a domU, the memory map is provided via hvm_start_info
> > and the hypercall is skipped.  The domU is probably running from the
> > kernel high mapping when it issues hypercalls.
> >
> > Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
> > ---
> > I think this sort of address mismatch would be addresed by
> > e8fbc0d9cab6 ("x86/pvh: Call C code via the kernel virtual mapping")
> >
> > That could be backported instead, but it depends on a fair number of
> > patches.
>
> I've just spoken to Ard, and he thinks that it's standalone.  Should be
> ok to backport as a fix.
>

I've tried building and booting 6.6.y with the patch applied - GS will
still be set to the 1:1 mapped address but that shouldn't matter,
given that it is only used for the stack canary, and we don't do
address comparisons on that afaik.

> > Not sure on how getting a patch just into 6.6 would work.  This patch
> > could go into upstream Linux though it's not strictly necessary when the
> > rip-relative address is a high address.
>
> Do we know which other trees are broken?  I only found 6.6 because I was
> messing around with other bits of CI that happen to use 6.6.
>

I'd assume all trees that had the hypercall page removal patch
backported to them will be broken in the same way.

Re: [PATCH TEST-ARTEFACTS 0/8] Cleanup and Linux ARM64 support

[Alejandro Vallejo]

On Thu Apr 10, 2025 at 10:50 PM BST, Jason Andryuk wrote:
> On 2025-04-10 17:16, Jason Andryuk wrote:
>> On 2025-04-10 05:17, Andrew Cooper wrote:
>>> On 10/04/2025 1:09 am, Jason Andryuk wrote:
>>>> On 2025-04-09 13:01, Andrew Cooper wrote:
>>>>> On 09/04/2025 5:36 pm, Andrew Cooper wrote:
>>>
>>> 6.6.74 is broken too.  (That's the revision that the ARM tests want).
>>> So it broke somewhere between .56 and .74 which narrows the bisect a 
>>> little.
>>>
>>> https://gitlab.com/xen-project/hardware/xen-staging/-/ 
>>> pipelines/1761323774
>>>
>>> In Gitlab, both AMD and Intel are failing in roughly the same way.
>> 
>> Something else goes wrong in QEMU even with my patch for the hypercall, 
>> and Linux eventually crashes.  Lots of unhandled memory read/write in 
>> 0x1bfffe000 - 0x1bfffeff8, which is marked unusable for dom0.  I trimmed 
>> lots of the consecutive "unhandled memory" from the attached log (313KB- 
>>  >22KB)
>
> Seems like Rogers patches need backporting too:
>
> x86/xen: fix memblock_reserve() usage on PVH
> x86/xen: move xen_reserve_extra_memory()
>
> Regards,
> Jason

I just tested this with your RFC change + those 2 backports on top of
stable/v6.6.y and Linux does boot afterwards.

Well found.

Cheers,
Alejandro

Re: [RFC PATCH] x86/xen: Fix PVH dom0 xen_hypercall detection

[Alejandro Vallejo]

On Thu Apr 10, 2025 at 8:50 PM BST, Jason Andryuk wrote:
> A Xen PVH dom0 on an AMD processor triple faults early in boot on
> 6.6.86.  CPU detection appears to fail, as the faulting instruction is
> vmcall in xen_hypercall_intel() and not vmmcall in xen_hypercall_amd().
>
> Detection fails because __xen_hypercall_setfunc() returns the full
> kernel mapped address of xen_hypercall_amd() or xen_hypercall_intel() -
> e.g. 0xffffffff815b93f0.  But this is compared against the rip-relative
> xen_hypercall_amd(%rip), which when running from identity mapping, is
> only 0x015b93f0.
>
> Replace the rip-relative address with just loading the actual address to
> restore the proper comparision.
>
> This only seems to affect PVH dom0 boot.  This is probably because the
> XENMEM_memory_map hypercall is issued early on from the identity
> mappings.  With a domU, the memory map is provided via hvm_start_info
> and the hypercall is skipped.  The domU is probably running from the
> kernel high mapping when it issues hypercalls.
>
> Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
> ---
> I think this sort of address mismatch would be addresed by
> e8fbc0d9cab6 ("x86/pvh: Call C code via the kernel virtual mapping")
>
> That could be backported instead, but it depends on a fair number of
> patches.
>
> Not sure on how getting a patch just into 6.6 would work.  This patch
> could go into upstream Linux though it's not strictly necessary when the
> rip-relative address is a high address.
> ---
>  arch/x86/xen/xen-head.S | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/x86/xen/xen-head.S b/arch/x86/xen/xen-head.S
> index 059f343da76d..71a0eda2da60 100644
> --- a/arch/x86/xen/xen-head.S
> +++ b/arch/x86/xen/xen-head.S
> @@ -117,7 +117,7 @@ SYM_FUNC_START(xen_hypercall_hvm)
>  	pop %ebx
>  	pop %eax
>  #else
> -	lea xen_hypercall_amd(%rip), %rcx
> +	mov $xen_hypercall_amd, %rcx

(Now that this is known to be the fix upstream) This probably wants to
be plain lea without RIP-relative addressing, like the x86_32 branch
above?

>  	cmp %rax, %rcx
>  #ifdef CONFIG_FRAME_POINTER
>  	pop %rax	/* Dummy pop. */

Re: [RFC PATCH] x86/xen: Fix PVH dom0 xen_hypercall detection

[Jan Beulich]

On 11.04.2025 14:46, Alejandro Vallejo wrote:
> On Thu Apr 10, 2025 at 8:50 PM BST, Jason Andryuk wrote:
>> A Xen PVH dom0 on an AMD processor triple faults early in boot on
>> 6.6.86.  CPU detection appears to fail, as the faulting instruction is
>> vmcall in xen_hypercall_intel() and not vmmcall in xen_hypercall_amd().
>>
>> Detection fails because __xen_hypercall_setfunc() returns the full
>> kernel mapped address of xen_hypercall_amd() or xen_hypercall_intel() -
>> e.g. 0xffffffff815b93f0.  But this is compared against the rip-relative
>> xen_hypercall_amd(%rip), which when running from identity mapping, is
>> only 0x015b93f0.
>>
>> Replace the rip-relative address with just loading the actual address to
>> restore the proper comparision.
>>
>> This only seems to affect PVH dom0 boot.  This is probably because the
>> XENMEM_memory_map hypercall is issued early on from the identity
>> mappings.  With a domU, the memory map is provided via hvm_start_info
>> and the hypercall is skipped.  The domU is probably running from the
>> kernel high mapping when it issues hypercalls.
>>
>> Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
>> ---
>> I think this sort of address mismatch would be addresed by
>> e8fbc0d9cab6 ("x86/pvh: Call C code via the kernel virtual mapping")
>>
>> That could be backported instead, but it depends on a fair number of
>> patches.
>>
>> Not sure on how getting a patch just into 6.6 would work.  This patch
>> could go into upstream Linux though it's not strictly necessary when the
>> rip-relative address is a high address.
>> ---
>>  arch/x86/xen/xen-head.S | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/arch/x86/xen/xen-head.S b/arch/x86/xen/xen-head.S
>> index 059f343da76d..71a0eda2da60 100644
>> --- a/arch/x86/xen/xen-head.S
>> +++ b/arch/x86/xen/xen-head.S
>> @@ -117,7 +117,7 @@ SYM_FUNC_START(xen_hypercall_hvm)
>>  	pop %ebx
>>  	pop %eax
>>  #else
>> -	lea xen_hypercall_amd(%rip), %rcx
>> +	mov $xen_hypercall_amd, %rcx
> 
> (Now that this is known to be the fix upstream) This probably wants to
> be plain lea without RIP-relative addressing, like the x86_32 branch
> above?

Why would you want to use LEA there? It's functionally identical, but the
MOV can be encoded without ModR/M byte.

Jan

Re: [RFC PATCH] x86/xen: Fix PVH dom0 xen_hypercall detection

[Alejandro Vallejo]

On Fri Apr 11, 2025 at 2:08 PM BST, Jan Beulich wrote:
> On 11.04.2025 14:46, Alejandro Vallejo wrote:
>> On Thu Apr 10, 2025 at 8:50 PM BST, Jason Andryuk wrote:
>>> A Xen PVH dom0 on an AMD processor triple faults early in boot on
>>> 6.6.86.  CPU detection appears to fail, as the faulting instruction is
>>> vmcall in xen_hypercall_intel() and not vmmcall in xen_hypercall_amd().
>>>
>>> Detection fails because __xen_hypercall_setfunc() returns the full
>>> kernel mapped address of xen_hypercall_amd() or xen_hypercall_intel() -
>>> e.g. 0xffffffff815b93f0.  But this is compared against the rip-relative
>>> xen_hypercall_amd(%rip), which when running from identity mapping, is
>>> only 0x015b93f0.
>>>
>>> Replace the rip-relative address with just loading the actual address to
>>> restore the proper comparision.
>>>
>>> This only seems to affect PVH dom0 boot.  This is probably because the
>>> XENMEM_memory_map hypercall is issued early on from the identity
>>> mappings.  With a domU, the memory map is provided via hvm_start_info
>>> and the hypercall is skipped.  The domU is probably running from the
>>> kernel high mapping when it issues hypercalls.
>>>
>>> Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
>>> ---
>>> I think this sort of address mismatch would be addresed by
>>> e8fbc0d9cab6 ("x86/pvh: Call C code via the kernel virtual mapping")
>>>
>>> That could be backported instead, but it depends on a fair number of
>>> patches.
>>>
>>> Not sure on how getting a patch just into 6.6 would work.  This patch
>>> could go into upstream Linux though it's not strictly necessary when the
>>> rip-relative address is a high address.
>>> ---
>>>  arch/x86/xen/xen-head.S | 2 +-
>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/arch/x86/xen/xen-head.S b/arch/x86/xen/xen-head.S
>>> index 059f343da76d..71a0eda2da60 100644
>>> --- a/arch/x86/xen/xen-head.S
>>> +++ b/arch/x86/xen/xen-head.S
>>> @@ -117,7 +117,7 @@ SYM_FUNC_START(xen_hypercall_hvm)
>>>  	pop %ebx
>>>  	pop %eax
>>>  #else
>>> -	lea xen_hypercall_amd(%rip), %rcx
>>> +	mov $xen_hypercall_amd, %rcx
>> 
>> (Now that this is known to be the fix upstream) This probably wants to
>> be plain lea without RIP-relative addressing, like the x86_32 branch
>> above?
>
> Why would you want to use LEA there? It's functionally identical, but the
> MOV can be encoded without ModR/M byte.
>
> Jan

It's not the using of a particular encoding that I meant, but not using
the same on both 32 and 64 bit paths. Surely whatever argument in favour
of either would hold for both 32 and 64 bits.

Cheers,
Alejandro

Re: [RFC PATCH] x86/xen: Fix PVH dom0 xen_hypercall detection

[Jason Andryuk]

On 2025-04-11 07:35, Ard Biesheuvel wrote:
> On Thu, 10 Apr 2025 at 23:49, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>>
>> On 10/04/2025 8:50 pm, Jason Andryuk wrote:
>>> A Xen PVH dom0 on an AMD processor triple faults early in boot on
>>> 6.6.86.  CPU detection appears to fail, as the faulting instruction is
>>> vmcall in xen_hypercall_intel() and not vmmcall in xen_hypercall_amd().
>>>
>>> Detection fails because __xen_hypercall_setfunc() returns the full
>>> kernel mapped address of xen_hypercall_amd() or xen_hypercall_intel() -
>>> e.g. 0xffffffff815b93f0.  But this is compared against the rip-relative
>>> xen_hypercall_amd(%rip), which when running from identity mapping, is
>>> only 0x015b93f0.
>>>
>>> Replace the rip-relative address with just loading the actual address to
>>> restore the proper comparision.
>>>
>>> This only seems to affect PVH dom0 boot.  This is probably because the
>>> XENMEM_memory_map hypercall is issued early on from the identity
>>> mappings.  With a domU, the memory map is provided via hvm_start_info
>>> and the hypercall is skipped.  The domU is probably running from the
>>> kernel high mapping when it issues hypercalls.
>>>
>>> Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
>>> ---
>>> I think this sort of address mismatch would be addresed by
>>> e8fbc0d9cab6 ("x86/pvh: Call C code via the kernel virtual mapping")
>>>
>>> That could be backported instead, but it depends on a fair number of
>>> patches.
>>
>> I've just spoken to Ard, and he thinks that it's standalone.  Should be
>> ok to backport as a fix.
>>
> 
> I've tried building and booting 6.6.y with the patch applied - GS will
> still be set to the 1:1 mapped address but that shouldn't matter,
> given that it is only used for the stack canary, and we don't do
> address comparisons on that afaik.

Yes, it seems to work - I tested with dom0 and it booted.  I removed the 
use of phys_base - the diff is included below.  Does that match what you 
did?

>>> Not sure on how getting a patch just into 6.6 would work.  This patch
>>> could go into upstream Linux though it's not strictly necessary when the
>>> rip-relative address is a high address.
>>
>> Do we know which other trees are broken?  I only found 6.6 because I was
>> messing around with other bits of CI that happen to use 6.6.
>>
> 
> I'd assume all trees that had the hypercall page removal patch
> backported to them will be broken in the same way.

Yes, I think so.  Looks like it went back to 5.10 but not to 5.4.

Ard, I can submit the stable request unless you want to.

Regards,
Jason

diff --git a/arch/x86/platform/pvh/head.S b/arch/x86/platform/pvh/head.S
index c4365a05ab83..9bf4cc04f079 100644
--- a/arch/x86/platform/pvh/head.S
+++ b/arch/x86/platform/pvh/head.S
@@ -100,7 +100,11 @@ SYM_CODE_START_LOCAL(pvh_start_xen)
         xor %edx, %edx
         wrmsr

-       call xen_prepare_pvh
+       /* Call xen_prepare_pvh() via the kernel virtual mapping */
+       leaq xen_prepare_pvh(%rip), %rax
+       addq $__START_KERNEL_map, %rax
+       ANNOTATE_RETPOLINE_SAFE
+       call *%rax

         /* startup_64 expects boot_params in %rsi. */
         mov $_pa(pvh_bootparams), %rsi

Re: [RFC PATCH] x86/xen: Fix PVH dom0 xen_hypercall detection

[Ard Biesheuvel]

On Fri, 11 Apr 2025 at 16:28, Jason Andryuk <jason.andryuk@amd.com> wrote:
>
>
>
> On 2025-04-11 07:35, Ard Biesheuvel wrote:
> > On Thu, 10 Apr 2025 at 23:49, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> >>
> >> On 10/04/2025 8:50 pm, Jason Andryuk wrote:
> >>> A Xen PVH dom0 on an AMD processor triple faults early in boot on
> >>> 6.6.86.  CPU detection appears to fail, as the faulting instruction is
> >>> vmcall in xen_hypercall_intel() and not vmmcall in xen_hypercall_amd().
> >>>
> >>> Detection fails because __xen_hypercall_setfunc() returns the full
> >>> kernel mapped address of xen_hypercall_amd() or xen_hypercall_intel() -
> >>> e.g. 0xffffffff815b93f0.  But this is compared against the rip-relative
> >>> xen_hypercall_amd(%rip), which when running from identity mapping, is
> >>> only 0x015b93f0.
> >>>
> >>> Replace the rip-relative address with just loading the actual address to
> >>> restore the proper comparision.
> >>>
> >>> This only seems to affect PVH dom0 boot.  This is probably because the
> >>> XENMEM_memory_map hypercall is issued early on from the identity
> >>> mappings.  With a domU, the memory map is provided via hvm_start_info
> >>> and the hypercall is skipped.  The domU is probably running from the
> >>> kernel high mapping when it issues hypercalls.
> >>>
> >>> Signed-off-by: Jason Andryuk <jason.andryuk@amd.com>
> >>> ---
> >>> I think this sort of address mismatch would be addresed by
> >>> e8fbc0d9cab6 ("x86/pvh: Call C code via the kernel virtual mapping")
> >>>
> >>> That could be backported instead, but it depends on a fair number of
> >>> patches.
> >>
> >> I've just spoken to Ard, and he thinks that it's standalone.  Should be
> >> ok to backport as a fix.
> >>
> >
> > I've tried building and booting 6.6.y with the patch applied - GS will
> > still be set to the 1:1 mapped address but that shouldn't matter,
> > given that it is only used for the stack canary, and we don't do
> > address comparisons on that afaik.
>
> Yes, it seems to work - I tested with dom0 and it booted.  I removed the
> use of phys_base - the diff is included below.  Does that match what you
> did?
>

The stable tree maintainers generally prefer the backports to be as
close to the originals as possible, and given that phys_base is
guaranteed to be 0x0, you might as well keep the subtraction.

> >>> Not sure on how getting a patch just into 6.6 would work.  This patch
> >>> could go into upstream Linux though it's not strictly necessary when the
> >>> rip-relative address is a high address.
> >>
> >> Do we know which other trees are broken?  I only found 6.6 because I was
> >> messing around with other bits of CI that happen to use 6.6.
> >>
> >
> > I'd assume all trees that had the hypercall page removal patch
> > backported to them will be broken in the same way.
>
> Yes, I think so.  Looks like it went back to 5.10 but not to 5.4.
>
> Ard, I can submit the stable request unless you want to.
>

Please go ahead.

Re: [PATCH TEST-ARTEFACTS 0/8] Cleanup and Linux ARM64 support

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1019 bytes --]

On Thu, 2025-04-10 at 19:28 +0100, Alejandro Vallejo wrote:
> The cutoff point is bcf0e2fda80c6("x86/xen: remove hypercall page").
> 
> Together with Jason's observation it would seem that Linux doesn't guess
> the correct instruction (or not early enough) when running as PVH dom0.
> On PV it's just "syscall", but on PVH it's a tad more complicated.
> 

I never understood why we did it this way anyway.

All this bogus complexity to do early detection of AMD vs. Intel and
use the right trap instruction, when we could have just continued to
use the proper Xen hypercall page at early boot.

After all, if you don't know what kind of CPU you're on, you *also*
haven't enabled CET or any other fancy return-tracking stuff yet. Just
fill in your own hypercall page *then*, instead. And then you can free
the original one.

Much simpler, much less fragile, and less prone to other potential
breakage from 64-bit latching side-effect that we forgot Xen does when
the guest sets the hypercall page.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5069 bytes --]