[PATCH v2 0/2] automation: qemu32 smoke test

[PATCH v2 0/2] automation: qemu32 smoke test

[Stefano Stabellini]

Hi all,

This small series adds a simple Xen + Dom0 boot arm32 test to gitlab-ci
using QEMU, similar to the existing tests for arm64 and x86.

Cheers,

Stefano

Stefano Stabellini (2):
      gitlab-ci: add qemu-system-arm to the existing tests-artifacts container
      gitlab-ci: add an ARM32 qemu-based smoke test

 automation/gitlab-ci/build.yaml                    | 14 ++++-
 automation/gitlab-ci/test.yaml                     | 27 +++++++-
 automation/scripts/qemu-smoke-arm32.sh             | 72 ++++++++++++++++++++++
 ...arm64v8.dockerfile => 6.0.0-arm64v8.dockerfile} |  5 +-
 4 files changed, 112 insertions(+), 6 deletions(-)
 create mode 100755 automation/scripts/qemu-smoke-arm32.sh
 rename automation/tests-artifacts/qemu-system-aarch64/{5.2.0-arm64v8.dockerfile => 6.0.0-arm64v8.dockerfile} (95%)

[PATCH v2 1/2] gitlab-ci: add qemu-system-arm to the existing tests-artifacts container

[Stefano Stabellini]

From: Stefano Stabellini <stefano.stabellini@xilinx.com>

Add qemu-system-arm to the existing test-artifacts qemu container (which
doesn't get build for every iteration but only updated once in a while.)

With qemu-system-arm available, we'll be able to run ARM32 tests.

This patch also bumps the QEMU version to v6.0.0 for both arm32 and
arm64 (the test-artifacts container is one, shared for both).

Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>
---
 automation/gitlab-ci/build.yaml                    | 14 ++++++++++++--
 automation/gitlab-ci/test.yaml                     |  4 ++--
 ...arm64v8.dockerfile => 6.0.0-arm64v8.dockerfile} |  5 +++--
 3 files changed, 17 insertions(+), 6 deletions(-)
 rename automation/tests-artifacts/qemu-system-aarch64/{5.2.0-arm64v8.dockerfile => 6.0.0-arm64v8.dockerfile} (95%)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index cc36428cf5..72f2a317ac 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -591,9 +591,9 @@ kernel-5.9.9-arm64-export:
   tags:
     - arm64
 
-qemu-system-aarch64-5.2.0-arm64-export:
+qemu-system-aarch64-6.0.0-arm64-export:
   stage: build
-  image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-aarch64:5.2.0-arm64v8
+  image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-aarch64:6.0.0-arm64v8
   script:
     - mkdir binaries && cp /qemu-system-aarch64 binaries/qemu-system-aarch64
   artifacts:
@@ -602,6 +602,16 @@ qemu-system-aarch64-5.2.0-arm64-export:
   tags:
     - arm64
 
+qemu-system-aarch64-6.0.0-arm32-export:
+  stage: build
+  image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-aarch64:6.0.0-arm64v8
+  script:
+    - mkdir binaries && cp /qemu-system-arm binaries/qemu-system-arm
+  artifacts:
+    paths:
+      - binaries/qemu-system-arm
+  tags:
+    - arm64
 
 # x86_64 test artifacts
 
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 47e8704df3..ec2a2e1607 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -33,7 +33,7 @@ qemu-alpine-arm64-gcc:
     - alpine-3.12-gcc-arm64
     - alpine-3.12-arm64-rootfs-export
     - kernel-5.9.9-arm64-export
-    - qemu-system-aarch64-5.2.0-arm64-export
+    - qemu-system-aarch64-6.0.0-arm64-export
   artifacts:
     paths:
       - smoke.serial
@@ -81,7 +81,7 @@ qemu-smoke-arm64-gcc:
   dependencies:
     - debian-unstable-gcc-arm64
     - kernel-5.9.9-arm64-export
-    - qemu-system-aarch64-5.2.0-arm64-export
+    - qemu-system-aarch64-6.0.0-arm64-export
   artifacts:
     paths:
       - smoke.serial
diff --git a/automation/tests-artifacts/qemu-system-aarch64/5.2.0-arm64v8.dockerfile b/automation/tests-artifacts/qemu-system-aarch64/6.0.0-arm64v8.dockerfile
similarity index 95%
rename from automation/tests-artifacts/qemu-system-aarch64/5.2.0-arm64v8.dockerfile
rename to automation/tests-artifacts/qemu-system-aarch64/6.0.0-arm64v8.dockerfile
index e105a1c636..793432d40b 100644
--- a/automation/tests-artifacts/qemu-system-aarch64/5.2.0-arm64v8.dockerfile
+++ b/automation/tests-artifacts/qemu-system-aarch64/6.0.0-arm64v8.dockerfile
@@ -3,7 +3,7 @@ LABEL maintainer.name="The Xen Project" \
       maintainer.email="xen-devel@lists.xenproject.org"
 
 ENV DEBIAN_FRONTEND=noninteractive
-ENV QEMU_VERSION=5.2.0
+ENV QEMU_VERSION=6.0.0
 ENV USER root
 
 RUN mkdir /build
@@ -25,7 +25,7 @@ RUN apt-get update && \
     tar xvJf qemu-"$QEMU_VERSION".tar.xz && \
     cd qemu-"$QEMU_VERSION" && \
     ./configure                \
-        --target-list=aarch64-softmmu \
+        --target-list=arm-softmmu,aarch64-softmmu \
         --enable-system        \
         --disable-blobs        \
         --disable-bsd-user     \
@@ -68,6 +68,7 @@ RUN apt-get update && \
     && \
     make -j$(nproc) && \
     cp ./build/qemu-system-aarch64 / && \
+    cp ./build/qemu-system-arm / && \
     cd /build && \
     rm -rf qemu-"$QEMU_VERSION"* && \
     apt-get autoremove -y && \
-- 
2.25.1

Re: [PATCH v2 1/2] gitlab-ci: add qemu-system-arm to the existing tests-artifacts container

[Anthony PERARD]

On Wed, Mar 16, 2022 at 06:46:52PM -0700, Stefano Stabellini wrote:
> +qemu-system-aarch64-6.0.0-arm32-export:
> +  stage: build
> +  image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-aarch64:6.0.0-arm64v8
> +  script:
> +    - mkdir binaries && cp /qemu-system-arm binaries/qemu-system-arm
> +  artifacts:
> +    paths:
> +      - binaries/qemu-system-arm
> +  tags:
> +    - arm64
>  

I wonder whether there's a better way to make available these kind of
test artifacts than using containers. But I don't have a good answer for
now.

Anyway, the patch looks good enough:
Reviewed-by: Anthony PERARD <anthony.perard@citrix.com>

Thanks,

-- 
Anthony PERARD

Re: [PATCH v2 1/2] gitlab-ci: add qemu-system-arm to the existing tests-artifacts container

[Stefano Stabellini]

On Fri, 18 Mar 2022, Anthony PERARD wrote:
> On Wed, Mar 16, 2022 at 06:46:52PM -0700, Stefano Stabellini wrote:
> > +qemu-system-aarch64-6.0.0-arm32-export:
> > +  stage: build
> > +  image: registry.gitlab.com/xen-project/xen/tests-artifacts/qemu-system-aarch64:6.0.0-arm64v8
> > +  script:
> > +    - mkdir binaries && cp /qemu-system-arm binaries/qemu-system-arm
> > +  artifacts:
> > +    paths:
> > +      - binaries/qemu-system-arm
> > +  tags:
> > +    - arm64
> >  
> 
> I wonder whether there's a better way to make available these kind of
> test artifacts than using containers. But I don't have a good answer for
> now.

I wonder the same but couldn't find anything better.


> Anyway, the patch looks good enough:
> Reviewed-by: Anthony PERARD <anthony.perard@citrix.com>

Thanks!

[PATCH v2 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Stefano Stabellini]

Add a minimal ARM32 smoke test based on qemu-system-arm, as provided by
the test-artifacts qemu container. The minimal test simply boots Xen
(built from previous build stages) and Dom0. The test is fetching the
Dom0 kernel and initrd from Debian Jessie: they work just fine and this
way we don't have to maintain a build for them too.

Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com>

---
Changes in v2:
- improve comments
- don't limit dom0 cpus
- decrease mem to 1024M and dom0_mem=512M
- use Debian Bullseye instead of Jessie
---
 automation/gitlab-ci/test.yaml         | 23 ++++++++
 automation/scripts/qemu-smoke-arm32.sh | 72 ++++++++++++++++++++++++++
 2 files changed, 95 insertions(+)
 create mode 100755 automation/scripts/qemu-smoke-arm32.sh

diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index ec2a2e1607..42cd725a12 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -95,6 +95,29 @@ qemu-smoke-arm64-gcc:
     - /^coverity-tested\/.*/
     - /^stable-.*/
 
+qemu-smoke-arm32-gcc:
+  stage: test
+  image: registry.gitlab.com/xen-project/xen/${CONTAINER}
+  variables:
+    CONTAINER: debian:unstable-arm64v8
+  script:
+    - ./automation/scripts/qemu-smoke-arm32.sh 2>&1 | tee qemu-smoke-arm32.log
+  dependencies:
+    - debian-unstable-gcc-arm32
+    - qemu-system-aarch64-6.0.0-arm32-export
+  artifacts:
+    paths:
+      - smoke.serial
+      - '*.log'
+    when: always
+  tags:
+    - arm64
+  except:
+    - master
+    - smoke
+    - /^coverity-tested\/.*/
+    - /^stable-.*/
+
 qemu-smoke-x86-64-gcc:
   stage: test
   image: registry.gitlab.com/xen-project/xen/${CONTAINER}
diff --git a/automation/scripts/qemu-smoke-arm32.sh b/automation/scripts/qemu-smoke-arm32.sh
new file mode 100755
index 0000000000..11883bed3c
--- /dev/null
+++ b/automation/scripts/qemu-smoke-arm32.sh
@@ -0,0 +1,72 @@
+#!/bin/bash
+
+set -ex
+
+export DEBIAN_FRONTENT=noninteractive
+apt-get -qy update
+apt-get -qy install --no-install-recommends device-tree-compiler \
+                                            curl
+
+cd binaries
+curl -fsSLO http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
+curl -fsSLO http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz
+
+kernel=`stat -L --printf="%s" vmlinuz`
+initrd=`stat -L --printf="%s" initrd.gz`
+
+# For Xen, we need a couple of more node. Dump the DT from QEMU and add them
+# XXX QEMU looks for "efi-virtio.rom" even if it is unneeded
+curl -fsSLO https://github.com/qemu/qemu/raw/v5.2.0/pc-bios/efi-virtio.rom
+./qemu-system-arm \
+   -machine virt-6.0 \
+   -machine virtualization=true \
+   -smp 4 \
+   -m 1024 \
+   -serial stdio \
+   -monitor none \
+   -display none \
+   -machine dumpdtb=virt.dtb
+
+dtc -I dtb -O dts virt.dtb > virt.dts
+
+cat >> virt.dts << EOF
+/ {
+	chosen {
+		#address-cells = <0x2>;
+		#size-cells = <0x2>;
+		stdout-path = "/pl011@9000000";
+        xen,xen-bootargs = "console=dtuart dtuart=/pl011@9000000 dom0_mem=512M bootscrub=0";
+		xen,dom0-bootargs = "console=tty0 console=hvc0 earlyprintk clk_ignore_unused root=/dev/ram0 rdinit=/bin/sh init=/bin/sh";
+		dom0 {
+			compatible = "xen,linux-zimage", "xen,multiboot-module";
+			reg = <0x0 0x1000000 0x0 $kernel>;
+		};
+        dom0-ramdisk {
+			compatible = "xen,linux-initrd", "xen,multiboot-module";
+			reg = <0x0 0x3200000 0x0 $initrd>;
+		};
+	};
+};
+EOF
+dtc -I dts -O dtb virt.dts > virt.dtb
+
+rm -f smoke.serial
+set +e
+timeout -k 1 240 \
+./qemu-system-arm \
+   -machine virt-6.0 \
+   -machine virtualization=true \
+   -smp 4 \
+   -m 1024 \
+   -serial stdio \
+   -monitor none \
+   -display none \
+   -dtb virt.dtb \
+   -no-reboot \
+   -kernel ./xen \
+   -device loader,file=./vmlinuz,addr=0x1000000 \
+   -device loader,file=./initrd.gz,addr=0x3200000 |& tee smoke.serial
+
+set -e
+(grep -q "^BusyBox" smoke.serial) || exit 1
+exit 0
-- 
2.25.1

Re: [PATCH v2 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Anthony PERARD]

On Wed, Mar 16, 2022 at 06:46:53PM -0700, Stefano Stabellini wrote:
> +curl -fsSLO http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
> +curl -fsSLO http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz

    curl --fail --silent --show-error --location --remote-name

I didn't know what all those short options were, I had to write it
down. In script to be shared with others, I think it's better to use the
long options, as it's kind of self-documenting.

Now, there an issue with the command line, "--remote-name" (or -O) needs
to be replaced. We don't care about how the server calls the file, it is
much more important do know where it is going to be stored. Instead, you
should use "--output" to write the file to the location the script is
going to use. It happened to work now, but we don't have to trust
the Internet when not needed.

Also, maybe use "https"?


Also, maybe as an improvement for later, and to avoid having to rely on
the Internet, we could probably store those artifacts in the container
that's going to run the test. But I'm not asking this for now.

> +timeout -k 1 240 \
> +./qemu-system-arm \

There's probably a better way than waiting for the "timeout" like running an
"init" that print something and shutdown the machine. But I guess
that's ok for now.


Thanks,

-- 
Anthony PERARD

Re: [PATCH v2 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Stefano Stabellini]

On Fri, 18 Mar 2022, Anthony PERARD wrote:
> On Wed, Mar 16, 2022 at 06:46:53PM -0700, Stefano Stabellini wrote:
> > +curl -fsSLO http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
> > +curl -fsSLO http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz
> 
>     curl --fail --silent --show-error --location --remote-name
> 
> I didn't know what all those short options were, I had to write it
> down. In script to be shared with others, I think it's better to use the
> long options, as it's kind of self-documenting.
> 
> Now, there an issue with the command line, "--remote-name" (or -O) needs
> to be replaced. We don't care about how the server calls the file, it is
> much more important do know where it is going to be stored. Instead, you
> should use "--output" to write the file to the location the script is
> going to use. It happened to work now, but we don't have to trust
> the Internet when not needed.
> 

Sure I copy-pasted it from the other scripts. With your suggestion it
becomes:
curl --fail --silent --show-error --location --remote-name http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz

I can make the change


> Also, maybe use "https"?

https doesn't work for this:

$ curl --fail --silent --show-error --location --remote-name https://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
curl: (60) SSL: no alternative certificate subject name matches target host name 'http.us.debian.org'


> Also, maybe as an improvement for later, and to avoid having to rely on
> the Internet, we could probably store those artifacts in the container
> that's going to run the test. But I'm not asking this for now.

I agree! Even better if we could update the kernel too. I think for now
it is OK but I'll try to improve it in the following weeks.


> > +timeout -k 1 240 \
> > +./qemu-system-arm \
> 
> There's probably a better way than waiting for the "timeout" like running an
> "init" that print something and shutdown the machine. But I guess
> that's ok for now.

Yeah I agree on both points. The other 3 tests are in the same
situation.

Re: [PATCH v2 2/2] gitlab-ci: add an ARM32 qemu-based smoke test

[Stefano Stabellini]

On Fri, 18 Mar 2022, Stefano Stabellini wrote:
> On Fri, 18 Mar 2022, Anthony PERARD wrote:
> > On Wed, Mar 16, 2022 at 06:46:53PM -0700, Stefano Stabellini wrote:
> > > +curl -fsSLO http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
> > > +curl -fsSLO http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz
> > 
> >     curl --fail --silent --show-error --location --remote-name
> > 
> > I didn't know what all those short options were, I had to write it
> > down. In script to be shared with others, I think it's better to use the
> > long options, as it's kind of self-documenting.
> > 
> > Now, there an issue with the command line, "--remote-name" (or -O) needs
> > to be replaced. We don't care about how the server calls the file, it is
> > much more important do know where it is going to be stored. Instead, you
> > should use "--output" to write the file to the location the script is
> > going to use. It happened to work now, but we don't have to trust
> > the Internet when not needed.
> > 
> 
> Sure I copy-pasted it from the other scripts. With your suggestion it
> becomes:
> curl --fail --silent --show-error --location --remote-name http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
> 
> I can make the change

Sorry, I replied too quickly. It should be:

curl --fail --silent --show-error --location --output vmlinuz http://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz

 
> > Also, maybe use "https"?
> 
> https doesn't work for this:
> 
> $ curl --fail --silent --show-error --location --remote-name https://http.us.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz
> curl: (60) SSL: no alternative certificate subject name matches target host name 'http.us.debian.org'
 
> 
> > Also, maybe as an improvement for later, and to avoid having to rely on
> > the Internet, we could probably store those artifacts in the container
> > that's going to run the test. But I'm not asking this for now.
> 
> I agree! Even better if we could update the kernel too. I think for now
> it is OK but I'll try to improve it in the following weeks.
> 
> 
> > > +timeout -k 1 240 \
> > > +./qemu-system-arm \
> > 
> > There's probably a better way than waiting for the "timeout" like running an
> > "init" that print something and shutdown the machine. But I guess
> > that's ok for now.
> 
> Yeah I agree on both points. The other 3 tests are in the same
> situation.
>