Re: [PATCH v2 2/2] automation: introduce a dom0less test run on Xilinx hardware

[Michal Orzel <michal.orzel@amd.com>]

Hi Stefano,

On 04/03/2023 00:57, Stefano Stabellini wrote:
> 
> 
> From: Stefano Stabellini <stefano.stabellini@amd.com>
> 
> The test prepares dom0 and domU binaries and boot artifacts, similarly
> to the existing QEMU test. (TBD: share preparation steps with the
> regular QEMU tests.)
> 
> However, instead of running the test inside QEMU as usual, it copies
> the binaries to the tftp server root, triggers a Xilinx ZCU102 board
> reboot, and connects to the real serial of the board.
> 
> For now and due to its novelty, allow_failure on the Xilinx hardware
> test, and only run the job on protected branches with XILINX_JOBS set to
> true (the "master" and "staging" on gitlab.com/xen-project/xen qualify).
> 
> Signed-off-by: Stefano Stabellini <stefano.stabellini@amd.com>
> ---
> Changes in v2:
> - only execute the xilinx job on protected branches with XILINX_JOBS set
> to true
> ---
>  automation/gitlab-ci/test.yaml                |  22 ++++
>  .../scripts/xilinx-smoke-dom0less-arm64.sh    | 115 ++++++++++++++++++
>  2 files changed, 137 insertions(+)
>  create mode 100755 automation/scripts/xilinx-smoke-dom0less-arm64.sh
> 
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 1c5f400b68..8f94cad32c 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -85,6 +85,28 @@ build-each-commit-gcc:
>    tags:
>      - x86_64
> 
> +xilinx-smoke-dom0less-arm64-gcc:
> +  extends: .test-jobs-common
> +  variables:
> +    CONTAINER: ubuntu:xenial-xilinx
> +    LOGFILE: qemu-smoke-xilinx.log
> +  artifacts:
> +    paths:
> +      - smoke.serial
> +      - '*.log'
> +    when: always
> +  tags:
> +    - xilinx
> +  script:
> +    - ./automation/scripts/xilinx-smoke-dom0less-arm64.sh 2>&1 | tee ${LOGFILE}
> +  needs:
> +    - *arm64-test-needs
> +    - alpine-3.12-gcc-arm64
> +  allow_failure: true
> +  only:
> +    variables:
> +      - $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
> +
Surely we will want to have more tests running on our HW board.
For that, let's introduce a template and a job separately right away:

.xilinx-arm64:
  extends: .test-jobs-common
  variables:
    CONTAINER: ubuntu:xenial-xilinx
    LOGFILE: qemu-smoke-xilinx.log
  artifacts:
    paths:
      - smoke.serial
      - '*.log'
    when: always
  allow_failure: true
  only:
    variables:
      - $XILINX_JOBS == "true" && $CI_COMMIT_REF_PROTECTED == "true"
  tags:
    - xilinx


xilinx-smoke-dom0less-arm64-gcc:
  extends: .xilinx-arm64
  script:
    - ./automation/scripts/xilinx-smoke-dom0less-arm64.sh 2>&1 | tee ${LOGFILE}
  needs:
    - *arm64-test-needs
    - alpine-3.12-gcc-arm64


>  qemu-smoke-dom0-arm64-gcc:
>    extends: .qemu-arm64
>    script:
> diff --git a/automation/scripts/xilinx-smoke-dom0less-arm64.sh b/automation/scripts/xilinx-smoke-dom0less-arm64.sh
> new file mode 100755
> index 0000000000..180c8b5f1c
> --- /dev/null
> +++ b/automation/scripts/xilinx-smoke-dom0less-arm64.sh
> @@ -0,0 +1,115 @@
> +#!/bin/bash
> +
> +set -ex
> +
> +test_variant=$1
> +
> +if [ -z "${test_variant}" ]; then
> +    passed="ping test passed"
> +    domU_check="
> +until ifconfig eth0 192.168.0.2 &> /dev/null && ping -c 10 192.168.0.1; do
> +    sleep 30
> +done
> +echo \"${passed}\"
> +"
> +fi
> +
> +# DomU
> +mkdir -p rootfs
> +cd rootfs
> +tar xzf ../binaries/initrd.tar.gz
> +mkdir proc
> +mkdir run
> +mkdir srv
> +mkdir sys
> +rm var/run
> +echo "#!/bin/sh
> +
> +${domU_check}
> +/bin/sh" > etc/local.d/xen.start
> +chmod +x etc/local.d/xen.start
> +echo "rc_verbose=yes" >> etc/rc.conf
> +find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz
> +cd ..
> +rm -rf rootfs
Why do we create a domU rootfs in a different manner compared to qemu one?
I think this could be done using busybox (it would have to be installed in container)
and it would make it easier to share qemu/xilinx script.

> +
> +# DOM0 rootfs
> +mkdir -p rootfs
> +cd rootfs
> +tar xzf ../binaries/initrd.tar.gz
> +mkdir proc
> +mkdir run
> +mkdir srv
> +mkdir sys
> +rm var/run
> +cp -ar ../binaries/dist/install/* .
> +
> +echo "#!/bin/bash
> +
> +export LD_LIBRARY_PATH=/usr/local/lib
> +bash /etc/init.d/xencommons start
> +
> +/usr/local/lib/xen/bin/init-dom0less
> +
> +brctl addbr xenbr0
> +brctl addif xenbr0 eth0
> +ifconfig eth0 up
> +ifconfig xenbr0 up
> +ifconfig xenbr0 192.168.0.1
> +
> +xl network-attach 1 type=vif
> +${dom0_check}
> +" > etc/local.d/xen.start
> +chmod +x etc/local.d/xen.start
> +echo "rc_verbose=yes" >> etc/rc.conf
> +find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz
> +cd ..
> +
> +
> +TFTP=/scratch/gitlab-runner/tftp
> +START=`pwd`
> +
> +# ImageBuilder
> +echo 'MEMORY_START="0"
> +MEMORY_END="0xC0000000"
This is incorrect for zcu102. It should be the end address of a first memory bank
which is 0x7ff00000.

> +
> +DEVICE_TREE="mpsoc.dtb"
Can we do something to expose this dtb so that everyone with no access to a runner can see it?
It will become necessary when we start introducing some passthrough test jobs.
Also, how about naming this: mpsoc_smmu.dtb to make it clear that we are running with IOMMU enabled?
In the future we might want to test both configurations, especially for static partitioning use cases.

> +XEN="xen"
> +DOM0_KERNEL="Image"
> +DOM0_RAMDISK="dom0-rootfs.cpio.gz"
> +XEN_CMD="console=dtuart dom0_mem=1024M"
With just "console=dtuart" we are relying on a presence of "stdout-path" under /chosen which is optional.
I would suggest: "console=dtuart dtuart=serial0"

> +
> +NUM_DOMUS=1
> +DOMU_KERNEL[0]="Image"
> +DOMU_RAMDISK[0]="domU-rootfs.cpio.gz"
> +DOMU_MEM[0]="1024"
> +
> +LOAD_CMD="tftpb"
> +UBOOT_SOURCE="boot.source"
> +UBOOT_SCRIPT="boot.scr"' > $TFTP/config
> +
> +cp -f binaries/xen $TFTP/
> +cp -f binaries/Image $TFTP/
> +cp -f binaries/dom0-rootfs.cpio.gz $TFTP/
> +cp -f binaries/domU-rootfs.cpio.gz $TFTP/
> +
> +rm -rf imagebuilder
> +git clone https://gitlab.com/ViryaOS/imagebuilder
> +bash imagebuilder/scripts/uboot-script-gen -t tftp -d $TFTP/ -c $TFTP/config
> +
> +# restart the board
> +cd /scratch/gitlab-runner
> +bash zcu102.sh 2
> +sleep 5
> +bash zcu102.sh 1
> +sleep 5
> +cd $START
> +
> +# connect to serial
> +set +e
> +stty -F /dev/ttyUSB0 115200
> +timeout -k 1 120 nohup sh -c "cat /dev/ttyUSB0 | tee smoke.serial"
> +
> +set -e
> +(grep -q "^Welcome to Alpine Linux" smoke.serial && grep -q "${passed}" smoke.serial) || exit 1
> +exit 0
> --
> 2.25.1
> 

Great job, this is awesome.

~Michal