* [RFC TEST_ARTIFACTS PATCH 0/2] artifacts: Initial commit @ 2024-10-23 2:53 victorm.lira 2024-10-23 2:53 ` [RFC TEST_ARTIFACTS PATCH 1/2] " victorm.lira 0 siblings, 1 reply; 5+ messages in thread From: victorm.lira @ 2024-10-23 2:53 UTC (permalink / raw) To: xen-devel Cc: Victor Lira, Andrew Cooper, Marek Marczykowski-Górecki, Doug Goldstein, Stefano Stabellini From: Victor Lira <victorm.lira@amd.com> This patch series is a proof of concept that a separate repository can be used to generate kernel, rootfs, and other commonly used binary artifacts, and have Xen test jobs load these instead of generating them every run. The current configuration of Xen CI generates these using "dummy" jobs in every pipeline. This unecessarily uses a large amount of storage and network traffic because artifacts are duplicated overprojects/branches. Instead, use GitLab CI built-in functionality to retrieve them from a separate repository using the "needs" keyword. The "needs:project" feature is currently limited to 5 artifacts archives from external pipelines per job, but web requests can be used if more are needed. It's preferable to use this as opposed to just web requests since this doesn't requre modying the any test scripts as long as the archive has the same directory structure. The example of a test artifacts repository: https://gitlab.com/xen-project/people/victormlira/test-artifacts/-/pipelines/1508031895 The example of the use: https://gitlab.com/xen-project/people/sstabellini/xen/-/pipelines/1508069266 Stefano Stabellini (1): Initial commit -- 2.25.1 ^ permalink raw reply [flat|nested] 5+ messages in thread
* [RFC TEST_ARTIFACTS PATCH 1/2] artifacts: Initial commit 2024-10-23 2:53 [RFC TEST_ARTIFACTS PATCH 0/2] artifacts: Initial commit victorm.lira @ 2024-10-23 2:53 ` victorm.lira 2024-10-23 2:53 ` [RFC XEN PATCH 2/2] automation: add xilinx test (linux argo) victorm.lira 2024-10-23 4:57 ` [RFC TEST_ARTIFACTS PATCH 1/2] artifacts: Initial commit Stefano Stabellini 0 siblings, 2 replies; 5+ messages in thread From: victorm.lira @ 2024-10-23 2:53 UTC (permalink / raw) To: xen-devel Cc: Stefano Stabellini, Andrew Cooper, Victor Lira, Marek Marczykowski-Górecki, Doug Goldstein From: Stefano Stabellini <sstabellini@kernel.org> Create new repository to produce common artifacts for Xen tests Requested-by: Andrew Cooper <andrew.cooper3@citrix.com> Signed-off-by: Victor Lira <victorm.lira@amd.com> --- Cc: Andrew Cooper <andrew.cooper3@citrix.com> Cc: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> Cc: Doug Goldstein <cardoe@cardoe.com> Cc: Stefano Stabellini <sstabellini@kernel.org> Cc: xen-devel@lists.xenproject.org --- .gitlab-ci.yml | 48 ++++++++++++++++++++ README.md | 10 ++++ binaries/.gitignore | 3 ++ images/Makefile | 20 ++++++++ images/alpine/x86_64-build.dockerfile | 25 ++++++++++ images/alpine/x86_64-rootfs.dockerfile | 4 ++ scripts/x86_64-argo-linux.sh | 63 ++++++++++++++++++++++++++ scripts/x86_64-kernel-linux.sh | 31 +++++++++++++ scripts/x86_64-rootfs-alpine.sh | 58 ++++++++++++++++++++++++ 9 files changed, 262 insertions(+) create mode 100644 .gitlab-ci.yml create mode 100644 README.md create mode 100644 binaries/.gitignore create mode 100644 images/Makefile create mode 100644 images/alpine/x86_64-build.dockerfile create mode 100644 images/alpine/x86_64-rootfs.dockerfile create mode 100755 scripts/x86_64-argo-linux.sh create mode 100755 scripts/x86_64-kernel-linux.sh create mode 100755 scripts/x86_64-rootfs-alpine.sh diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000..f2f7738 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,48 @@ +workflow: + name: "xen test artifacts" + +.artifacts: + stage: build + image: + name: registry.gitlab.com/xen-project/people/victormlira/test-artifacts/${CONTAINER} + artifacts: + name: "${CI_JOB_NAME_SLUG}" + paths: + - binaries/ + +.x86_64-artifacts: + extends: .artifacts + image: + docker: + platform: linux/amd64 + user: xenproject + variables: + CONTAINER: alpine:x86_64-build + tags: + - x86_64 + +x86_64-kernel-linux-6.6.56: + extends: .x86_64-artifacts + script: + - . scripts/x86_64-kernel-linux.sh + variables: + LINUX_VERSION: 6.6.56 + +x86_64-rootfs-alpine-3.18: + extends: .x86_64-artifacts + variables: + CONTAINER: alpine:x86_64-rootfs + image: + docker: + user: root + script: + - . scripts/x86_64-rootfs-alpine.sh + +x86_64-argo-linux-6.6.56: + extends: .x86_64-artifacts + script: + - . scripts/x86_64-argo-linux.sh + variables: + LINUX_VERSION: 6.6.56 + ARGO_SHA: "705a7a8a624b42e13e655d3042059b8a85cdf6a3" + ARGOEXEC_SHA: "d900429f6640acc6f68a3d3a4c945d7da60625d8" diff --git a/README.md b/README.md new file mode 100644 index 0000000..0d72519 --- /dev/null +++ b/README.md @@ -0,0 +1,10 @@ +# Xen Project Test Artifacts + +## Overview + +This repository is used to produce common artifacts for Xen tests. + +## + + +## diff --git a/binaries/.gitignore b/binaries/.gitignore new file mode 100644 index 0000000..95e2a2e --- /dev/null +++ b/binaries/.gitignore @@ -0,0 +1,3 @@ +# Keep this directory around to be available in CI/CD jobs. +* +!.gitignore diff --git a/images/Makefile b/images/Makefile new file mode 100644 index 0000000..41638d7 --- /dev/null +++ b/images/Makefile @@ -0,0 +1,20 @@ + +# The base of where these containers will appear +REGISTRY := registry.gitlab.com/xen-project/people/victormlira/test-artifacts +CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile)) + +help: + @echo "Containers to build test artifacts." + @echo "To build one run 'make DISTRO/TAG'." + @echo "Available containers:" + @$(foreach file,$(sort $(CONTAINERS)),echo ${file};) + @echo "To push container builds, set the environment variable PUSH" + +%: %.dockerfile ## Builds containers + docker build --pull -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D) + @if [ ! -z $${PUSH+x} ]; then \ + docker push $(REGISTRY)/$(@D):$(@F); \ + fi + +.PHONY: all +all: $(CONTAINERS) diff --git a/images/alpine/x86_64-build.dockerfile b/images/alpine/x86_64-build.dockerfile new file mode 100644 index 0000000..374952f --- /dev/null +++ b/images/alpine/x86_64-build.dockerfile @@ -0,0 +1,25 @@ +# syntax=docker/dockerfile:1 +FROM --platform=linux/amd64 alpine:3.18 +LABEL maintainer.name="The Xen Project" \ + maintainer.email="xen-devel@lists.xenproject.org" + +WORKDIR /build/ + +RUN <<EOF +adduser -D xenproject +apk --no-cache add \ + musl-dev \ + build-base \ + libc6-compat \ + linux-headers \ + bash \ + git \ + curl \ + flex \ + bison \ + elfutils-dev \ + autoconf \ + automake \ + libtool \ + openssl-dev +EOF diff --git a/images/alpine/x86_64-rootfs.dockerfile b/images/alpine/x86_64-rootfs.dockerfile new file mode 100644 index 0000000..b912e9c --- /dev/null +++ b/images/alpine/x86_64-rootfs.dockerfile @@ -0,0 +1,4 @@ +# syntax=docker/dockerfile:1 +FROM --platform=linux/amd64 alpine:3.18 +LABEL maintainer.name="The Xen Project" \ + maintainer.email="xen-devel@lists.xenproject.org" diff --git a/scripts/x86_64-argo-linux.sh b/scripts/x86_64-argo-linux.sh new file mode 100755 index 0000000..a110a33 --- /dev/null +++ b/scripts/x86_64-argo-linux.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash + +if test -z "${LINUX_VERSION}" +then + >&2 echo "LINUX_VERSION must be set"; exit 1 +fi + +set -ex -o pipefail + +BUILDDIR="${PWD}" +COPYDIR="${BUILDDIR}/binaries/" + +# Prepare Linux sources +curl -fsSLO \ + https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-"${LINUX_VERSION}".tar.xz +tar xJf linux-"${LINUX_VERSION}".tar.xz +cd linux-"${LINUX_VERSION}" +make ARCH=x86 defconfig +make ARCH=x86 xen.config +./scripts/config --enable BRIDGE +./scripts/config --enable IGC +./scripts/config --enable TUN +cp .config .config.orig +cat .config.orig \ + | grep 'XEN' \ + | grep '=m' \ + | sed 's/=m/=y/g' \ + >> .config +make ARCH=x86 olddefconfig +make ARCH=x86 modules_prepare + +# Build Linux kernel module for Xen Argo +cd "${BUILDDIR}" +git clone \ + --depth=1 --branch=master \ + https://github.com/OpenXT/linux-xen-argo.git +git -C "${BUILDDIR}/linux-xen-argo" switch --detach "${ARGO_SHA}" +make -C "linux-${LINUX_VERSION}" M="${BUILDDIR}/linux-xen-argo/argo-linux" \ + CFLAGS_MODULE="-Wno-error" KBUILD_MODPOST_WARN=1 +cp "linux-xen-argo/argo-linux/xen-argo.ko" "${COPYDIR}/xen-argo.ko" + +# Build Linux libargo shared library, applying fixes to build in Alpine Linux +cd "${BUILDDIR}/linux-xen-argo/libargo" +sed -i "s|AM_INIT_AUTOMAKE|AC_CONFIG_AUX_DIR(.)\nAM_INIT_AUTOMAKE|" configure.ac +sed -i "s/__SOCKADDR_COMMON (sxenargo_)/sa_family_t sxenargo_family/" src/libargo.h +sed -i "s/__SOCKADDR_COMMON_SIZE/(sizeof (unsigned short int))/" src/libargo.h +autoreconf --install +./configure --prefix="${COPYDIR}" CPPFLAGS="-I${PWD}/../argo-linux/include" +make +make install + +# Build Linux user program, modifying for xilinx argo test +cd "${BUILDDIR}" +wget "https://raw.githubusercontent.com/OpenXT/xenclient-oe/${ARGOEXEC_SHA}/\ +recipes-openxt/argo-exec/argo-exec/argo-exec.c" +sed -i "s|#include <xen/xen.h>||" argo-exec.c +sed -i "s|ret = shuffle(s, fds\[0\], fds\[1\]);|ret = shuffle(s, 0, 1);|" \ + argo-exec.c +gcc -I"${BUILDDIR}/linux-xen-argo/libargo/src" \ + -I"${BUILDDIR}/linux-xen-argo/argo-linux/include" \ + -L"${COPYDIR}/lib/" \ + -o argo-exec argo-exec.c -largo +cp argo-exec "${COPYDIR}" diff --git a/scripts/x86_64-kernel-linux.sh b/scripts/x86_64-kernel-linux.sh new file mode 100755 index 0000000..17be36b --- /dev/null +++ b/scripts/x86_64-kernel-linux.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash + +if test -z "${LINUX_VERSION}" +then + >&2 echo "LINUX_VERSION must be set"; exit 1 +fi + +set -ex -o pipefail + +WORKDIR="${PWD}" +COPYDIR="${WORKDIR}/binaries/" + +# Build Linux +curl -fsSLO \ + https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-"${LINUX_VERSION}".tar.xz +tar xJf linux-"${LINUX_VERSION}".tar.xz +cd linux-"${LINUX_VERSION}" +make ARCH=x86 defconfig +make ARCH=x86 xen.config +./scripts/config --enable BRIDGE +./scripts/config --enable IGC +./scripts/config --enable TUN +cp .config .config.orig +cat .config.orig \ + | grep 'XEN' \ + | grep '=m' \ + | sed 's/=m/=y/g' \ + >> .config +make ARCH=x86 olddefconfig +make ARCH=x86 "-j$(nproc)" --quiet +cp arch/x86/boot/bzImage "${COPYDIR}" diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/x86_64-rootfs-alpine.sh new file mode 100755 index 0000000..28d8e30 --- /dev/null +++ b/scripts/x86_64-rootfs-alpine.sh @@ -0,0 +1,58 @@ +WORKDIR="${PWD}" + +apk update + +# xen runtime deps +apk add musl +apk add libgcc +apk add openrc +apk add busybox +apk add sudo +apk add dbus +apk add bash +apk add python3 +apk add zlib +apk add ncurses +apk add yajl +apk add libaio +apk add xz +apk add util-linux +apk add argp-standalone +apk add libfdt +apk add glib +apk add pixman +apk add curl +apk add udev +apk add pciutils +apk add libelf + +# Xen +cd / +# Minimal ramdisk environment in case of cpio output +rc-update add udev +rc-update add udev-trigger +rc-update add udev-settle +rc-update add loopback sysinit +rc-update add bootmisc boot +rc-update add devfs sysinit +rc-update add dmesg sysinit +rc-update add hostname boot +rc-update add hwclock boot +rc-update add hwdrivers sysinit +rc-update add modules boot +rc-update add killprocs shutdown +rc-update add mount-ro shutdown +rc-update add savecache shutdown +rc-update add local default +cp -a /sbin/init /init +echo "ttyS0" >> /etc/securetty +echo "hvc0" >> /etc/securetty +echo "ttyS0::respawn:/sbin/getty -L ttyS0 115200 vt100" >> /etc/inittab +echo "hvc0::respawn:/sbin/getty -L hvc0 115200 vt100" >> /etc/inittab +echo > /etc/modules +passwd -d "root" root + +# Create rootfs +cd / +tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \ + bin dev etc home init lib mnt opt root sbin usr var -- 2.25.1 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* [RFC XEN PATCH 2/2] automation: add xilinx test (linux argo) 2024-10-23 2:53 ` [RFC TEST_ARTIFACTS PATCH 1/2] " victorm.lira @ 2024-10-23 2:53 ` victorm.lira 2024-10-23 5:03 ` Stefano Stabellini 2024-10-23 4:57 ` [RFC TEST_ARTIFACTS PATCH 1/2] artifacts: Initial commit Stefano Stabellini 1 sibling, 1 reply; 5+ messages in thread From: victorm.lira @ 2024-10-23 2:53 UTC (permalink / raw) To: xen-devel Cc: Victor Lira, Andrew Cooper, Marek Marczykowski-Górecki, Doug Goldstein, Stefano Stabellini From: Victor Lira <victorm.lira@amd.com> Add x86_64 hardware test that creates a Xen Argo communication connection between two PVH domains. In the test, dom0 creates a domU and listens for messages sent by the domU through Argo. To accomplish this, build Xen with CONFIG_ARGO=y. Update the xilinx x86_64 test script to support the new test, and add "sync_console" to command line to avoid an issue with console messages being lost. Signed-off-by: Victor Lira <victorm.lira@amd.com> --- Cc: Andrew Cooper <andrew.cooper3@citrix.com> Cc: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> Cc: Doug Goldstein <cardoe@cardoe.com> Cc: Stefano Stabellini <sstabellini@kernel.org> Cc: xen-devel@lists.xenproject.org --- automation/gitlab-ci/build.yaml | 6 ++ automation/gitlab-ci/test.yaml | 16 ++++ .../scripts/xilinx-smoke-dom0-x86_64.sh | 76 +++++++++++++------ 3 files changed, 73 insertions(+), 25 deletions(-) diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml index c2db69ecf8..f425744f8b 100644 --- a/automation/gitlab-ci/build.yaml +++ b/automation/gitlab-ci/build.yaml @@ -340,6 +340,12 @@ alpine-3.18-gcc-debug: variables: CONTAINER: alpine:3.18 BUILD_QEMU_XEN: y + EXTRA_XEN_CONFIG: | + CONFIG_EXPERT=y + CONFIG_UNSUPPORTED=y + CONFIG_ARGO=y debian-bookworm-gcc-debug: extends: .gcc-x86-64-build-debug diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml index b27c2be174..2d4709d393 100644 --- a/automation/gitlab-ci/test.yaml +++ b/automation/gitlab-ci/test.yaml @@ -192,6 +192,22 @@ xilinx-smoke-dom0-x86_64-gcc-debug: - *x86-64-test-needs - alpine-3.18-gcc-debug +xilinx-smoke-dom0-x86_64-gcc-debug-argo: + extends: .xilinx-x86_64 + script: + - ./automation/scripts/xilinx-smoke-dom0-x86_64.sh argo 2>&1 | tee ${LOGFILE} + needs: + - alpine-3.18-gcc-debug + - project: xen-project/people/victormlira/test-artifacts + job: x86_64-kernel-linux-6.6.56 + ref: vml-tests-separate-artifacts + - project: xen-project/people/victormlira/test-artifacts + job: x86_64-rootfs-alpine-3.18 + ref: vml-tests-separate-artifacts + - project: xen-project/people/victormlira/test-artifacts + job: x86_64-argo-linux-6.6.56 + ref: vml-tests-separate-artifacts + adl-smoke-x86-64-gcc-debug: extends: .adl-x86-64 script: diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh index 7027f083ba..f70cfdc155 100755 --- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh +++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh @@ -9,6 +9,8 @@ fatal() { exit 1 } +WORKDIR="${PWD}" + # Test parameter defaults. TEST="$1" PASS_MSG="Test passed: ${TEST}" @@ -24,9 +26,10 @@ kernel = "/boot/vmlinuz" ramdisk = "/boot/initrd-domU" extra = "root=/dev/ram0 console=hvc0" memory = 512 -vif = [ "bridge=xenbr0", ] -disk = [ ] ' +DOMU_CFG_EXTRA="" +copy_domU_files () { :; } +copy_dom0_files () { :; } # Select test variant. if [ "${TEST}" = "ping" ]; then @@ -39,6 +42,14 @@ done echo \"${DOMU_MSG}\" " DOM0_CMD=" +brctl addbr xenbr0 +brctl addif xenbr0 eth0 +ifconfig eth0 up +ifconfig xenbr0 up +ifconfig xenbr0 192.168.0.1 +# get domU console content into test log +tail -F /var/log/xen/console/guest-domU.log 2>/dev/null | sed -e \"s/^/(domU) /\" & +xl create /etc/xen/domU.cfg set +x until grep -q \"${DOMU_MSG}\" /var/log/xen/console/guest-domU.log; do sleep 1 @@ -46,6 +57,34 @@ done set -x echo \"${PASS_MSG}\" " + DOMU_CFG_EXTRA=' +vif = [ "bridge=xenbr0", ] +disk = [ ] +' +elif [ "${TEST}" = "argo" ] +then + PASS_MSG="TEST: Message from DOMU" + XEN_CMD_EXTRA="argo=1,mac-permissive=1" + DOMU_CMD=" +insmod /root/xen-argo.ko +until false +do + echo \"${PASS_MSG}\" + sleep 1 +done | argo-exec -p 28333 -d 0 -- /bin/echo +" + DOM0_CMD=" +insmod /root/xen-argo.ko +xl create /etc/xen/domU.cfg +argo-exec -l -p 28333 -- /bin/echo +" +copy_dom0_files () +{ + cp "${WORKDIR}/binaries/xen-argo.ko" "root/" + cp -ar "${WORKDIR}/binaries/lib/"* "usr/local/lib/" + cp "${WORKDIR}/binaries/argo-exec" "usr/local/bin/" +} +copy_domU_files () { copy_dom0_files; } else fatal "Unknown test: ${TEST}" fi @@ -54,18 +93,18 @@ fi mkdir -p rootfs cd rootfs tar xzf ../binaries/initrd.tar.gz -mkdir proc -mkdir run -mkdir srv -mkdir sys +mkdir proc run srv sys rm var/run echo "#!/bin/sh - +set -x +export LD_LIBRARY_PATH=/usr/local/lib +PATH=/usr/local/bin:/usr/local/sbin:\$PATH ${DOMU_CMD} " > etc/local.d/xen.start chmod +x etc/local.d/xen.start echo "rc_verbose=yes" >> etc/rc.conf sed -i -e 's/^Welcome/domU \0/' etc/issue +copy_domU_files find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz cd .. rm -rf rootfs @@ -74,37 +113,24 @@ rm -rf rootfs mkdir -p rootfs cd rootfs tar xzf ../binaries/initrd.tar.gz -mkdir boot -mkdir proc -mkdir run -mkdir srv -mkdir sys +mkdir boot proc run srv sys rm var/run cp -ar ../binaries/dist/install/* . echo "#!/bin/bash - +set -x export LD_LIBRARY_PATH=/usr/local/lib bash /etc/init.d/xencommons start - -brctl addbr xenbr0 -brctl addif xenbr0 eth0 -ifconfig eth0 up -ifconfig xenbr0 up -ifconfig xenbr0 192.168.0.1 - -# get domU console content into test log -tail -F /var/log/xen/console/guest-domU.log 2>/dev/null | sed -e \"s/^/(domU) /\" & -xl create /etc/xen/domU.cfg ${DOM0_CMD} " > etc/local.d/xen.start chmod +x etc/local.d/xen.start -echo "${DOMU_CFG}" > etc/xen/domU.cfg +echo "${DOMU_CFG}${DOMU_CFG_EXTRA}" > etc/xen/domU.cfg echo "rc_verbose=yes" >> etc/rc.conf echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons echo "QEMU_XEN=/bin/false" >> etc/default/xencommons mkdir -p var/log/xen/console cp ../binaries/bzImage boot/vmlinuz cp ../binaries/domU-rootfs.cpio.gz boot/initrd-domU +copy_dom0_files find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz cd .. @@ -116,7 +142,7 @@ cp -f binaries/bzImage ${TFTP}/${TEST_BOARD}/vmlinuz cp -f binaries/dom0-rootfs.cpio.gz ${TFTP}/${TEST_BOARD}/initrd-dom0 echo " net_default_server=10.0.6.1 -multiboot2 (tftp)/${TEST_BOARD}/xen ${XEN_CMDLINE} +multiboot2 (tftp)/${TEST_BOARD}/xen ${XEN_CMDLINE} sync_console module2 (tftp)/${TEST_BOARD}/vmlinuz console=hvc0 root=/dev/ram0 earlyprintk=xen module2 (tftp)/${TEST_BOARD}/initrd-dom0 boot -- 2.25.1 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [RFC XEN PATCH 2/2] automation: add xilinx test (linux argo) 2024-10-23 2:53 ` [RFC XEN PATCH 2/2] automation: add xilinx test (linux argo) victorm.lira @ 2024-10-23 5:03 ` Stefano Stabellini 0 siblings, 0 replies; 5+ messages in thread From: Stefano Stabellini @ 2024-10-23 5:03 UTC (permalink / raw) To: Victor Lira Cc: xen-devel, Andrew Cooper, Marek Marczykowski-Górecki, Doug Goldstein, Stefano Stabellini [-- Attachment #1: Type: text/plain, Size: 7276 bytes --] On Tue, 22 Oct 2024, victorm.lira@amd.com wrote: > From: Victor Lira <victorm.lira@amd.com> > > Add x86_64 hardware test that creates a Xen Argo communication > connection between two PVH domains. In the test, dom0 creates a domU and > listens for messages sent by the domU through Argo. > > To accomplish this, build Xen with CONFIG_ARGO=y. > > Update the xilinx x86_64 test script to support the new test, and add > "sync_console" to command line to avoid an issue with console messages > being lost. > > Signed-off-by: Victor Lira <victorm.lira@amd.com> > --- > Cc: Andrew Cooper <andrew.cooper3@citrix.com> > Cc: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > Cc: Doug Goldstein <cardoe@cardoe.com> > Cc: Stefano Stabellini <sstabellini@kernel.org> > Cc: xen-devel@lists.xenproject.org > --- > automation/gitlab-ci/build.yaml | 6 ++ > automation/gitlab-ci/test.yaml | 16 ++++ > .../scripts/xilinx-smoke-dom0-x86_64.sh | 76 +++++++++++++------ > 3 files changed, 73 insertions(+), 25 deletions(-) > > diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml > index c2db69ecf8..f425744f8b 100644 > --- a/automation/gitlab-ci/build.yaml > +++ b/automation/gitlab-ci/build.yaml > @@ -340,6 +340,12 @@ alpine-3.18-gcc-debug: > variables: > CONTAINER: alpine:3.18 > BUILD_QEMU_XEN: y > + EXTRA_XEN_CONFIG: | > + CONFIG_EXPERT=y > + CONFIG_UNSUPPORTED=y > + CONFIG_ARGO=y > > debian-bookworm-gcc-debug: > extends: .gcc-x86-64-build-debug > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > index b27c2be174..2d4709d393 100644 > --- a/automation/gitlab-ci/test.yaml > +++ b/automation/gitlab-ci/test.yaml > @@ -192,6 +192,22 @@ xilinx-smoke-dom0-x86_64-gcc-debug: > - *x86-64-test-needs > - alpine-3.18-gcc-debug > > +xilinx-smoke-dom0-x86_64-gcc-debug-argo: > + extends: .xilinx-x86_64 > + script: > + - ./automation/scripts/xilinx-smoke-dom0-x86_64.sh argo 2>&1 | tee ${LOGFILE} > + needs: > + - alpine-3.18-gcc-debug > + - project: xen-project/people/victormlira/test-artifacts This needs to be changed before commit Overall this looks great, thank you for exploring this option! Much better than before. This patch does some nice refactoring and also add the argo test. I would ask that you split it into two patches, the first one does the refactoring, the second one add the argo test > + job: x86_64-kernel-linux-6.6.56 > + ref: vml-tests-separate-artifacts > + - project: xen-project/people/victormlira/test-artifacts > + job: x86_64-rootfs-alpine-3.18 > + ref: vml-tests-separate-artifacts > + - project: xen-project/people/victormlira/test-artifacts > + job: x86_64-argo-linux-6.6.56 > + ref: vml-tests-separate-artifacts > + > adl-smoke-x86-64-gcc-debug: > extends: .adl-x86-64 > script: > diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh > index 7027f083ba..f70cfdc155 100755 > --- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh > +++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh > @@ -9,6 +9,8 @@ fatal() { > exit 1 > } > > +WORKDIR="${PWD}" > + > # Test parameter defaults. > TEST="$1" > PASS_MSG="Test passed: ${TEST}" > @@ -24,9 +26,10 @@ kernel = "/boot/vmlinuz" > ramdisk = "/boot/initrd-domU" > extra = "root=/dev/ram0 console=hvc0" > memory = 512 > -vif = [ "bridge=xenbr0", ] > -disk = [ ] > ' > +DOMU_CFG_EXTRA="" > +copy_domU_files () { :; } > +copy_dom0_files () { :; } > > # Select test variant. > if [ "${TEST}" = "ping" ]; then > @@ -39,6 +42,14 @@ done > echo \"${DOMU_MSG}\" > " > DOM0_CMD=" > +brctl addbr xenbr0 > +brctl addif xenbr0 eth0 > +ifconfig eth0 up > +ifconfig xenbr0 up > +ifconfig xenbr0 192.168.0.1 > +# get domU console content into test log > +tail -F /var/log/xen/console/guest-domU.log 2>/dev/null | sed -e \"s/^/(domU) /\" & > +xl create /etc/xen/domU.cfg This is something that should go in the refactoring patch > set +x > until grep -q \"${DOMU_MSG}\" /var/log/xen/console/guest-domU.log; do > sleep 1 > @@ -46,6 +57,34 @@ done > set -x > echo \"${PASS_MSG}\" > " > + DOMU_CFG_EXTRA=' > +vif = [ "bridge=xenbr0", ] > +disk = [ ] > +' > +elif [ "${TEST}" = "argo" ] > +then > + PASS_MSG="TEST: Message from DOMU" > + XEN_CMD_EXTRA="argo=1,mac-permissive=1" > + DOMU_CMD=" > +insmod /root/xen-argo.ko > +until false > +do > + echo \"${PASS_MSG}\" > + sleep 1 > +done | argo-exec -p 28333 -d 0 -- /bin/echo > +" > + DOM0_CMD=" > +insmod /root/xen-argo.ko > +xl create /etc/xen/domU.cfg > +argo-exec -l -p 28333 -- /bin/echo > +" > +copy_dom0_files () > +{ > + cp "${WORKDIR}/binaries/xen-argo.ko" "root/" > + cp -ar "${WORKDIR}/binaries/lib/"* "usr/local/lib/" > + cp "${WORKDIR}/binaries/argo-exec" "usr/local/bin/" > +} > +copy_domU_files () { copy_dom0_files; } > else > fatal "Unknown test: ${TEST}" > fi > @@ -54,18 +93,18 @@ fi > mkdir -p rootfs > cd rootfs > tar xzf ../binaries/initrd.tar.gz > -mkdir proc > -mkdir run > -mkdir srv > -mkdir sys > +mkdir proc run srv sys > rm var/run > echo "#!/bin/sh > - > +set -x > +export LD_LIBRARY_PATH=/usr/local/lib > +PATH=/usr/local/bin:/usr/local/sbin:\$PATH > ${DOMU_CMD} > " > etc/local.d/xen.start > chmod +x etc/local.d/xen.start > echo "rc_verbose=yes" >> etc/rc.conf > sed -i -e 's/^Welcome/domU \0/' etc/issue > +copy_domU_files > find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz > cd .. > rm -rf rootfs > @@ -74,37 +113,24 @@ rm -rf rootfs > mkdir -p rootfs > cd rootfs > tar xzf ../binaries/initrd.tar.gz > -mkdir boot > -mkdir proc > -mkdir run > -mkdir srv > -mkdir sys > +mkdir boot proc run srv sys > rm var/run > cp -ar ../binaries/dist/install/* . > echo "#!/bin/bash > - > +set -x > export LD_LIBRARY_PATH=/usr/local/lib > bash /etc/init.d/xencommons start > - > -brctl addbr xenbr0 > -brctl addif xenbr0 eth0 > -ifconfig eth0 up > -ifconfig xenbr0 up > -ifconfig xenbr0 192.168.0.1 > - > -# get domU console content into test log > -tail -F /var/log/xen/console/guest-domU.log 2>/dev/null | sed -e \"s/^/(domU) /\" & > -xl create /etc/xen/domU.cfg > ${DOM0_CMD} > " > etc/local.d/xen.start > chmod +x etc/local.d/xen.start > -echo "${DOMU_CFG}" > etc/xen/domU.cfg > +echo "${DOMU_CFG}${DOMU_CFG_EXTRA}" > etc/xen/domU.cfg > echo "rc_verbose=yes" >> etc/rc.conf > echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons > echo "QEMU_XEN=/bin/false" >> etc/default/xencommons > mkdir -p var/log/xen/console > cp ../binaries/bzImage boot/vmlinuz > cp ../binaries/domU-rootfs.cpio.gz boot/initrd-domU > +copy_dom0_files > find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz > cd .. > > @@ -116,7 +142,7 @@ cp -f binaries/bzImage ${TFTP}/${TEST_BOARD}/vmlinuz > cp -f binaries/dom0-rootfs.cpio.gz ${TFTP}/${TEST_BOARD}/initrd-dom0 > echo " > net_default_server=10.0.6.1 > -multiboot2 (tftp)/${TEST_BOARD}/xen ${XEN_CMDLINE} > +multiboot2 (tftp)/${TEST_BOARD}/xen ${XEN_CMDLINE} sync_console > module2 (tftp)/${TEST_BOARD}/vmlinuz console=hvc0 root=/dev/ram0 earlyprintk=xen > module2 (tftp)/${TEST_BOARD}/initrd-dom0 > boot > -- > 2.25.1 > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [RFC TEST_ARTIFACTS PATCH 1/2] artifacts: Initial commit 2024-10-23 2:53 ` [RFC TEST_ARTIFACTS PATCH 1/2] " victorm.lira 2024-10-23 2:53 ` [RFC XEN PATCH 2/2] automation: add xilinx test (linux argo) victorm.lira @ 2024-10-23 4:57 ` Stefano Stabellini 1 sibling, 0 replies; 5+ messages in thread From: Stefano Stabellini @ 2024-10-23 4:57 UTC (permalink / raw) To: Victor Lira Cc: xen-devel, Stefano Stabellini, Andrew Cooper, Marek Marczykowski-Górecki, Doug Goldstein [-- Attachment #1: Type: text/plain, Size: 10976 bytes --] On Tue, 22 Oct 2024, victorm.lira@amd.com wrote: > From: Stefano Stabellini <sstabellini@kernel.org> Hi Victor, I think this was a mistake, maybe due to the fact that I created the repository for you. The From field represent the vauthor and in this case it should be you. > Create new repository to produce common artifacts for Xen tests > > Requested-by: Andrew Cooper <andrew.cooper3@citrix.com> > Signed-off-by: Victor Lira <victorm.lira@amd.com> > --- > Cc: Andrew Cooper <andrew.cooper3@citrix.com> > Cc: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > Cc: Doug Goldstein <cardoe@cardoe.com> > Cc: Stefano Stabellini <sstabellini@kernel.org> > Cc: xen-devel@lists.xenproject.org > --- > .gitlab-ci.yml | 48 ++++++++++++++++++++ > README.md | 10 ++++ > binaries/.gitignore | 3 ++ > images/Makefile | 20 ++++++++ > images/alpine/x86_64-build.dockerfile | 25 ++++++++++ > images/alpine/x86_64-rootfs.dockerfile | 4 ++ > scripts/x86_64-argo-linux.sh | 63 ++++++++++++++++++++++++++ > scripts/x86_64-kernel-linux.sh | 31 +++++++++++++ > scripts/x86_64-rootfs-alpine.sh | 58 ++++++++++++++++++++++++ > 9 files changed, 262 insertions(+) > create mode 100644 .gitlab-ci.yml > create mode 100644 README.md > create mode 100644 binaries/.gitignore > create mode 100644 images/Makefile > create mode 100644 images/alpine/x86_64-build.dockerfile > create mode 100644 images/alpine/x86_64-rootfs.dockerfile > create mode 100755 scripts/x86_64-argo-linux.sh > create mode 100755 scripts/x86_64-kernel-linux.sh > create mode 100755 scripts/x86_64-rootfs-alpine.sh > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > new file mode 100644 > index 0000000..f2f7738 > --- /dev/null > +++ b/.gitlab-ci.yml > @@ -0,0 +1,48 @@ > +workflow: > + name: "xen test artifacts" > + > +.artifacts: > + stage: build > + image: > + name: registry.gitlab.com/xen-project/people/victormlira/test-artifacts/${CONTAINER} This of course needs to be changed before the series is committed > + artifacts: > + name: "${CI_JOB_NAME_SLUG}" > + paths: > + - binaries/ > + > +.x86_64-artifacts: > + extends: .artifacts > + image: > + docker: > + platform: linux/amd64 > + user: xenproject > + variables: > + CONTAINER: alpine:x86_64-build > + tags: > + - x86_64 > + > +x86_64-kernel-linux-6.6.56: > + extends: .x86_64-artifacts > + script: > + - . scripts/x86_64-kernel-linux.sh > + variables: > + LINUX_VERSION: 6.6.56 > + > +x86_64-rootfs-alpine-3.18: > + extends: .x86_64-artifacts > + variables: > + CONTAINER: alpine:x86_64-rootfs > + image: > + docker: > + user: root > + script: > + - . scripts/x86_64-rootfs-alpine.sh > + > +x86_64-argo-linux-6.6.56: > + extends: .x86_64-artifacts > + script: > + - . scripts/x86_64-argo-linux.sh > + variables: > + LINUX_VERSION: 6.6.56 > + ARGO_SHA: "705a7a8a624b42e13e655d3042059b8a85cdf6a3" > + ARGOEXEC_SHA: "d900429f6640acc6f68a3d3a4c945d7da60625d8" > diff --git a/README.md b/README.md > new file mode 100644 > index 0000000..0d72519 > --- /dev/null > +++ b/README.md > @@ -0,0 +1,10 @@ > +# Xen Project Test Artifacts > + > +## Overview > + > +This repository is used to produce common artifacts for Xen tests. > + > +## > + > + > +## > diff --git a/binaries/.gitignore b/binaries/.gitignore > new file mode 100644 > index 0000000..95e2a2e > --- /dev/null > +++ b/binaries/.gitignore > @@ -0,0 +1,3 @@ > +# Keep this directory around to be available in CI/CD jobs. > +* > +!.gitignore > diff --git a/images/Makefile b/images/Makefile > new file mode 100644 > index 0000000..41638d7 > --- /dev/null > +++ b/images/Makefile > @@ -0,0 +1,20 @@ > + > +# The base of where these containers will appear > +REGISTRY := registry.gitlab.com/xen-project/people/victormlira/test-artifacts also this > +CONTAINERS = $(subst .dockerfile,,$(wildcard */*.dockerfile)) > + > +help: > + @echo "Containers to build test artifacts." > + @echo "To build one run 'make DISTRO/TAG'." > + @echo "Available containers:" > + @$(foreach file,$(sort $(CONTAINERS)),echo ${file};) > + @echo "To push container builds, set the environment variable PUSH" > + > +%: %.dockerfile ## Builds containers > + docker build --pull -t $(REGISTRY)/$(@D):$(@F) -f $< $(<D) > + @if [ ! -z $${PUSH+x} ]; then \ > + docker push $(REGISTRY)/$(@D):$(@F); \ > + fi > + > +.PHONY: all > +all: $(CONTAINERS) > diff --git a/images/alpine/x86_64-build.dockerfile b/images/alpine/x86_64-build.dockerfile > new file mode 100644 > index 0000000..374952f > --- /dev/null > +++ b/images/alpine/x86_64-build.dockerfile > @@ -0,0 +1,25 @@ > +# syntax=docker/dockerfile:1 > +FROM --platform=linux/amd64 alpine:3.18 > +LABEL maintainer.name="The Xen Project" \ > + maintainer.email="xen-devel@lists.xenproject.org" > + > +WORKDIR /build/ > + > +RUN <<EOF > +adduser -D xenproject > +apk --no-cache add \ > + musl-dev \ > + build-base \ > + libc6-compat \ > + linux-headers \ > + bash \ > + git \ > + curl \ > + flex \ > + bison \ > + elfutils-dev \ > + autoconf \ > + automake \ > + libtool \ > + openssl-dev Use the DEPS variable like here to make it a bit more readable: automation/build/fedora/40-x86_64.dockerfile > +EOF > diff --git a/images/alpine/x86_64-rootfs.dockerfile b/images/alpine/x86_64-rootfs.dockerfile > new file mode 100644 > index 0000000..b912e9c > --- /dev/null > +++ b/images/alpine/x86_64-rootfs.dockerfile > @@ -0,0 +1,4 @@ > +# syntax=docker/dockerfile:1 > +FROM --platform=linux/amd64 alpine:3.18 > +LABEL maintainer.name="The Xen Project" \ > + maintainer.email="xen-devel@lists.xenproject.org" > diff --git a/scripts/x86_64-argo-linux.sh b/scripts/x86_64-argo-linux.sh > new file mode 100755 > index 0000000..a110a33 > --- /dev/null > +++ b/scripts/x86_64-argo-linux.sh > @@ -0,0 +1,63 @@ > +#!/usr/bin/env bash > + > +if test -z "${LINUX_VERSION}" > +then > + >&2 echo "LINUX_VERSION must be set"; exit 1 > +fi > + > +set -ex -o pipefail > + > +BUILDDIR="${PWD}" > +COPYDIR="${BUILDDIR}/binaries/" > + > +# Prepare Linux sources > +curl -fsSLO \ > + https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-"${LINUX_VERSION}".tar.xz > +tar xJf linux-"${LINUX_VERSION}".tar.xz > +cd linux-"${LINUX_VERSION}" > +make ARCH=x86 defconfig > +make ARCH=x86 xen.config > +./scripts/config --enable BRIDGE > +./scripts/config --enable IGC > +./scripts/config --enable TUN > +cp .config .config.orig > +cat .config.orig \ > + | grep 'XEN' \ > + | grep '=m' \ > + | sed 's/=m/=y/g' \ > + >> .config > +make ARCH=x86 olddefconfig > +make ARCH=x86 modules_prepare > + > +# Build Linux kernel module for Xen Argo > +cd "${BUILDDIR}" > +git clone \ > + --depth=1 --branch=master \ > + https://github.com/OpenXT/linux-xen-argo.git > +git -C "${BUILDDIR}/linux-xen-argo" switch --detach "${ARGO_SHA}" > +make -C "linux-${LINUX_VERSION}" M="${BUILDDIR}/linux-xen-argo/argo-linux" \ > + CFLAGS_MODULE="-Wno-error" KBUILD_MODPOST_WARN=1 > +cp "linux-xen-argo/argo-linux/xen-argo.ko" "${COPYDIR}/xen-argo.ko" > + > +# Build Linux libargo shared library, applying fixes to build in Alpine Linux > +cd "${BUILDDIR}/linux-xen-argo/libargo" > +sed -i "s|AM_INIT_AUTOMAKE|AC_CONFIG_AUX_DIR(.)\nAM_INIT_AUTOMAKE|" configure.ac > +sed -i "s/__SOCKADDR_COMMON (sxenargo_)/sa_family_t sxenargo_family/" src/libargo.h > +sed -i "s/__SOCKADDR_COMMON_SIZE/(sizeof (unsigned short int))/" src/libargo.h > +autoreconf --install > +./configure --prefix="${COPYDIR}" CPPFLAGS="-I${PWD}/../argo-linux/include" > +make > +make install > + > +# Build Linux user program, modifying for xilinx argo test > +cd "${BUILDDIR}" > +wget "https://raw.githubusercontent.com/OpenXT/xenclient-oe/${ARGOEXEC_SHA}/\ > +recipes-openxt/argo-exec/argo-exec/argo-exec.c" > +sed -i "s|#include <xen/xen.h>||" argo-exec.c > +sed -i "s|ret = shuffle(s, fds\[0\], fds\[1\]);|ret = shuffle(s, 0, 1);|" \ > + argo-exec.c > +gcc -I"${BUILDDIR}/linux-xen-argo/libargo/src" \ > + -I"${BUILDDIR}/linux-xen-argo/argo-linux/include" \ > + -L"${COPYDIR}/lib/" \ > + -o argo-exec argo-exec.c -largo > +cp argo-exec "${COPYDIR}" > diff --git a/scripts/x86_64-kernel-linux.sh b/scripts/x86_64-kernel-linux.sh > new file mode 100755 > index 0000000..17be36b > --- /dev/null > +++ b/scripts/x86_64-kernel-linux.sh > @@ -0,0 +1,31 @@ > +#!/usr/bin/env bash > + > +if test -z "${LINUX_VERSION}" > +then > + >&2 echo "LINUX_VERSION must be set"; exit 1 > +fi > + > +set -ex -o pipefail > + > +WORKDIR="${PWD}" > +COPYDIR="${WORKDIR}/binaries/" > + > +# Build Linux > +curl -fsSLO \ > + https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-"${LINUX_VERSION}".tar.xz > +tar xJf linux-"${LINUX_VERSION}".tar.xz > +cd linux-"${LINUX_VERSION}" > +make ARCH=x86 defconfig > +make ARCH=x86 xen.config > +./scripts/config --enable BRIDGE > +./scripts/config --enable IGC > +./scripts/config --enable TUN > +cp .config .config.orig > +cat .config.orig \ > + | grep 'XEN' \ > + | grep '=m' \ > + | sed 's/=m/=y/g' \ > + >> .config > +make ARCH=x86 olddefconfig > +make ARCH=x86 "-j$(nproc)" --quiet > +cp arch/x86/boot/bzImage "${COPYDIR}" > diff --git a/scripts/x86_64-rootfs-alpine.sh b/scripts/x86_64-rootfs-alpine.sh > new file mode 100755 > index 0000000..28d8e30 > --- /dev/null > +++ b/scripts/x86_64-rootfs-alpine.sh > @@ -0,0 +1,58 @@ > +WORKDIR="${PWD}" > + > +apk update > + > +# xen runtime deps > +apk add musl > +apk add libgcc > +apk add openrc > +apk add busybox > +apk add sudo > +apk add dbus > +apk add bash > +apk add python3 > +apk add zlib > +apk add ncurses > +apk add yajl > +apk add libaio > +apk add xz > +apk add util-linux > +apk add argp-standalone > +apk add libfdt > +apk add glib > +apk add pixman > +apk add curl > +apk add udev > +apk add pciutils > +apk add libelf > + > +# Xen > +cd / > +# Minimal ramdisk environment in case of cpio output > +rc-update add udev > +rc-update add udev-trigger > +rc-update add udev-settle > +rc-update add loopback sysinit > +rc-update add bootmisc boot > +rc-update add devfs sysinit > +rc-update add dmesg sysinit > +rc-update add hostname boot > +rc-update add hwclock boot > +rc-update add hwdrivers sysinit > +rc-update add modules boot > +rc-update add killprocs shutdown > +rc-update add mount-ro shutdown > +rc-update add savecache shutdown > +rc-update add local default > +cp -a /sbin/init /init > +echo "ttyS0" >> /etc/securetty > +echo "hvc0" >> /etc/securetty > +echo "ttyS0::respawn:/sbin/getty -L ttyS0 115200 vt100" >> /etc/inittab > +echo "hvc0::respawn:/sbin/getty -L hvc0 115200 vt100" >> /etc/inittab > +echo > /etc/modules > +passwd -d "root" root > + > +# Create rootfs > +cd / > +tar cvzf "${WORKDIR}/binaries/initrd.tar.gz" \ > + bin dev etc home init lib mnt opt root sbin usr var > -- > 2.25.1 > ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-10-23 5:03 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-10-23 2:53 [RFC TEST_ARTIFACTS PATCH 0/2] artifacts: Initial commit victorm.lira 2024-10-23 2:53 ` [RFC TEST_ARTIFACTS PATCH 1/2] " victorm.lira 2024-10-23 2:53 ` [RFC XEN PATCH 2/2] automation: add xilinx test (linux argo) victorm.lira 2024-10-23 5:03 ` Stefano Stabellini 2024-10-23 4:57 ` [RFC TEST_ARTIFACTS PATCH 1/2] artifacts: Initial commit Stefano Stabellini
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.