* [PATCH 1/2] automation: preserve built xen.efi @ 2024-10-02 12:42 Marek Marczykowski-Górecki 2024-10-02 12:42 ` [PATCH 2/2] automation: add a smoke test for xen.efi on X86 Marek Marczykowski-Górecki 2024-10-02 20:42 ` [PATCH 1/2] automation: preserve built xen.efi Andrew Cooper 0 siblings, 2 replies; 14+ messages in thread From: Marek Marczykowski-Górecki @ 2024-10-02 12:42 UTC (permalink / raw) To: xen-devel Cc: Marek Marczykowski-Górecki, Doug Goldstein, Stefano Stabellini It will be useful for further tests. Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> --- automation/scripts/build | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/automation/scripts/build b/automation/scripts/build index b3c71fb6fb60..4cd41cb2c471 100755 --- a/automation/scripts/build +++ b/automation/scripts/build @@ -47,6 +47,7 @@ if [[ "${CPPCHECK}" == "y" ]] && [[ "${HYPERVISOR_ONLY}" == "y" ]]; then # Preserve artefacts cp xen/xen binaries/xen + if [[ -f xen/xen.efi ]]; then cp xen/xen.efi binaries/xen.efi; fi cp xen/cppcheck-report/xen-cppcheck.txt xen-cppcheck.txt elif [[ "${HYPERVISOR_ONLY}" == "y" ]]; then # Xen-only build @@ -54,6 +55,7 @@ elif [[ "${HYPERVISOR_ONLY}" == "y" ]]; then # Preserve artefacts cp xen/xen binaries/xen + if [[ -f xen/xen.efi ]]; then cp xen/xen.efi binaries/xen.efi; fi else # Full build. Figure out our ./configure options cfgargs=() @@ -101,5 +103,8 @@ else # even though dist/ contains everything, while some containers don't even # build Xen cp -r dist binaries/ - if [[ -f xen/xen ]] ; then cp xen/xen binaries/xen; fi + if [[ -f xen/xen ]] ; then + cp xen/xen binaries/xen + if [[ -f xen/xen.efi ]]; then cp xen/xen.efi binaries/xen.efi; fi + fi fi -- 2.46.0 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 2/2] automation: add a smoke test for xen.efi on X86 2024-10-02 12:42 [PATCH 1/2] automation: preserve built xen.efi Marek Marczykowski-Górecki @ 2024-10-02 12:42 ` Marek Marczykowski-Górecki 2024-10-02 21:03 ` Andrew Cooper 2024-10-02 22:16 ` Stefano Stabellini 2024-10-02 20:42 ` [PATCH 1/2] automation: preserve built xen.efi Andrew Cooper 1 sibling, 2 replies; 14+ messages in thread From: Marek Marczykowski-Górecki @ 2024-10-02 12:42 UTC (permalink / raw) To: xen-devel Cc: Marek Marczykowski-Górecki, Doug Goldstein, Stefano Stabellini Check if xen.efi is bootable with an XTF dom0. The TEST_TIMEOUT is set in the script to override project-global value. Setting it in the gitlab yaml file doesn't work, as it's too low priority (https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence). The multiboot2+EFI path is tested on hardware tests already. Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> --- This requires rebuilding debian:bookworm container. The TEST_TIMEOUT issue mentioned above applies to xilix-* jobs too. It's not clear to me why the default TEST_TIMEOUT is set at the group level instead of in the yaml file, so I'm not adjusting the other places. --- automation/build/debian/bookworm.dockerfile | 1 + automation/gitlab-ci/test.yaml | 7 ++++ automation/scripts/qemu-smoke-x86-64-efi.sh | 44 +++++++++++++++++++++ 3 files changed, 52 insertions(+) create mode 100755 automation/scripts/qemu-smoke-x86-64-efi.sh diff --git a/automation/build/debian/bookworm.dockerfile b/automation/build/debian/bookworm.dockerfile index 3dd70cb6b2e3..061114ba522d 100644 --- a/automation/build/debian/bookworm.dockerfile +++ b/automation/build/debian/bookworm.dockerfile @@ -46,6 +46,7 @@ RUN apt-get update && \ # for test phase, qemu-smoke-* jobs qemu-system-x86 \ expect \ + ovmf \ # for test phase, qemu-alpine-* jobs cpio \ busybox-static \ diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml index 8675016b6a37..74fd3f3109ae 100644 --- a/automation/gitlab-ci/test.yaml +++ b/automation/gitlab-ci/test.yaml @@ -463,6 +463,13 @@ qemu-smoke-x86-64-clang-pvh: needs: - debian-bookworm-clang-debug +qemu-smoke-x86-64-gcc-efi: + extends: .qemu-x86-64 + script: + - ./automation/scripts/qemu-smoke-x86-64-efi.sh pv 2>&1 | tee ${LOGFILE} + needs: + - debian-bookworm-gcc-debug + qemu-smoke-riscv64-gcc: extends: .qemu-riscv64 script: diff --git a/automation/scripts/qemu-smoke-x86-64-efi.sh b/automation/scripts/qemu-smoke-x86-64-efi.sh new file mode 100755 index 000000000000..e053cfa995ba --- /dev/null +++ b/automation/scripts/qemu-smoke-x86-64-efi.sh @@ -0,0 +1,44 @@ +#!/bin/bash + +set -ex -o pipefail + +# variant should be either pv or pvh +variant=$1 + +# Clone and build XTF +git clone https://xenbits.xen.org/git-http/xtf.git +cd xtf && make -j$(nproc) && cd - + +case $variant in + pvh) k=test-hvm64-example extra="dom0-iommu=none dom0=pvh" ;; + *) k=test-pv64-example extra= ;; +esac + +mkdir -p boot-esp/EFI/BOOT +cp binaries/xen.efi boot-esp/EFI/BOOT/BOOTX64.EFI +cp xtf/tests/example/$k boot-esp/EFI/BOOT/kernel + +cat > boot-esp/EFI/BOOT/BOOTX64.cfg <<EOF +[global] +default=test + +[test] +options=loglvl=all console=com1 noreboot console_timestamps=boot $extra +kernel=kernel +EOF + +cp /usr/share/OVMF/OVMF_CODE.fd OVMF_CODE.fd +cp /usr/share/OVMF/OVMF_VARS.fd OVMF_VARS.fd + +rm -f smoke.serial +export TEST_CMD="qemu-system-x86_64 -nographic -M q35,kernel-irqchip=split \ + -drive if=pflash,format=raw,readonly=on,file=OVMF_CODE.fd \ + -drive if=pflash,format=raw,file=OVMF_VARS.fd \ + -drive file=fat:rw:boot-esp,media=disk,index=0,format=raw \ + -m 512 -monitor none -serial stdio" + +export TEST_LOG="smoke.serial" +export PASSED="Test result: SUCCESS" +export TEST_TIMEOUT=120 + +./automation/scripts/console.exp | sed 's/\r\+$//' -- 2.46.0 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] automation: add a smoke test for xen.efi on X86 2024-10-02 12:42 ` [PATCH 2/2] automation: add a smoke test for xen.efi on X86 Marek Marczykowski-Górecki @ 2024-10-02 21:03 ` Andrew Cooper 2024-10-02 22:16 ` Stefano Stabellini 1 sibling, 0 replies; 14+ messages in thread From: Andrew Cooper @ 2024-10-02 21:03 UTC (permalink / raw) To: Marek Marczykowski-Górecki, xen-devel Cc: Doug Goldstein, Stefano Stabellini On 02/10/2024 1:42 pm, Marek Marczykowski-Górecki wrote: > Check if xen.efi is bootable with an XTF dom0. > > The TEST_TIMEOUT is set in the script to override project-global value. > Setting it in the gitlab yaml file doesn't work, as it's too low > priority > (https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence). > > The multiboot2+EFI path is tested on hardware tests already. > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > --- > This requires rebuilding debian:bookworm container. Noted. > > The TEST_TIMEOUT issue mentioned above applies to xilix-* jobs too. It's > not clear to me why the default TEST_TIMEOUT is set at the group level > instead of in the yaml file, so I'm not adjusting the other places. I'll leave that side of things to Stefano. > --- > automation/build/debian/bookworm.dockerfile | 1 + > automation/gitlab-ci/test.yaml | 7 ++++ > automation/scripts/qemu-smoke-x86-64-efi.sh | 44 +++++++++++++++++++++ > 3 files changed, 52 insertions(+) > create mode 100755 automation/scripts/qemu-smoke-x86-64-efi.sh > > diff --git a/automation/build/debian/bookworm.dockerfile b/automation/build/debian/bookworm.dockerfile > index 3dd70cb6b2e3..061114ba522d 100644 > --- a/automation/build/debian/bookworm.dockerfile > +++ b/automation/build/debian/bookworm.dockerfile > @@ -46,6 +46,7 @@ RUN apt-get update && \ > # for test phase, qemu-smoke-* jobs > qemu-system-x86 \ > expect \ # for test phase efi jobs > + ovmf \ > # for test phase, qemu-alpine-* jobs > cpio \ > busybox-static \ > diff --git a/automation/scripts/qemu-smoke-x86-64-efi.sh b/automation/scripts/qemu-smoke-x86-64-efi.sh > new file mode 100755 > index 000000000000..e053cfa995ba > --- /dev/null > +++ b/automation/scripts/qemu-smoke-x86-64-efi.sh > @@ -0,0 +1,44 @@ > +#!/bin/bash > + > +set -ex -o pipefail > + > +# variant should be either pv or pvh > +variant=$1 > + > +# Clone and build XTF > +git clone https://xenbits.xen.org/git-http/xtf.git > +cd xtf && make -j$(nproc) && cd - make -C xtf -j$(nproc) I still haven't got XTF nicely working in Gitlab CI, but if my plans work out, we will be able to replace this with a job-level artefact import. > + > +case $variant in > + pvh) k=test-hvm64-example extra="dom0-iommu=none dom0=pvh" ;; > + *) k=test-pv64-example extra= ;; > +esac > + > +mkdir -p boot-esp/EFI/BOOT > +cp binaries/xen.efi boot-esp/EFI/BOOT/BOOTX64.EFI > +cp xtf/tests/example/$k boot-esp/EFI/BOOT/kernel Looking at this split, I'd suggest having k=example/test-... in the case, and cp xtf/tests/$k boot-... here. That way, it's easier to swap out e.g. k=selftest/test-pv64-selftest > + > +cat > boot-esp/EFI/BOOT/BOOTX64.cfg <<EOF > +[global] > +default=test > + > +[test] > +options=loglvl=all console=com1 noreboot console_timestamps=boot $extra Bah - we still can't select the timestamp mode at Kconfig time, can we. Because that would have been helpful... Something for the todo list. Individual tests should not be needing to arrange this. I think you can drop the loglvl=all. That (and guest_loglvl, and CONFIG_VERBOSE which you depend on) is already arranged by virtue of using a debug Xen. I don't think this is liable to change. > +kernel=kernel > +EOF > + > +cp /usr/share/OVMF/OVMF_CODE.fd OVMF_CODE.fd > +cp /usr/share/OVMF/OVMF_VARS.fd OVMF_VARS.fd > + > +rm -f smoke.serial > +export TEST_CMD="qemu-system-x86_64 -nographic -M q35,kernel-irqchip=split \ > + -drive if=pflash,format=raw,readonly=on,file=OVMF_CODE.fd \ > + -drive if=pflash,format=raw,file=OVMF_VARS.fd \ Why do you copy OVMF_*.fd out of /usr/share to the local dir, then pass them as paths here? Can't qemu be pointed at /usr/share directly? ~Andrew ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] automation: add a smoke test for xen.efi on X86 2024-10-02 12:42 ` [PATCH 2/2] automation: add a smoke test for xen.efi on X86 Marek Marczykowski-Górecki 2024-10-02 21:03 ` Andrew Cooper @ 2024-10-02 22:16 ` Stefano Stabellini 2024-10-02 22:22 ` Stefano Stabellini 1 sibling, 1 reply; 14+ messages in thread From: Stefano Stabellini @ 2024-10-02 22:16 UTC (permalink / raw) To: Marek Marczykowski-Górecki Cc: xen-devel, Doug Goldstein, Stefano Stabellini [-- Attachment #1: Type: text/plain, Size: 5071 bytes --] On Wed, 2 Oct 2024, Marek Marczykowski-Górecki wrote: > Check if xen.efi is bootable with an XTF dom0. > > The TEST_TIMEOUT is set in the script to override project-global value. > Setting it in the gitlab yaml file doesn't work, as it's too low > priority > (https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence). > > The multiboot2+EFI path is tested on hardware tests already. > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > --- > This requires rebuilding debian:bookworm container. > > The TEST_TIMEOUT issue mentioned above applies to xilix-* jobs too. It's > not clear to me why the default TEST_TIMEOUT is set at the group level > instead of in the yaml file, so I'm not adjusting the other places. Let me premise that now that we use "expect" all successful tests will terminate as soon as the success condition is met, without waiting for the test timeout to expire. There is a CI/CD variable called TEST_TIMEOUT set at the gitlab.com/xen-project level. (There is also a check in console.exp in case TEST_TIMEOUT is not set so that we don't run into problems in case the CI/CD variable is removed accidentally.) The global TEST_TIMEOUT is meant to be a high value to account for slow QEMU tests running potentially on our slowest cloud runners. However, for hardware-based tests such as the xilinx-* jobs, we know that the timeout is supposed to be less than that. The test is running on real hardware which is considerably faster than QEMU running on our slowest runners. Basically, the timeout depends on the runner more than the test. So we override the TEST_TIMEOUT variable for the xilinx-* jobs providing a lower timeout value. The global TEST_TIMEOUT is set to 1500. The xilinx-* timeout is set to 120 for ARM and 1000 for x86. You are welcome to override the TEST_TIMEOUT value for the hardware-based QubesOS tests. At the same time, given that on success the timeout is not really used, it is also OK to leave it like this. > --- > automation/build/debian/bookworm.dockerfile | 1 + > automation/gitlab-ci/test.yaml | 7 ++++ > automation/scripts/qemu-smoke-x86-64-efi.sh | 44 +++++++++++++++++++++ > 3 files changed, 52 insertions(+) > create mode 100755 automation/scripts/qemu-smoke-x86-64-efi.sh > > diff --git a/automation/build/debian/bookworm.dockerfile b/automation/build/debian/bookworm.dockerfile > index 3dd70cb6b2e3..061114ba522d 100644 > --- a/automation/build/debian/bookworm.dockerfile > +++ b/automation/build/debian/bookworm.dockerfile > @@ -46,6 +46,7 @@ RUN apt-get update && \ > # for test phase, qemu-smoke-* jobs > qemu-system-x86 \ > expect \ > + ovmf \ > # for test phase, qemu-alpine-* jobs > cpio \ > busybox-static \ > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > index 8675016b6a37..74fd3f3109ae 100644 > --- a/automation/gitlab-ci/test.yaml > +++ b/automation/gitlab-ci/test.yaml > @@ -463,6 +463,13 @@ qemu-smoke-x86-64-clang-pvh: > needs: > - debian-bookworm-clang-debug > > +qemu-smoke-x86-64-gcc-efi: > + extends: .qemu-x86-64 > + script: > + - ./automation/scripts/qemu-smoke-x86-64-efi.sh pv 2>&1 | tee ${LOGFILE} > + needs: > + - debian-bookworm-gcc-debug Given that the script you wrote (thank you!) can also handle pvh, can we directly add a pvh job to test.yaml too? > qemu-smoke-riscv64-gcc: > extends: .qemu-riscv64 > script: > diff --git a/automation/scripts/qemu-smoke-x86-64-efi.sh b/automation/scripts/qemu-smoke-x86-64-efi.sh > new file mode 100755 > index 000000000000..e053cfa995ba > --- /dev/null > +++ b/automation/scripts/qemu-smoke-x86-64-efi.sh > @@ -0,0 +1,44 @@ > +#!/bin/bash > + > +set -ex -o pipefail > + > +# variant should be either pv or pvh > +variant=$1 > + > +# Clone and build XTF > +git clone https://xenbits.xen.org/git-http/xtf.git > +cd xtf && make -j$(nproc) && cd - > + > +case $variant in > + pvh) k=test-hvm64-example extra="dom0-iommu=none dom0=pvh" ;; > + *) k=test-pv64-example extra= ;; > +esac > + > +mkdir -p boot-esp/EFI/BOOT > +cp binaries/xen.efi boot-esp/EFI/BOOT/BOOTX64.EFI > +cp xtf/tests/example/$k boot-esp/EFI/BOOT/kernel > + > +cat > boot-esp/EFI/BOOT/BOOTX64.cfg <<EOF > +[global] > +default=test > + > +[test] > +options=loglvl=all console=com1 noreboot console_timestamps=boot $extra > +kernel=kernel > +EOF > + > +cp /usr/share/OVMF/OVMF_CODE.fd OVMF_CODE.fd > +cp /usr/share/OVMF/OVMF_VARS.fd OVMF_VARS.fd > + > +rm -f smoke.serial > +export TEST_CMD="qemu-system-x86_64 -nographic -M q35,kernel-irqchip=split \ > + -drive if=pflash,format=raw,readonly=on,file=OVMF_CODE.fd \ > + -drive if=pflash,format=raw,file=OVMF_VARS.fd \ > + -drive file=fat:rw:boot-esp,media=disk,index=0,format=raw \ > + -m 512 -monitor none -serial stdio" > + > +export TEST_LOG="smoke.serial" > +export PASSED="Test result: SUCCESS" > +export TEST_TIMEOUT=120 > + > +./automation/scripts/console.exp | sed 's/\r\+$//' > -- > 2.46.0 > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] automation: add a smoke test for xen.efi on X86 2024-10-02 22:16 ` Stefano Stabellini @ 2024-10-02 22:22 ` Stefano Stabellini 2024-10-02 22:55 ` Marek Marczykowski-Górecki 0 siblings, 1 reply; 14+ messages in thread From: Stefano Stabellini @ 2024-10-02 22:22 UTC (permalink / raw) To: Stefano Stabellini Cc: Marek Marczykowski-Górecki, xen-devel, Doug Goldstein [-- Attachment #1: Type: text/plain, Size: 6102 bytes --] I forgot to reply to one important part below On Wed, 2 Oct 2024, Stefano Stabellini wrote: > On Wed, 2 Oct 2024, Marek Marczykowski-Górecki wrote: > > Check if xen.efi is bootable with an XTF dom0. > > > > The TEST_TIMEOUT is set in the script to override project-global value. > > Setting it in the gitlab yaml file doesn't work, as it's too low > > priority > > (https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence). > > > > The multiboot2+EFI path is tested on hardware tests already. > > > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > > --- > > This requires rebuilding debian:bookworm container. > > > > The TEST_TIMEOUT issue mentioned above applies to xilix-* jobs too. It's > > not clear to me why the default TEST_TIMEOUT is set at the group level > > instead of in the yaml file, so I'm not adjusting the other places. > > Let me premise that now that we use "expect" all successful tests will > terminate as soon as the success condition is met, without waiting for > the test timeout to expire. > > There is a CI/CD variable called TEST_TIMEOUT set at the > gitlab.com/xen-project level. (There is also a check in console.exp in > case TEST_TIMEOUT is not set so that we don't run into problems in case > the CI/CD variable is removed accidentally.) The global TEST_TIMEOUT is > meant to be a high value to account for slow QEMU tests running > potentially on our slowest cloud runners. > > However, for hardware-based tests such as the xilinx-* jobs, we know > that the timeout is supposed to be less than that. The test is running > on real hardware which is considerably faster than QEMU running on our > slowest runners. Basically, the timeout depends on the runner more than > the test. So we override the TEST_TIMEOUT variable for the xilinx-* jobs > providing a lower timeout value. > > The global TEST_TIMEOUT is set to 1500. > The xilinx-* timeout is set to 120 for ARM and 1000 for x86. > > You are welcome to override the TEST_TIMEOUT value for the > hardware-based QubesOS tests. At the same time, given that on success > the timeout is not really used, it is also OK to leave it like this. > > --- > > automation/build/debian/bookworm.dockerfile | 1 + > > automation/gitlab-ci/test.yaml | 7 ++++ > > automation/scripts/qemu-smoke-x86-64-efi.sh | 44 +++++++++++++++++++++ > > 3 files changed, 52 insertions(+) > > create mode 100755 automation/scripts/qemu-smoke-x86-64-efi.sh > > > > diff --git a/automation/build/debian/bookworm.dockerfile b/automation/build/debian/bookworm.dockerfile > > index 3dd70cb6b2e3..061114ba522d 100644 > > --- a/automation/build/debian/bookworm.dockerfile > > +++ b/automation/build/debian/bookworm.dockerfile > > @@ -46,6 +46,7 @@ RUN apt-get update && \ > > # for test phase, qemu-smoke-* jobs > > qemu-system-x86 \ > > expect \ > > + ovmf \ > > # for test phase, qemu-alpine-* jobs > > cpio \ > > busybox-static \ > > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > > index 8675016b6a37..74fd3f3109ae 100644 > > --- a/automation/gitlab-ci/test.yaml > > +++ b/automation/gitlab-ci/test.yaml > > @@ -463,6 +463,13 @@ qemu-smoke-x86-64-clang-pvh: > > needs: > > - debian-bookworm-clang-debug > > > > +qemu-smoke-x86-64-gcc-efi: > > + extends: .qemu-x86-64 > > + script: > > + - ./automation/scripts/qemu-smoke-x86-64-efi.sh pv 2>&1 | tee ${LOGFILE} > > + needs: > > + - debian-bookworm-gcc-debug > > Given that the script you wrote (thank you!) can also handle pvh, can we > directly add a pvh job to test.yaml too? > > > > qemu-smoke-riscv64-gcc: > > extends: .qemu-riscv64 > > script: > > diff --git a/automation/scripts/qemu-smoke-x86-64-efi.sh b/automation/scripts/qemu-smoke-x86-64-efi.sh > > new file mode 100755 > > index 000000000000..e053cfa995ba > > --- /dev/null > > +++ b/automation/scripts/qemu-smoke-x86-64-efi.sh > > @@ -0,0 +1,44 @@ > > +#!/bin/bash > > + > > +set -ex -o pipefail > > + > > +# variant should be either pv or pvh > > +variant=$1 > > + > > +# Clone and build XTF > > +git clone https://xenbits.xen.org/git-http/xtf.git > > +cd xtf && make -j$(nproc) && cd - > > + > > +case $variant in > > + pvh) k=test-hvm64-example extra="dom0-iommu=none dom0=pvh" ;; > > + *) k=test-pv64-example extra= ;; > > +esac > > + > > +mkdir -p boot-esp/EFI/BOOT > > +cp binaries/xen.efi boot-esp/EFI/BOOT/BOOTX64.EFI > > +cp xtf/tests/example/$k boot-esp/EFI/BOOT/kernel > > + > > +cat > boot-esp/EFI/BOOT/BOOTX64.cfg <<EOF > > +[global] > > +default=test > > + > > +[test] > > +options=loglvl=all console=com1 noreboot console_timestamps=boot $extra > > +kernel=kernel > > +EOF > > + > > +cp /usr/share/OVMF/OVMF_CODE.fd OVMF_CODE.fd > > +cp /usr/share/OVMF/OVMF_VARS.fd OVMF_VARS.fd > > + > > +rm -f smoke.serial > > +export TEST_CMD="qemu-system-x86_64 -nographic -M q35,kernel-irqchip=split \ > > + -drive if=pflash,format=raw,readonly=on,file=OVMF_CODE.fd \ > > + -drive if=pflash,format=raw,file=OVMF_VARS.fd \ > > + -drive file=fat:rw:boot-esp,media=disk,index=0,format=raw \ > > + -m 512 -monitor none -serial stdio" > > + > > +export TEST_LOG="smoke.serial" > > +export PASSED="Test result: SUCCESS" > > +export TEST_TIMEOUT=120 Although this works, I would prefer keeping the TEST_TIMEOUT overrides in test.yaml for consistency. However, it might be better not to override it (or to override to a higher timeout value), as successful tests will terminate immediately anyway. We need to be cautious about setting TEST_TIMEOUT values too low, as using a slow runner (like a small, busy cloud instance) can lead to false positive failures. This issue occurred frequently with ARM tests when we temporarily moved from a fast ARM server to slower ARM cloud instances a couple of months ago. On the other hand, adjusting TEST_TIMEOUT for non-QEMU hardware-based tests is acceptable since those tests rely on real hardware availability, which is unlikely to become suddenly slower. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] automation: add a smoke test for xen.efi on X86 2024-10-02 22:22 ` Stefano Stabellini @ 2024-10-02 22:55 ` Marek Marczykowski-Górecki 2024-10-02 23:08 ` Andrew Cooper 2024-10-02 23:30 ` Stefano Stabellini 0 siblings, 2 replies; 14+ messages in thread From: Marek Marczykowski-Górecki @ 2024-10-02 22:55 UTC (permalink / raw) To: Stefano Stabellini; +Cc: xen-devel, Doug Goldstein [-- Attachment #1: Type: text/plain, Size: 6959 bytes --] On Wed, Oct 02, 2024 at 03:22:59PM -0700, Stefano Stabellini wrote: > I forgot to reply to one important part below > > > On Wed, 2 Oct 2024, Stefano Stabellini wrote: > > On Wed, 2 Oct 2024, Marek Marczykowski-Górecki wrote: > > > Check if xen.efi is bootable with an XTF dom0. > > > > > > The TEST_TIMEOUT is set in the script to override project-global value. > > > Setting it in the gitlab yaml file doesn't work, as it's too low > > > priority > > > (https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence). > > > > > > The multiboot2+EFI path is tested on hardware tests already. > > > > > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > > > --- > > > This requires rebuilding debian:bookworm container. > > > > > > The TEST_TIMEOUT issue mentioned above applies to xilix-* jobs too. It's > > > not clear to me why the default TEST_TIMEOUT is set at the group level > > > instead of in the yaml file, so I'm not adjusting the other places. > > > > Let me premise that now that we use "expect" all successful tests will > > terminate as soon as the success condition is met, without waiting for > > the test timeout to expire. > > > > There is a CI/CD variable called TEST_TIMEOUT set at the > > gitlab.com/xen-project level. (There is also a check in console.exp in > > case TEST_TIMEOUT is not set so that we don't run into problems in case > > the CI/CD variable is removed accidentally.) The global TEST_TIMEOUT is > > meant to be a high value to account for slow QEMU tests running > > potentially on our slowest cloud runners. > > > > However, for hardware-based tests such as the xilinx-* jobs, we know > > that the timeout is supposed to be less than that. The test is running > > on real hardware which is considerably faster than QEMU running on our > > slowest runners. Basically, the timeout depends on the runner more than > > the test. So we override the TEST_TIMEOUT variable for the xilinx-* jobs > > providing a lower timeout value. > > > > The global TEST_TIMEOUT is set to 1500. > > The xilinx-* timeout is set to 120 for ARM and 1000 for x86. > > > > You are welcome to override the TEST_TIMEOUT value for the > > hardware-based QubesOS tests. At the same time, given that on success > > the timeout is not really used, it is also OK to leave it like this. > > > > > --- > > > automation/build/debian/bookworm.dockerfile | 1 + > > > automation/gitlab-ci/test.yaml | 7 ++++ > > > automation/scripts/qemu-smoke-x86-64-efi.sh | 44 +++++++++++++++++++++ > > > 3 files changed, 52 insertions(+) > > > create mode 100755 automation/scripts/qemu-smoke-x86-64-efi.sh > > > > > > diff --git a/automation/build/debian/bookworm.dockerfile b/automation/build/debian/bookworm.dockerfile > > > index 3dd70cb6b2e3..061114ba522d 100644 > > > --- a/automation/build/debian/bookworm.dockerfile > > > +++ b/automation/build/debian/bookworm.dockerfile > > > @@ -46,6 +46,7 @@ RUN apt-get update && \ > > > # for test phase, qemu-smoke-* jobs > > > qemu-system-x86 \ > > > expect \ > > > + ovmf \ > > > # for test phase, qemu-alpine-* jobs > > > cpio \ > > > busybox-static \ > > > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > > > index 8675016b6a37..74fd3f3109ae 100644 > > > --- a/automation/gitlab-ci/test.yaml > > > +++ b/automation/gitlab-ci/test.yaml > > > @@ -463,6 +463,13 @@ qemu-smoke-x86-64-clang-pvh: > > > needs: > > > - debian-bookworm-clang-debug > > > > > > +qemu-smoke-x86-64-gcc-efi: > > > + extends: .qemu-x86-64 > > > + script: > > > + - ./automation/scripts/qemu-smoke-x86-64-efi.sh pv 2>&1 | tee ${LOGFILE} > > > + needs: > > > + - debian-bookworm-gcc-debug > > > > Given that the script you wrote (thank you!) can also handle pvh, can we > > directly add a pvh job to test.yaml too? I guess we can, but is xen.efi + PVH dom0 actually different enough to worth testing given we already test MB2+EFI + PVH dom0? > > > qemu-smoke-riscv64-gcc: > > > extends: .qemu-riscv64 > > > script: > > > diff --git a/automation/scripts/qemu-smoke-x86-64-efi.sh b/automation/scripts/qemu-smoke-x86-64-efi.sh > > > new file mode 100755 > > > index 000000000000..e053cfa995ba > > > --- /dev/null > > > +++ b/automation/scripts/qemu-smoke-x86-64-efi.sh > > > @@ -0,0 +1,44 @@ > > > +#!/bin/bash > > > + > > > +set -ex -o pipefail > > > + > > > +# variant should be either pv or pvh > > > +variant=$1 > > > + > > > +# Clone and build XTF > > > +git clone https://xenbits.xen.org/git-http/xtf.git > > > +cd xtf && make -j$(nproc) && cd - > > > + > > > +case $variant in > > > + pvh) k=test-hvm64-example extra="dom0-iommu=none dom0=pvh" ;; > > > + *) k=test-pv64-example extra= ;; > > > +esac > > > + > > > +mkdir -p boot-esp/EFI/BOOT > > > +cp binaries/xen.efi boot-esp/EFI/BOOT/BOOTX64.EFI > > > +cp xtf/tests/example/$k boot-esp/EFI/BOOT/kernel > > > + > > > +cat > boot-esp/EFI/BOOT/BOOTX64.cfg <<EOF > > > +[global] > > > +default=test > > > + > > > +[test] > > > +options=loglvl=all console=com1 noreboot console_timestamps=boot $extra > > > +kernel=kernel > > > +EOF > > > + > > > +cp /usr/share/OVMF/OVMF_CODE.fd OVMF_CODE.fd > > > +cp /usr/share/OVMF/OVMF_VARS.fd OVMF_VARS.fd > > > + > > > +rm -f smoke.serial > > > +export TEST_CMD="qemu-system-x86_64 -nographic -M q35,kernel-irqchip=split \ > > > + -drive if=pflash,format=raw,readonly=on,file=OVMF_CODE.fd \ > > > + -drive if=pflash,format=raw,file=OVMF_VARS.fd \ > > > + -drive file=fat:rw:boot-esp,media=disk,index=0,format=raw \ > > > + -m 512 -monitor none -serial stdio" > > > + > > > +export TEST_LOG="smoke.serial" > > > +export PASSED="Test result: SUCCESS" > > > +export TEST_TIMEOUT=120 > > Although this works, I would prefer keeping the TEST_TIMEOUT overrides > in test.yaml for consistency. The problem is this doesn't work. The group-level variable overrides the one in yaml. See the commit message and the link there... > However, it might be better not to > override it (or to override to a higher timeout value), as successful > tests will terminate immediately anyway. We need to be cautious about > setting TEST_TIMEOUT values too low, as using a slow runner (like a > small, busy cloud instance) can lead to false positive failures. This > issue occurred frequently with ARM tests when we temporarily moved from > a fast ARM server to slower ARM cloud instances a couple of months ago. > > On the other hand, adjusting TEST_TIMEOUT for non-QEMU hardware-based > tests is acceptable since those tests rely on real hardware > availability, which is unlikely to become suddenly slower. -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] automation: add a smoke test for xen.efi on X86 2024-10-02 22:55 ` Marek Marczykowski-Górecki @ 2024-10-02 23:08 ` Andrew Cooper 2024-10-02 23:30 ` Stefano Stabellini 1 sibling, 0 replies; 14+ messages in thread From: Andrew Cooper @ 2024-10-02 23:08 UTC (permalink / raw) To: Marek Marczykowski-Górecki, Stefano Stabellini Cc: xen-devel, Doug Goldstein On 02/10/2024 11:55 pm, Marek Marczykowski-Górecki wrote: > On Wed, Oct 02, 2024 at 03:22:59PM -0700, Stefano Stabellini wrote: >> I forgot to reply to one important part below >> >> >> On Wed, 2 Oct 2024, Stefano Stabellini wrote: >>> On Wed, 2 Oct 2024, Marek Marczykowski-Górecki wrote: >>>> Check if xen.efi is bootable with an XTF dom0. >>>> >>>> The TEST_TIMEOUT is set in the script to override project-global value. >>>> Setting it in the gitlab yaml file doesn't work, as it's too low >>>> priority >>>> (https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence). >>>> >>>> The multiboot2+EFI path is tested on hardware tests already. >>>> >>>> Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> >>>> --- >>>> This requires rebuilding debian:bookworm container. >>>> >>>> The TEST_TIMEOUT issue mentioned above applies to xilix-* jobs too. It's >>>> not clear to me why the default TEST_TIMEOUT is set at the group level >>>> instead of in the yaml file, so I'm not adjusting the other places. >>> Let me premise that now that we use "expect" all successful tests will >>> terminate as soon as the success condition is met, without waiting for >>> the test timeout to expire. >>> >>> There is a CI/CD variable called TEST_TIMEOUT set at the >>> gitlab.com/xen-project level. (There is also a check in console.exp in >>> case TEST_TIMEOUT is not set so that we don't run into problems in case >>> the CI/CD variable is removed accidentally.) The global TEST_TIMEOUT is >>> meant to be a high value to account for slow QEMU tests running >>> potentially on our slowest cloud runners. >>> >>> However, for hardware-based tests such as the xilinx-* jobs, we know >>> that the timeout is supposed to be less than that. The test is running >>> on real hardware which is considerably faster than QEMU running on our >>> slowest runners. Basically, the timeout depends on the runner more than >>> the test. So we override the TEST_TIMEOUT variable for the xilinx-* jobs >>> providing a lower timeout value. >>> >>> The global TEST_TIMEOUT is set to 1500. >>> The xilinx-* timeout is set to 120 for ARM and 1000 for x86. >>> >>> You are welcome to override the TEST_TIMEOUT value for the >>> hardware-based QubesOS tests. At the same time, given that on success >>> the timeout is not really used, it is also OK to leave it like this. >> >> >>>> --- >>>> automation/build/debian/bookworm.dockerfile | 1 + >>>> automation/gitlab-ci/test.yaml | 7 ++++ >>>> automation/scripts/qemu-smoke-x86-64-efi.sh | 44 +++++++++++++++++++++ >>>> 3 files changed, 52 insertions(+) >>>> create mode 100755 automation/scripts/qemu-smoke-x86-64-efi.sh >>>> >>>> diff --git a/automation/build/debian/bookworm.dockerfile b/automation/build/debian/bookworm.dockerfile >>>> index 3dd70cb6b2e3..061114ba522d 100644 >>>> --- a/automation/build/debian/bookworm.dockerfile >>>> +++ b/automation/build/debian/bookworm.dockerfile >>>> @@ -46,6 +46,7 @@ RUN apt-get update && \ >>>> # for test phase, qemu-smoke-* jobs >>>> qemu-system-x86 \ >>>> expect \ >>>> + ovmf \ >>>> # for test phase, qemu-alpine-* jobs >>>> cpio \ >>>> busybox-static \ >>>> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml >>>> index 8675016b6a37..74fd3f3109ae 100644 >>>> --- a/automation/gitlab-ci/test.yaml >>>> +++ b/automation/gitlab-ci/test.yaml >>>> @@ -463,6 +463,13 @@ qemu-smoke-x86-64-clang-pvh: >>>> needs: >>>> - debian-bookworm-clang-debug >>>> >>>> +qemu-smoke-x86-64-gcc-efi: >>>> + extends: .qemu-x86-64 >>>> + script: >>>> + - ./automation/scripts/qemu-smoke-x86-64-efi.sh pv 2>&1 | tee ${LOGFILE} >>>> + needs: >>>> + - debian-bookworm-gcc-debug >>> Given that the script you wrote (thank you!) can also handle pvh, can we >>> directly add a pvh job to test.yaml too? > I guess we can, but is xen.efi + PVH dom0 actually different enough to > worth testing given we already test MB2+EFI + PVH dom0? Given that its only an XTF dom0, no. We're only smoke testing the "native EFI" bits here. ~Andrew ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] automation: add a smoke test for xen.efi on X86 2024-10-02 22:55 ` Marek Marczykowski-Górecki 2024-10-02 23:08 ` Andrew Cooper @ 2024-10-02 23:30 ` Stefano Stabellini 2024-10-03 0:24 ` Marek Marczykowski-Górecki 1 sibling, 1 reply; 14+ messages in thread From: Stefano Stabellini @ 2024-10-02 23:30 UTC (permalink / raw) To: Marek Marczykowski-Górecki Cc: Stefano Stabellini, xen-devel, Doug Goldstein [-- Attachment #1: Type: text/plain, Size: 8284 bytes --] On Thu, 3 Oct 2024, Marek Marczykowski-Górecki wrote: > On Wed, Oct 02, 2024 at 03:22:59PM -0700, Stefano Stabellini wrote: > > I forgot to reply to one important part below > > > > > > On Wed, 2 Oct 2024, Stefano Stabellini wrote: > > > On Wed, 2 Oct 2024, Marek Marczykowski-Górecki wrote: > > > > Check if xen.efi is bootable with an XTF dom0. > > > > > > > > The TEST_TIMEOUT is set in the script to override project-global value. > > > > Setting it in the gitlab yaml file doesn't work, as it's too low > > > > priority > > > > (https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence). > > > > > > > > The multiboot2+EFI path is tested on hardware tests already. > > > > > > > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > > > > --- > > > > This requires rebuilding debian:bookworm container. > > > > > > > > The TEST_TIMEOUT issue mentioned above applies to xilix-* jobs too. It's > > > > not clear to me why the default TEST_TIMEOUT is set at the group level > > > > instead of in the yaml file, so I'm not adjusting the other places. > > > > > > Let me premise that now that we use "expect" all successful tests will > > > terminate as soon as the success condition is met, without waiting for > > > the test timeout to expire. > > > > > > There is a CI/CD variable called TEST_TIMEOUT set at the > > > gitlab.com/xen-project level. (There is also a check in console.exp in > > > case TEST_TIMEOUT is not set so that we don't run into problems in case > > > the CI/CD variable is removed accidentally.) The global TEST_TIMEOUT is > > > meant to be a high value to account for slow QEMU tests running > > > potentially on our slowest cloud runners. > > > > > > However, for hardware-based tests such as the xilinx-* jobs, we know > > > that the timeout is supposed to be less than that. The test is running > > > on real hardware which is considerably faster than QEMU running on our > > > slowest runners. Basically, the timeout depends on the runner more than > > > the test. So we override the TEST_TIMEOUT variable for the xilinx-* jobs > > > providing a lower timeout value. > > > > > > The global TEST_TIMEOUT is set to 1500. > > > The xilinx-* timeout is set to 120 for ARM and 1000 for x86. > > > > > > You are welcome to override the TEST_TIMEOUT value for the > > > hardware-based QubesOS tests. At the same time, given that on success > > > the timeout is not really used, it is also OK to leave it like this. > > > > > > > > --- > > > > automation/build/debian/bookworm.dockerfile | 1 + > > > > automation/gitlab-ci/test.yaml | 7 ++++ > > > > automation/scripts/qemu-smoke-x86-64-efi.sh | 44 +++++++++++++++++++++ > > > > 3 files changed, 52 insertions(+) > > > > create mode 100755 automation/scripts/qemu-smoke-x86-64-efi.sh > > > > > > > > diff --git a/automation/build/debian/bookworm.dockerfile b/automation/build/debian/bookworm.dockerfile > > > > index 3dd70cb6b2e3..061114ba522d 100644 > > > > --- a/automation/build/debian/bookworm.dockerfile > > > > +++ b/automation/build/debian/bookworm.dockerfile > > > > @@ -46,6 +46,7 @@ RUN apt-get update && \ > > > > # for test phase, qemu-smoke-* jobs > > > > qemu-system-x86 \ > > > > expect \ > > > > + ovmf \ > > > > # for test phase, qemu-alpine-* jobs > > > > cpio \ > > > > busybox-static \ > > > > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml > > > > index 8675016b6a37..74fd3f3109ae 100644 > > > > --- a/automation/gitlab-ci/test.yaml > > > > +++ b/automation/gitlab-ci/test.yaml > > > > @@ -463,6 +463,13 @@ qemu-smoke-x86-64-clang-pvh: > > > > needs: > > > > - debian-bookworm-clang-debug > > > > > > > > +qemu-smoke-x86-64-gcc-efi: > > > > + extends: .qemu-x86-64 > > > > + script: > > > > + - ./automation/scripts/qemu-smoke-x86-64-efi.sh pv 2>&1 | tee ${LOGFILE} > > > > + needs: > > > > + - debian-bookworm-gcc-debug > > > > > > Given that the script you wrote (thank you!) can also handle pvh, can we > > > directly add a pvh job to test.yaml too? > > I guess we can, but is xen.efi + PVH dom0 actually different enough to > worth testing given we already test MB2+EFI + PVH dom0? As it looks like also Andy thinks this is not useful, I am happy not to do this. Let's go with PV only as you did. > > > > qemu-smoke-riscv64-gcc: > > > > extends: .qemu-riscv64 > > > > script: > > > > diff --git a/automation/scripts/qemu-smoke-x86-64-efi.sh b/automation/scripts/qemu-smoke-x86-64-efi.sh > > > > new file mode 100755 > > > > index 000000000000..e053cfa995ba > > > > --- /dev/null > > > > +++ b/automation/scripts/qemu-smoke-x86-64-efi.sh > > > > @@ -0,0 +1,44 @@ > > > > +#!/bin/bash > > > > + > > > > +set -ex -o pipefail > > > > + > > > > +# variant should be either pv or pvh > > > > +variant=$1 > > > > + > > > > +# Clone and build XTF > > > > +git clone https://xenbits.xen.org/git-http/xtf.git > > > > +cd xtf && make -j$(nproc) && cd - > > > > + > > > > +case $variant in > > > > + pvh) k=test-hvm64-example extra="dom0-iommu=none dom0=pvh" ;; > > > > + *) k=test-pv64-example extra= ;; > > > > +esac > > > > + > > > > +mkdir -p boot-esp/EFI/BOOT > > > > +cp binaries/xen.efi boot-esp/EFI/BOOT/BOOTX64.EFI > > > > +cp xtf/tests/example/$k boot-esp/EFI/BOOT/kernel > > > > + > > > > +cat > boot-esp/EFI/BOOT/BOOTX64.cfg <<EOF > > > > +[global] > > > > +default=test > > > > + > > > > +[test] > > > > +options=loglvl=all console=com1 noreboot console_timestamps=boot $extra > > > > +kernel=kernel > > > > +EOF > > > > + > > > > +cp /usr/share/OVMF/OVMF_CODE.fd OVMF_CODE.fd > > > > +cp /usr/share/OVMF/OVMF_VARS.fd OVMF_VARS.fd > > > > + > > > > +rm -f smoke.serial > > > > +export TEST_CMD="qemu-system-x86_64 -nographic -M q35,kernel-irqchip=split \ > > > > + -drive if=pflash,format=raw,readonly=on,file=OVMF_CODE.fd \ > > > > + -drive if=pflash,format=raw,file=OVMF_VARS.fd \ > > > > + -drive file=fat:rw:boot-esp,media=disk,index=0,format=raw \ > > > > + -m 512 -monitor none -serial stdio" > > > > + > > > > +export TEST_LOG="smoke.serial" > > > > +export PASSED="Test result: SUCCESS" > > > > +export TEST_TIMEOUT=120 > > > > Although this works, I would prefer keeping the TEST_TIMEOUT overrides > > in test.yaml for consistency. > > The problem is this doesn't work. The group-level variable overrides the > one in yaml. See the commit message and the link there... Now I understand the problem, well spotted, thanks! The idea behind having TEST_TIMEOUT defined as a project CI/CD variable is that it depends on the test infrastructure rather than the Xen code. So if we suddenly had slower runners we could change TEST_TIMEOUT without having to change the Xen code itself. So I think we should keep TEST_TIMEOUT as a project CI/CD variable. I am not a fan of overwriting the TEST_TIMEOUT variable in the test scripts, because one test script can be used for multiple different tests, possibly even with different runners. For instance qubes-x86-64.sh works with a couple of different hardware runners that could have different timeout values. But I think it would work OK for now for our hardware-based tests (e.g. xilinx-smoke-dom0less-arm64.sh and qubes-x86-64.sh could overwrite TEST_TIMEOUT). For this specific XTF test, I am not sure it is worth optimizing the timeout, maybe we should leave it as default. However if we wanted to lower the timeout value, overwriting it the way you did is OKish as I cannot think of another way. > > However, it might be better not to > > override it (or to override to a higher timeout value), as successful > > tests will terminate immediately anyway. We need to be cautious about > > setting TEST_TIMEOUT values too low, as using a slow runner (like a > > small, busy cloud instance) can lead to false positive failures. This > > issue occurred frequently with ARM tests when we temporarily moved from > > a fast ARM server to slower ARM cloud instances a couple of months ago. > > > > On the other hand, adjusting TEST_TIMEOUT for non-QEMU hardware-based > > tests is acceptable since those tests rely on real hardware > > availability, which is unlikely to become suddenly slower. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] automation: add a smoke test for xen.efi on X86 2024-10-02 23:30 ` Stefano Stabellini @ 2024-10-03 0:24 ` Marek Marczykowski-Górecki 2024-10-03 0:32 ` Stefano Stabellini 0 siblings, 1 reply; 14+ messages in thread From: Marek Marczykowski-Górecki @ 2024-10-03 0:24 UTC (permalink / raw) To: Stefano Stabellini; +Cc: xen-devel, Doug Goldstein [-- Attachment #1: Type: text/plain, Size: 1909 bytes --] On Wed, Oct 02, 2024 at 04:30:25PM -0700, Stefano Stabellini wrote: > On Thu, 3 Oct 2024, Marek Marczykowski-Górecki wrote: > > The problem is this doesn't work. The group-level variable overrides the > > one in yaml. See the commit message and the link there... > > Now I understand the problem, well spotted, thanks! > > The idea behind having TEST_TIMEOUT defined as a project CI/CD variable > is that it depends on the test infrastructure rather than the Xen code. > So if we suddenly had slower runners we could change TEST_TIMEOUT > without having to change the Xen code itself. So I think we should keep > TEST_TIMEOUT as a project CI/CD variable. > > I am not a fan of overwriting the TEST_TIMEOUT variable in the test > scripts, because one test script can be used for multiple different > tests, possibly even with different runners. For instance > qubes-x86-64.sh works with a couple of different hardware runners that > could have different timeout values. But I think it would work OK for > now for our hardware-based tests (e.g. xilinx-smoke-dom0less-arm64.sh > and qubes-x86-64.sh could overwrite TEST_TIMEOUT). > > For this specific XTF test, I am not sure it is worth optimizing the > timeout, maybe we should leave it as default. The default of 25min is quite wasteful for XTF test that failed... > However if we wanted to > lower the timeout value, overwriting it the way you did is OKish as I > cannot think of another way. If we'd need this option more often, Maybe we could set TEST_TIMEOUT_OVERRIDE in test yaml, and then test script use that (if present) instead? Or maybe have few "classes" of timeouts set globally (TEST_TIMEOUT_SHORT, TEST_TIMEOUT_MEDIUM, TEST_TIMEOUT_LONG? or some better named categories). But I don't think it's worth it for this XTF test yet. -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] automation: add a smoke test for xen.efi on X86 2024-10-03 0:24 ` Marek Marczykowski-Górecki @ 2024-10-03 0:32 ` Stefano Stabellini 2024-10-03 20:24 ` Stefano Stabellini 0 siblings, 1 reply; 14+ messages in thread From: Stefano Stabellini @ 2024-10-03 0:32 UTC (permalink / raw) To: Marek Marczykowski-Górecki Cc: Stefano Stabellini, xen-devel, Doug Goldstein [-- Attachment #1: Type: text/plain, Size: 1977 bytes --] On Thu, 3 Oct 2024, Marek Marczykowski-Górecki wrote: > On Wed, Oct 02, 2024 at 04:30:25PM -0700, Stefano Stabellini wrote: > > On Thu, 3 Oct 2024, Marek Marczykowski-Górecki wrote: > > > The problem is this doesn't work. The group-level variable overrides the > > > one in yaml. See the commit message and the link there... > > > > Now I understand the problem, well spotted, thanks! > > > > The idea behind having TEST_TIMEOUT defined as a project CI/CD variable > > is that it depends on the test infrastructure rather than the Xen code. > > So if we suddenly had slower runners we could change TEST_TIMEOUT > > without having to change the Xen code itself. So I think we should keep > > TEST_TIMEOUT as a project CI/CD variable. > > > > I am not a fan of overwriting the TEST_TIMEOUT variable in the test > > scripts, because one test script can be used for multiple different > > tests, possibly even with different runners. For instance > > qubes-x86-64.sh works with a couple of different hardware runners that > > could have different timeout values. But I think it would work OK for > > now for our hardware-based tests (e.g. xilinx-smoke-dom0less-arm64.sh > > and qubes-x86-64.sh could overwrite TEST_TIMEOUT). > > > > For this specific XTF test, I am not sure it is worth optimizing the > > timeout, maybe we should leave it as default. > > The default of 25min is quite wasteful for XTF test that failed... > > > However if we wanted to > > lower the timeout value, overwriting it the way you did is OKish as I > > cannot think of another way. > > If we'd need this option more often, Maybe we could set > TEST_TIMEOUT_OVERRIDE in test yaml, and then test script use that (if > present) instead? Or maybe have few "classes" of timeouts set globally > (TEST_TIMEOUT_SHORT, TEST_TIMEOUT_MEDIUM, TEST_TIMEOUT_LONG? or some > better named categories). But I don't think it's worth it for this XTF > test yet. Agreed, and good idea about TEST_TIMEOUT_OVERRIDE ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] automation: add a smoke test for xen.efi on X86 2024-10-03 0:32 ` Stefano Stabellini @ 2024-10-03 20:24 ` Stefano Stabellini 0 siblings, 0 replies; 14+ messages in thread From: Stefano Stabellini @ 2024-10-03 20:24 UTC (permalink / raw) To: Stefano Stabellini Cc: Marek Marczykowski-Górecki, xen-devel, Doug Goldstein [-- Attachment #1: Type: text/plain, Size: 2354 bytes --] On Wed, 2 Oct 2024, Stefano Stabellini wrote: > On Thu, 3 Oct 2024, Marek Marczykowski-Górecki wrote: > > On Wed, Oct 02, 2024 at 04:30:25PM -0700, Stefano Stabellini wrote: > > > On Thu, 3 Oct 2024, Marek Marczykowski-Górecki wrote: > > > > The problem is this doesn't work. The group-level variable overrides the > > > > one in yaml. See the commit message and the link there... > > > > > > Now I understand the problem, well spotted, thanks! > > > > > > The idea behind having TEST_TIMEOUT defined as a project CI/CD variable > > > is that it depends on the test infrastructure rather than the Xen code. > > > So if we suddenly had slower runners we could change TEST_TIMEOUT > > > without having to change the Xen code itself. So I think we should keep > > > TEST_TIMEOUT as a project CI/CD variable. > > > > > > I am not a fan of overwriting the TEST_TIMEOUT variable in the test > > > scripts, because one test script can be used for multiple different > > > tests, possibly even with different runners. For instance > > > qubes-x86-64.sh works with a couple of different hardware runners that > > > could have different timeout values. But I think it would work OK for > > > now for our hardware-based tests (e.g. xilinx-smoke-dom0less-arm64.sh > > > and qubes-x86-64.sh could overwrite TEST_TIMEOUT). > > > > > > For this specific XTF test, I am not sure it is worth optimizing the > > > timeout, maybe we should leave it as default. > > > > The default of 25min is quite wasteful for XTF test that failed... > > > > > However if we wanted to > > > lower the timeout value, overwriting it the way you did is OKish as I > > > cannot think of another way. > > > > If we'd need this option more often, Maybe we could set > > TEST_TIMEOUT_OVERRIDE in test yaml, and then test script use that (if > > present) instead? Or maybe have few "classes" of timeouts set globally > > (TEST_TIMEOUT_SHORT, TEST_TIMEOUT_MEDIUM, TEST_TIMEOUT_LONG? or some > > better named categories). But I don't think it's worth it for this XTF > > test yet. > > Agreed, and good idea about TEST_TIMEOUT_OVERRIDE I decided to send a patch to implement it as I didn't want to keep the TEST_TIMEOUT as is (ignored) for the Xilinx jobs https://marc.info/?l=xen-devel&m=172798685928204 You should be able to use TEST_TIMEOUT_OVERRIDE in your test.yaml for this test ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] automation: preserve built xen.efi 2024-10-02 12:42 [PATCH 1/2] automation: preserve built xen.efi Marek Marczykowski-Górecki 2024-10-02 12:42 ` [PATCH 2/2] automation: add a smoke test for xen.efi on X86 Marek Marczykowski-Górecki @ 2024-10-02 20:42 ` Andrew Cooper 2024-10-02 20:46 ` Marek Marczykowski-Górecki 1 sibling, 1 reply; 14+ messages in thread From: Andrew Cooper @ 2024-10-02 20:42 UTC (permalink / raw) To: Marek Marczykowski-Górecki, xen-devel Cc: Doug Goldstein, Stefano Stabellini On 02/10/2024 1:42 pm, Marek Marczykowski-Górecki wrote: > It will be useful for further tests. > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > --- > automation/scripts/build | 7 ++++++- > 1 file changed, 6 insertions(+), 1 deletion(-) > > diff --git a/automation/scripts/build b/automation/scripts/build > index b3c71fb6fb60..4cd41cb2c471 100755 > --- a/automation/scripts/build > +++ b/automation/scripts/build > @@ -47,6 +47,7 @@ if [[ "${CPPCHECK}" == "y" ]] && [[ "${HYPERVISOR_ONLY}" == "y" ]]; then > > # Preserve artefacts > cp xen/xen binaries/xen > + if [[ -f xen/xen.efi ]]; then cp xen/xen.efi binaries/xen.efi; fi Wouldn't # Preserve xen and optionally xen.efi cp -f xen/xen xen/xen.efi binaries/ do this in a more concise way? Alternatively, what about this: diff --git a/automation/scripts/build b/automation/scripts/build index b3c71fb6fb60..14815ea7ad9c 100755 --- a/automation/scripts/build +++ b/automation/scripts/build @@ -41,6 +41,15 @@ cp xen/.config xen-config # Directory for the artefacts to be dumped into mkdir -p binaries +collect_xen_artefacts () +{ + for A in xen/xen xen/xen.efi; do + if [[ -f $A ]]; then + cp $A binaries/ + fi + done +} + if [[ "${CPPCHECK}" == "y" ]] && [[ "${HYPERVISOR_ONLY}" == "y" ]]; then # Cppcheck analysis invokes Xen-only build xen/scripts/xen-analysis.py --run-cppcheck --cppcheck-misra -- -j$(nproc) @@ -53,7 +62,7 @@ elif [[ "${HYPERVISOR_ONLY}" == "y" ]]; then make -j$(nproc) xen # Preserve artefacts - cp xen/xen binaries/xen + collect_xen_artefacts else # Full build. Figure out our ./configure options cfgargs=() so we don't triplicate the handling? ~Andrew ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] automation: preserve built xen.efi 2024-10-02 20:42 ` [PATCH 1/2] automation: preserve built xen.efi Andrew Cooper @ 2024-10-02 20:46 ` Marek Marczykowski-Górecki 2024-10-02 22:16 ` Stefano Stabellini 0 siblings, 1 reply; 14+ messages in thread From: Marek Marczykowski-Górecki @ 2024-10-02 20:46 UTC (permalink / raw) To: Andrew Cooper; +Cc: xen-devel, Doug Goldstein, Stefano Stabellini [-- Attachment #1: Type: text/plain, Size: 2257 bytes --] On Wed, Oct 02, 2024 at 09:42:13PM +0100, Andrew Cooper wrote: > On 02/10/2024 1:42 pm, Marek Marczykowski-Górecki wrote: > > It will be useful for further tests. > > > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > > --- > > automation/scripts/build | 7 ++++++- > > 1 file changed, 6 insertions(+), 1 deletion(-) > > > > diff --git a/automation/scripts/build b/automation/scripts/build > > index b3c71fb6fb60..4cd41cb2c471 100755 > > --- a/automation/scripts/build > > +++ b/automation/scripts/build > > @@ -47,6 +47,7 @@ if [[ "${CPPCHECK}" == "y" ]] && [[ "${HYPERVISOR_ONLY}" == "y" ]]; then > > > > # Preserve artefacts > > cp xen/xen binaries/xen > > + if [[ -f xen/xen.efi ]]; then cp xen/xen.efi binaries/xen.efi; fi > > Wouldn't > > # Preserve xen and optionally xen.efi > cp -f xen/xen xen/xen.efi binaries/ > > do this in a more concise way? I don't think so, `cp -f` still fails if the source cannot be found. > Alternatively, what about this: > > diff --git a/automation/scripts/build b/automation/scripts/build > index b3c71fb6fb60..14815ea7ad9c 100755 > --- a/automation/scripts/build > +++ b/automation/scripts/build > @@ -41,6 +41,15 @@ cp xen/.config xen-config > # Directory for the artefacts to be dumped into > mkdir -p binaries > > +collect_xen_artefacts () > +{ > + for A in xen/xen xen/xen.efi; do > + if [[ -f $A ]]; then > + cp $A binaries/ > + fi > + done > +} > + > if [[ "${CPPCHECK}" == "y" ]] && [[ "${HYPERVISOR_ONLY}" == "y" ]]; then > # Cppcheck analysis invokes Xen-only build > xen/scripts/xen-analysis.py --run-cppcheck --cppcheck-misra -- > -j$(nproc) > @@ -53,7 +62,7 @@ elif [[ "${HYPERVISOR_ONLY}" == "y" ]]; then > make -j$(nproc) xen > > # Preserve artefacts > - cp xen/xen binaries/xen > + collect_xen_artefacts > else > # Full build. Figure out our ./configure options > cfgargs=() > > so we don't triplicate the handling? That may be a better idea indeed. -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] automation: preserve built xen.efi 2024-10-02 20:46 ` Marek Marczykowski-Górecki @ 2024-10-02 22:16 ` Stefano Stabellini 0 siblings, 0 replies; 14+ messages in thread From: Stefano Stabellini @ 2024-10-02 22:16 UTC (permalink / raw) To: Marek Marczykowski-Górecki Cc: Andrew Cooper, xen-devel, Doug Goldstein, Stefano Stabellini [-- Attachment #1: Type: text/plain, Size: 2677 bytes --] On Wed, 2 Oct 2024, Marek Marczykowski-Górecki wrote: > On Wed, Oct 02, 2024 at 09:42:13PM +0100, Andrew Cooper wrote: > > On 02/10/2024 1:42 pm, Marek Marczykowski-Górecki wrote: > > > It will be useful for further tests. > > > > > > Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com> > > > --- > > > automation/scripts/build | 7 ++++++- > > > 1 file changed, 6 insertions(+), 1 deletion(-) > > > > > > diff --git a/automation/scripts/build b/automation/scripts/build > > > index b3c71fb6fb60..4cd41cb2c471 100755 > > > --- a/automation/scripts/build > > > +++ b/automation/scripts/build > > > @@ -47,6 +47,7 @@ if [[ "${CPPCHECK}" == "y" ]] && [[ "${HYPERVISOR_ONLY}" == "y" ]]; then > > > > > > # Preserve artefacts > > > cp xen/xen binaries/xen > > > + if [[ -f xen/xen.efi ]]; then cp xen/xen.efi binaries/xen.efi; fi > > > > Wouldn't > > > > # Preserve xen and optionally xen.efi > > cp -f xen/xen xen/xen.efi binaries/ > > > > do this in a more concise way? > > I don't think so, `cp -f` still fails if the source cannot be found. I think it would have to be something like: cp -f xen/xen xen/xen.efi binaries/ || true > > Alternatively, what about this: > > > > diff --git a/automation/scripts/build b/automation/scripts/build > > index b3c71fb6fb60..14815ea7ad9c 100755 > > --- a/automation/scripts/build > > +++ b/automation/scripts/build > > @@ -41,6 +41,15 @@ cp xen/.config xen-config > > # Directory for the artefacts to be dumped into > > mkdir -p binaries > > > > +collect_xen_artefacts () > > +{ > > + for A in xen/xen xen/xen.efi; do > > + if [[ -f $A ]]; then > > + cp $A binaries/ > > + fi > > + done > > +} > > + > > if [[ "${CPPCHECK}" == "y" ]] && [[ "${HYPERVISOR_ONLY}" == "y" ]]; then > > # Cppcheck analysis invokes Xen-only build > > xen/scripts/xen-analysis.py --run-cppcheck --cppcheck-misra -- > > -j$(nproc) > > @@ -53,7 +62,7 @@ elif [[ "${HYPERVISOR_ONLY}" == "y" ]]; then > > make -j$(nproc) xen > > > > # Preserve artefacts > > - cp xen/xen binaries/xen > > + collect_xen_artefacts > > else > > # Full build. Figure out our ./configure options > > cfgargs=() > > > > so we don't triplicate the handling? > > That may be a better idea indeed. collect_xen_artefacts is also a good option, perhaps even better. A couple of minor NITs: function collect_xen_artefacts() { local f for f in xen/xen xen/xen.efi; do if [[ -f $f ]]; then cp $f binaries/ fi done } ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2024-10-03 20:24 UTC | newest] Thread overview: 14+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-10-02 12:42 [PATCH 1/2] automation: preserve built xen.efi Marek Marczykowski-Górecki 2024-10-02 12:42 ` [PATCH 2/2] automation: add a smoke test for xen.efi on X86 Marek Marczykowski-Górecki 2024-10-02 21:03 ` Andrew Cooper 2024-10-02 22:16 ` Stefano Stabellini 2024-10-02 22:22 ` Stefano Stabellini 2024-10-02 22:55 ` Marek Marczykowski-Górecki 2024-10-02 23:08 ` Andrew Cooper 2024-10-02 23:30 ` Stefano Stabellini 2024-10-03 0:24 ` Marek Marczykowski-Górecki 2024-10-03 0:32 ` Stefano Stabellini 2024-10-03 20:24 ` Stefano Stabellini 2024-10-02 20:42 ` [PATCH 1/2] automation: preserve built xen.efi Andrew Cooper 2024-10-02 20:46 ` Marek Marczykowski-Górecki 2024-10-02 22:16 ` Stefano Stabellini
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.