Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Gustavo Romero <gustavo.romero@linaro.org>]

Hello, Thomas!

On 8/25/25 07:29, Thomas Huth wrote:
> On 19/08/2025 16.39, Gustavo Romero wrote:
>> The goal of this series is to remove Avocado as a dependency for running
>> the reverse_debugging functional test.
>>
>> This test, the last one I’m aware of that relies on Avocado, requires it
>> because of the need for GDB to test reverse stepping and continue.
> 
>   Hi!
> 
> Please note that there are currently also some efforts going on to extract the GDB part from avocado into a more self-contained python module called aautils, which might help here, too:
> 
>   https://github.com/avocado-framework/aautils/issues/82

Thanks, I was not aware of it. But note that the Avocado GDB API requires one
to deal with GDB packets directly, which is unnecessary for tests like reverse
debug. I think that in general, in QEMU, we should try to avoid using it as
much as possible because the tests became annoying to read without any strong
need for it.

In the commit message for 4/4 I mention that benefit of using the GDB Python
API instead.

I think we should aim to make the tests in QEMU _extremely_ easy to list, run,
and read. The original reverse_debugging test, using Avocado's GDB API, for
instance, is one example we should try to avoid. Say some developer is changing
something not related to RR but ends up breaking reverse_debugging. The developer
should at least be able to promptly run the QEMU, connect GDB, and reproduce the
issue manually following the exact commands in the test, without having to
consult the GDB protocol manual and to find out which GDB command is mapped to
which GDB packet.

In that sense I confess I'm not 100% happy with the output of reverse_debugging
yet, but one problem at a time :)


>> In this series, we leveraged the run-test.py script used in the
>> check-tcg tests, making it a GDB runner capable of calling a test script
>> without spawning any VMs. In this configuration, the test script can
>> manage the VMs and also import gdb from the test script, making the
>> Python GDB API available inside the functional test.
>>
>> The test is kept “skipped” for aarch64, ppc64, and x86_64, so it is
>> necessary to set QEMU_TEST_FLAKY_TESTS=1 in the test environment to
>> effectively run the test on these archs.
>>
>> On aarch64, the test is flaky, but there is a fix that I’ve tested while
>> writing this series [0] that resolves it. On ppc64 and x86_64, the test
>> always fails: on ppc64, GDB gets a bogus PC, and on x86_64, the last
>> part of the test (reverse-continue) does not hit the last executed PC
>> (as it should) but instead jumps to the beginning of the code (fist PC
>> in forward order).
> 
> At least the x86 test used to work for me most of the time, it just fails occasionally. Same for the ppc64 test if I disable the powernv subtest there.

ok, I'll double check the state on x86 in master. I understand we're not
testing reverse_debugging in our CI currently?

  >> Thus, to run the reverse_debugging test effectively on aarch64:
>>
>> $ export QEMU_TEST_FLAKY_TESTS=1
>> $ make check-functional
> 
> I gave it a try, but this did not work for me, the test was not run at all anymore. Are there any patches needed on top?

hmm that's odd. I'm able to run it with 'make check-functional' and with 'meson test'...

This is how I'm running it (let me know if I'm missing something):

gromero@gromero0:/mnt/git/qemu_$ b4 shazam 20250819143916.4138035-1-gustavo.romero@linaro.org
Grabbing thread from lore.kernel.org/all/20250819143916.4138035-1-gustavo.romero@linaro.org/t.mbox.gz
Checking for newer revisions
Grabbing search results from lore.kernel.org
Analyzing 9 messages in the thread
Looking for additional code-review trailers on lore.kernel.org
Analyzing 0 code-review messages
Checking attestation on all messages, may take a moment...
---
   ✓ [PATCH 1/4] tests/guest-debug: Make QEMU optional in run-test.py
   ✓ [PATCH 2/4] tests/functional: Support tests that require a runner
   ✓ [PATCH 3/4] tests/functional: Mark main in QemuBaseTest class as a static method
     + Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> (✓ DKIM/redhat.com)
   ✓ [PATCH 4/4] tests/functional: Adapt reverse_debugging to run w/o Avocado
   ---
   ✓ Signed: DKIM/linaro.org
---
Total patches: 4
---
Applying: tests/guest-debug: Make QEMU optional in run-test.py
Applying: tests/functional: Support tests that require a runner
Applying: tests/functional: Mark main in QemuBaseTest class as a static method
Applying: tests/functional: Adapt reverse_debugging to run w/o Avocado
gromero@gromero0:/mnt/git/qemu_$
gromero@gromero0:/mnt/git/qemu_/build$ ../configure --target-list=aarch64-softmmu --disable-docs
gromero@gromero0:/mnt/git/qemu_/build$ make -j 32
gromero@gromero0:/mnt/git/qemu_/build$ time make -j 15  check-functional
[1/2] Generating tests/functional/func-precache-aarch64-aarch64_reverse_debug with a custom command (wrapped by meson to set env)
2025-08-25 12:50:04,215 - qemu-test - INFO - Attempting to cache '/home/gromero/.cache/qemu/download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7'
2025-08-25 12:50:04,225 - qemu-test - DEBUG - Using cached asset /home/gromero/.cache/qemu/download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7 for https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz
GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py']" -x /mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
[0/1] Running external command precache-functional (wrapped by meson to set env)
make[1]: Entering directory '/mnt/git/qemu_/build'
[1/6] Generating qemu-version.h with a custom command (wrapped by meson to capture output)
/mnt/git/qemu_/build/pyvenv/bin/meson test  --no-rebuild -t 1 --setup thorough  --num-processes 10 --print-errorlogs  --suite func  --suite func-quick  --suite func-thorough
  1/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_virt_gpu                   SKIP              1.95s   0 subtests passed
  2/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_smmu                       SKIP              4.17s   0 subtests passed
  3/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_raspi4                     OK               28.11s   2 subtests passed
  4/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_device_passthrough         OK               31.34s   1 subtests passed
  5/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_reverse_debug              OK               11.73s   <------ 'exitcode', not TAP, so no subtests-like output
  6/27 qemu:func-quick+func-aarch64 / func-aarch64-migration                                               OK                0.97s   2 subtests passed
  7/27 qemu:func-quick+func-aarch64 / func-aarch64-empty_cpu_model                                         OK                0.17s   1 subtests passed
[...]

gromero@gromero0:/mnt/git/qemu_/build$ grep -A 58 "= 5/27 =" /mnt/git/qemu_/build/meson-logs/testlog-thorough.txt
==================================== 5/27 ====================================
test:         qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_reverse_debug
start time:   13:44:21
duration:     11.73s
result:       exit status 0
command:      LD_LIBRARY_PATH=/mnt/git/qemu_/build/contrib/plugins:/mnt/git/qemu_/build/tests/tcg/plugins QEMU_BUILD_ROOT=/mnt/git/qemu_/build MALLOC_PERTURB_=138 G_TEST_SLOW=1 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 RUST_BACKTRACE=1 QEMU_TEST_QEMU_IMG=/mnt/git/qemu_/build/qemu-img MESON_TEST_ITERATION=1 PYTHONPATH=/mnt/git/qemu_/python:/mnt/git/qemu_/tests/functional:/mnt/git/qemu_/build/pyvenv/lib/python3.10/site-packages QEMU_TEST_QEMU_BINARY=/mnt/git/qemu_/build/qemu-system-aarch64 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 /mnt/git/qemu_/build/pyvenv/bin/python3 /mnt/git/qemu_/tests/guest-debug/run-test.py --gdb /usr/bin/gdb-multiarch --test /mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
----------------------------------- stdout -----------------------------------
Formatting '/mnt/git/qemu_/build/tests/functional/aarch64/test_aarch64_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt/scratch/disk.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=134217728 lazy_refcounts=off refcount_bits=16
TAP version 13
0x0000000040000000 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000008 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000010 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040080004 in ?? ()
0x00000000410d0000 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0024 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0000 in ?? ()
0x0000000040080004 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040000010 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000008 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000000 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000008 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000010 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040080004 in ?? ()
0x00000000410d0000 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0024 in ?? ()
Breakpoint 1 at 0x40000000
Breakpoint 2 at 0x40000004
Breakpoint 3 at 0x40000008
Breakpoint 4 at 0x4000000c
Breakpoint 5 at 0x40000010
Breakpoint 6 at 0x40000014
Breakpoint 7 at 0x40080000
Breakpoint 8 at 0x40080004
Breakpoint 9 at 0x410d0000
Breakpoint 10 at 0x410d0020

Program received signal SIGINT, Interrupt.
0xffff3e25348a2050 in ?? ()

Breakpoint 10, 0x00000000410d0020 in ?? ()
ok 1 test_aarch64_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt
1..1
GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py']" -x /mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
==============================================================================
gromero@gromero0:/mnt/git/qemu_/build$


Also, using meson test:

gromero@gromero0:/mnt/git/qemu_/build$ cat /mnt/git/qemu_/build/meson-logs/testlog-thorough.txt
Log of Meson test suite run on 2025-08-25T13:54:13.544507

Inherited environment: SHELL=/bin/bash PYENV_SHELL=bash LC_ADDRESS=en_US.UTF-8 LC_NAME=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 PWD=/mnt/git/qemu_/build LOGNAME=gromero XDG_SESSION_TYPE=tty TVM_HOME=/home/gromero/git/tvm MOTD_SHOWN=pam HOME=/home/gromero LANG=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LS_COLORS='rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' SSH_CONNECTION='192.168.122.1 57300 192.168.122.158 22' LESSCLOSE='/usr/bin/lesspipe %s %s' XDG_SESSION_CLASS=user PYTHONPATH=/home/gromero/git/tvm/python: TERM=xterm-256color LC_IDENTIFICATION=en_US.UTF-8 LESSOPEN='| /usr/bin/lesspipe %s' QEMU_TEST_FLAKY_TESTS=1 USER=gromero SHLVL=1 LC_TELEPHONE=en_US.UTF-8 LC_MEASUREMENT=en_US.UTF-8 XDG_SESSION_ID=2738 XDG_RUNTIME_DIR=/run/user/1000 SSH_CLIENT='192.168.122.1 57300 22' PYENV_ROOT=/home/gromero/.pyenv LC_TIME=en_US.UTF-8 XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop PATH=/home/gromero/.pyenv/shims:/home/gromero/.pyenv/bin:/home/gromero/.local/bin:/home/gromero/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus SSH_TTY=/dev/pts/0 LC_NUMERIC=en_US.UTF-8 OLDPWD=/mnt/git/qemu_ _=/mnt/git/qemu_rr_fix/build/pyvenv/bin/meson

==================================== 1/1 =====================================
test:         qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_reverse_debug
start time:   13:54:13
duration:     2.65s
result:       exit status 0
command:      PYTHONPATH=/mnt/git/qemu_/python:/mnt/git/qemu_/tests/functional:/mnt/git/qemu_/build/pyvenv/lib/python3.10/site-packages MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 LD_LIBRARY_PATH=/mnt/git/qemu_/build/contrib/plugins:/mnt/git/qemu_/build/tests/tcg/plugins RUST_BACKTRACE=1 MALLOC_PERTURB_=243 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 QEMU_TEST_QEMU_BINARY=/mnt/git/qemu_/build/qemu-system-aarch64 QEMU_BUILD_ROOT=/mnt/git/qemu_/build UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 G_TEST_SLOW=1 MESON_TEST_ITERATION=1 QEMU_TEST_QEMU_IMG=/mnt/git/qemu_/build/qemu-img SPEED=thorough /mnt/git/qemu_/build/pyvenv/bin/python3 /mnt/git/qemu_/tests/guest-debug/run-test.py --gdb /usr/bin/gdb-multiarch --test /mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
----------------------------------- stdout -----------------------------------
Formatting '/mnt/git/qemu_/build/tests/functional/aarch64/test_aarch64_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt/scratch/disk.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=134217728 lazy_refcounts=off refcount_bits=16
TAP version 13
0x0000000040000000 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000008 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000010 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040080004 in ?? ()
0x00000000410d0000 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0024 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0000 in ?? ()
0x0000000040080004 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040000010 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000008 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000000 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000008 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000010 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040080004 in ?? ()
0x00000000410d0000 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0024 in ?? ()
Breakpoint 1 at 0x40000000
Breakpoint 2 at 0x40000004
Breakpoint 3 at 0x40000008
Breakpoint 4 at 0x4000000c
Breakpoint 5 at 0x40000010
Breakpoint 6 at 0x40000014
Breakpoint 7 at 0x40080000
Breakpoint 8 at 0x40080004
Breakpoint 9 at 0x410d0000
Breakpoint 10 at 0x410d0020

Program received signal SIGINT, Interrupt.
0xffff47bcc84a2050 in ?? ()

Breakpoint 10, 0x00000000410d0020 in ?? ()
ok 1 test_aarch64_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt
1..1
GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py']" -x /mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
==============================================================================


Ok:                1
Fail:              0
gromero@gromero0:/mnt/git/qemu_/build$


Any chance you got confused by no subtests-like output caused by 'exitcode'
used instead of ´tap'?


> Anyway, shifting to a different test harness here makes me wonder whether the whole reverse_debug test should maybe be rather moved to tests/tcg instead, where we already have the basic support for the stuff from tests/guest-debug/ ?
> The aarch64 would require a different guest payload, of course, in that case, so not sure whether it's feasible?

I think reverse_debugging is really a functional test. It requires GDB, yes,
but also QMP and booting a whole kernel and the feature itself makes me think
it's a functional test. I wouldn't move it to tcg-check just for the sake of
no adding a new way to run test in meson.build in functional tests.


Cheers,
Gustavo