[PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Gustavo Romero]

The goal of this series is to remove Avocado as a dependency for running
the reverse_debugging functional test.

This test, the last one I’m aware of that relies on Avocado, requires it
because of the need for GDB to test reverse stepping and continue.

In this series, we leveraged the run-test.py script used in the
check-tcg tests, making it a GDB runner capable of calling a test script
without spawning any VMs. In this configuration, the test script can
manage the VMs and also import gdb from the test script, making the
Python GDB API available inside the functional test.

The test is kept “skipped” for aarch64, ppc64, and x86_64, so it is
necessary to set QEMU_TEST_FLAKY_TESTS=1 in the test environment to
effectively run the test on these archs.

On aarch64, the test is flaky, but there is a fix that I’ve tested while
writing this series [0] that resolves it. On ppc64 and x86_64, the test
always fails: on ppc64, GDB gets a bogus PC, and on x86_64, the last
part of the test (reverse-continue) does not hit the last executed PC
(as it should) but instead jumps to the beginning of the code (fist PC
in forward order).

Thus, to run the reverse_debugging test effectively on aarch64:

$ export QEMU_TEST_FLAKY_TESTS=1
$ make check-functional 

Cheers,
Gustavo

[0] https://gitlab.com/qemu-project/qemu/-/issues/2921

Gustavo Romero (4):
  tests/guest-debug: Make QEMU optional in run-test.py
  tests/functional: Support tests that require a runner
  tests/functional: Mark main in QemuBaseTest class as a static method
  tests/functional: Adapt reverse_debugging to run w/o Avocado

 meson.build                                   |   4 +
 tests/functional/meson.build                  |  77 ++++++++-
 tests/functional/qemu_test/testcase.py        |   1 +
 tests/functional/reverse_debugging.py         | 158 +++++++++---------
 .../functional/test_aarch64_reverse_debug.py  |  19 +--
 tests/functional/test_ppc64_reverse_debug.py  |  17 +-
 tests/functional/test_x86_64_reverse_debug.py |  21 +--
 tests/guest-debug/run-test.py                 |  81 +++++----
 8 files changed, 223 insertions(+), 155 deletions(-)

-- 
2.34.1

[PATCH 1/4] tests/guest-debug: Make QEMU optional in run-test.py

[Gustavo Romero]

This commit makes QEMU optional in run-test.py, allowing it to be used
as a GDB runner, i.e., to call GDB and pass a test script to it without
launching QEMU. In this configuration, it is the test script’s duty to
configure and run the VMs that GDB connects to.

The --binary option continues to be required when --qemu is passed.
sys.argv now includes the full path to the test script in addition to
the script’s arguments, which allows unittest introspection to work
properly in case it is used in the test script.

Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
---
 tests/guest-debug/run-test.py | 81 +++++++++++++++++++----------------
 1 file changed, 45 insertions(+), 36 deletions(-)

diff --git a/tests/guest-debug/run-test.py b/tests/guest-debug/run-test.py
index 75e9c92e03..7fa17aedca 100755
--- a/tests/guest-debug/run-test.py
+++ b/tests/guest-debug/run-test.py
@@ -22,10 +22,10 @@
 def get_args():
     parser = argparse.ArgumentParser(description="A gdbstub test runner")
     parser.add_argument("--qemu", help="Qemu binary for test",
-                        required=True)
+                        required=False)
     parser.add_argument("--qargs", help="Qemu arguments for test")
     parser.add_argument("--binary", help="Binary to debug",
-                        required=True)
+                        required=False)
     parser.add_argument("--test", help="GDB test script")
     parser.add_argument('test_args', nargs='*',
                         help="Additional args for GDB test script. "
@@ -53,7 +53,7 @@ def log(output, msg):
 if __name__ == '__main__':
     args = get_args()
 
-    # Search for a gdb we can use
+    # Search for a gdb we can use.
     if not args.gdb:
         args.gdb = shutil.which("gdb-multiarch")
     if not args.gdb:
@@ -73,41 +73,49 @@ def log(output, msg):
     socket_dir = TemporaryDirectory("qemu-gdbstub")
     socket_name = os.path.join(socket_dir.name, "gdbstub.socket")
 
-    # Launch QEMU with binary
-    if "system" in args.qemu:
-        if args.no_suspend:
-            suspend = ''
-        else:
-            suspend = ' -S'
-        cmd = f'{args.qemu} {args.qargs} {args.binary}' \
-            f'{suspend} -gdb unix:path={socket_name},server=on'
-    else:
-        if args.no_suspend:
-            suspend = ',suspend=n'
-        else:
-            suspend = ''
-        cmd = f'{args.qemu} {args.qargs} -g {socket_name}{suspend}' \
-            f' {args.binary}'
-
-    log(output, "QEMU CMD: %s" % (cmd))
-    inferior = subprocess.Popen(shlex.split(cmd))
+    if args.qemu and not args.binary:
+        print("QEMU needs a binary to run, but no binary provided")
+        exit(-1)
 
-    # Now launch gdb with our test and collect the result
-    gdb_cmd = "%s %s" % (args.gdb, args.binary)
+    if args.qemu:
+        # Launch QEMU with binary.
+        if "system" in args.qemu:
+            if args.no_suspend:
+                suspend = ''
+            else:
+                suspend = ' -S'
+            cmd = f'{args.qemu} {args.qargs} {args.binary}' \
+                f'{suspend} -gdb unix:path={socket_name},server=on'
+        else:
+            if args.no_suspend:
+                suspend = ',suspend=n'
+            else:
+                suspend = ''
+            cmd = f'{args.qemu} {args.qargs} -g {socket_name}{suspend}' \
+                f' {args.binary}'
+
+        log(output, "QEMU CMD: %s" % (cmd))
+        inferior = subprocess.Popen(shlex.split(cmd))
+
+    # Now launch gdb with our test and collect the result.
+    gdb_cmd = args.gdb
+    if args.binary:
+        gdb_cmd += " %s" % (args.binary)
     if args.gdb_args:
         gdb_cmd += " %s" % (args.gdb_args)
-    # run quietly and ignore .gdbinit
+    # Run quietly and ignore .gdbinit.
     gdb_cmd += " -q -n -batch"
-    # disable pagination
+    # Disable pagination.
     gdb_cmd += " -ex 'set pagination off'"
-    # disable prompts in case of crash
+    # Disable prompts in case of crash.
     gdb_cmd += " -ex 'set confirm off'"
-    # connect to remote
-    gdb_cmd += " -ex 'target remote %s'" % (socket_name)
-    # finally the test script itself
+    # Connect automatically to remote only if QEMU is launched.
+    if args.qemu:
+        gdb_cmd += " -ex 'target remote %s'" % (socket_name)
+    # Finally the test script itself.
     if args.test:
-        if args.test_args:
-            gdb_cmd += f" -ex \"py sys.argv={args.test_args}\""
+        argv = [args.test] + args.test_args
+        gdb_cmd += f" -ex \"py sys.argv={argv}\""
         gdb_cmd += " -x %s" % (args.test)
 
 
@@ -129,10 +137,11 @@ def log(output, msg):
         log(output, "GDB crashed? (%d, %d) SKIPPING" % (result, result - 128))
         exit(0)
 
-    try:
-        inferior.wait(2)
-    except subprocess.TimeoutExpired:
-        log(output, "GDB never connected? Killed guest")
-        inferior.kill()
+    if args.qemu:
+        try:
+            inferior.wait(2)
+        except subprocess.TimeoutExpired:
+            log(output, "GDB never connected? Killed guest")
+            inferior.kill()
 
     exit(result)
-- 
2.34.1

Re: [PATCH 1/4] tests/guest-debug: Make QEMU optional in run-test.py

[Alex Bennée]

Gustavo Romero <gustavo.romero@linaro.org> writes:

> This commit makes QEMU optional in run-test.py, allowing it to be used
> as a GDB runner, i.e., to call GDB and pass a test script to it without
> launching QEMU. In this configuration, it is the test script’s duty to
> configure and run the VMs that GDB connects to.
>
> The --binary option continues to be required when --qemu is passed.
> sys.argv now includes the full path to the test script in addition to
> the script’s arguments, which allows unittest introspection to work
> properly in case it is used in the test script.
>
> Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
> ---
>  tests/guest-debug/run-test.py | 81 +++++++++++++++++++----------------
>  1 file changed, 45 insertions(+), 36 deletions(-)
>
> diff --git a/tests/guest-debug/run-test.py b/tests/guest-debug/run-test.py
> index 75e9c92e03..7fa17aedca 100755
> --- a/tests/guest-debug/run-test.py
> +++ b/tests/guest-debug/run-test.py
> @@ -22,10 +22,10 @@
>  def get_args():
>      parser = argparse.ArgumentParser(description="A gdbstub test runner")
>      parser.add_argument("--qemu", help="Qemu binary for test",
> -                        required=True)
> +                        required=False)
>      parser.add_argument("--qargs", help="Qemu arguments for test")
>      parser.add_argument("--binary", help="Binary to debug",
> -                        required=True)
> +                        required=False)

    parser.add_argument("--binary", help="Binary to debug",
                        required='--qemu' in sys.argv)

>      parser.add_argument("--test", help="GDB test script")
>      parser.add_argument('test_args', nargs='*',
>                          help="Additional args for GDB test script. "
> @@ -53,7 +53,7 @@ def log(output, msg):
>  if __name__ == '__main__':
>      args = get_args()
>  
> -    # Search for a gdb we can use
> +    # Search for a gdb we can use.
>      if not args.gdb:
>          args.gdb = shutil.which("gdb-multiarch")
>      if not args.gdb:
> @@ -73,41 +73,49 @@ def log(output, msg):
>      socket_dir = TemporaryDirectory("qemu-gdbstub")
>      socket_name = os.path.join(socket_dir.name, "gdbstub.socket")
>  
> -    # Launch QEMU with binary
> -    if "system" in args.qemu:
> -        if args.no_suspend:
> -            suspend = ''
> -        else:
> -            suspend = ' -S'
> -        cmd = f'{args.qemu} {args.qargs} {args.binary}' \
> -            f'{suspend} -gdb unix:path={socket_name},server=on'
> -    else:
> -        if args.no_suspend:
> -            suspend = ',suspend=n'
> -        else:
> -            suspend = ''
> -        cmd = f'{args.qemu} {args.qargs} -g {socket_name}{suspend}' \
> -            f' {args.binary}'
> -
> -    log(output, "QEMU CMD: %s" % (cmd))
> -    inferior = subprocess.Popen(shlex.split(cmd))
> +    if args.qemu and not args.binary:
> +        print("QEMU needs a binary to run, but no binary provided")
> +        exit(-1)

then we can avoid this.

>  
> -    # Now launch gdb with our test and collect the result
> -    gdb_cmd = "%s %s" % (args.gdb, args.binary)
> +    if args.qemu:
> +        # Launch QEMU with binary.
> +        if "system" in args.qemu:
> +            if args.no_suspend:
> +                suspend = ''
> +            else:
> +                suspend = ' -S'
> +            cmd = f'{args.qemu} {args.qargs} {args.binary}' \
> +                f'{suspend} -gdb unix:path={socket_name},server=on'
> +        else:
> +            if args.no_suspend:
> +                suspend = ',suspend=n'
> +            else:
> +                suspend = ''
> +            cmd = f'{args.qemu} {args.qargs} -g {socket_name}{suspend}' \
> +                f' {args.binary}'
> +
> +        log(output, "QEMU CMD: %s" % (cmd))
> +        inferior = subprocess.Popen(shlex.split(cmd))
> +
> +    # Now launch gdb with our test and collect the result.
> +    gdb_cmd = args.gdb
> +    if args.binary:
> +        gdb_cmd += " %s" % (args.binary)
>      if args.gdb_args:
>          gdb_cmd += " %s" % (args.gdb_args)
> -    # run quietly and ignore .gdbinit
> +    # Run quietly and ignore .gdbinit.
>      gdb_cmd += " -q -n -batch"
> -    # disable pagination
> +    # Disable pagination.
>      gdb_cmd += " -ex 'set pagination off'"
> -    # disable prompts in case of crash
> +    # Disable prompts in case of crash.
>      gdb_cmd += " -ex 'set confirm off'"

The re-formatting makes the diffs very noisy. If you want to clean up
the captilization of stuff do that in another commit.

> -    # connect to remote
> -    gdb_cmd += " -ex 'target remote %s'" % (socket_name)
> -    # finally the test script itself
> +    # Connect automatically to remote only if QEMU is launched.
> +    if args.qemu:
> +        gdb_cmd += " -ex 'target remote %s'" % (socket_name)
> +    # Finally the test script itself.
>      if args.test:
> -        if args.test_args:
> -            gdb_cmd += f" -ex \"py sys.argv={args.test_args}\""
> +        argv = [args.test] + args.test_args
> +        gdb_cmd += f" -ex \"py sys.argv={argv}\""
>          gdb_cmd += " -x %s" % (args.test)

I can see this echoes from:

   env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/meson test --suite thorough func-aarch64-aarch64_reverse_debug --verbose

Shows:

  GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py']" -x /home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py

But trying to piece that together on my the command line:

  env PYTHONPATH=/home/alex/lsrc/qemu.git/python:/home/alex/lsrc/qemu.git/tests/functional /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py']" -x /home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py
Python Exception <class 'ModuleNotFoundError'>: No module named 'pycotap'
Error occurred in Python: No module named 'pycotap'

What am I missing?

>  
>  
> @@ -129,10 +137,11 @@ def log(output, msg):
>          log(output, "GDB crashed? (%d, %d) SKIPPING" % (result, result - 128))
>          exit(0)
>  
> -    try:
> -        inferior.wait(2)
> -    except subprocess.TimeoutExpired:
> -        log(output, "GDB never connected? Killed guest")
> -        inferior.kill()
> +    if args.qemu:
> +        try:
> +            inferior.wait(2)
> +        except subprocess.TimeoutExpired:
> +            log(output, "GDB never connected? Killed guest")
> +            inferior.kill()
>  
>      exit(result)

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 1/4] tests/guest-debug: Make QEMU optional in run-test.py

[Gustavo Romero]

Hi Alex!

On 8/25/25 14:01, Alex Bennée wrote:
> Gustavo Romero <gustavo.romero@linaro.org> writes:
> 
>> This commit makes QEMU optional in run-test.py, allowing it to be used
>> as a GDB runner, i.e., to call GDB and pass a test script to it without
>> launching QEMU. In this configuration, it is the test script’s duty to
>> configure and run the VMs that GDB connects to.
>>
>> The --binary option continues to be required when --qemu is passed.
>> sys.argv now includes the full path to the test script in addition to
>> the script’s arguments, which allows unittest introspection to work
>> properly in case it is used in the test script.
>>
>> Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
>> ---
>>   tests/guest-debug/run-test.py | 81 +++++++++++++++++++----------------
>>   1 file changed, 45 insertions(+), 36 deletions(-)
>>
>> diff --git a/tests/guest-debug/run-test.py b/tests/guest-debug/run-test.py
>> index 75e9c92e03..7fa17aedca 100755
>> --- a/tests/guest-debug/run-test.py
>> +++ b/tests/guest-debug/run-test.py
>> @@ -22,10 +22,10 @@
>>   def get_args():
>>       parser = argparse.ArgumentParser(description="A gdbstub test runner")
>>       parser.add_argument("--qemu", help="Qemu binary for test",
>> -                        required=True)
>> +                        required=False)
>>       parser.add_argument("--qargs", help="Qemu arguments for test")
>>       parser.add_argument("--binary", help="Binary to debug",
>> -                        required=True)
>> +                        required=False)
> 
>      parser.add_argument("--binary", help="Binary to debug",
>                          required='--qemu' in sys.argv)
> 
>>       parser.add_argument("--test", help="GDB test script")
>>       parser.add_argument('test_args', nargs='*',
>>                           help="Additional args for GDB test script. "
>> @@ -53,7 +53,7 @@ def log(output, msg):
>>   if __name__ == '__main__':
>>       args = get_args()
>>   
>> -    # Search for a gdb we can use
>> +    # Search for a gdb we can use.
>>       if not args.gdb:
>>           args.gdb = shutil.which("gdb-multiarch")
>>       if not args.gdb:
>> @@ -73,41 +73,49 @@ def log(output, msg):
>>       socket_dir = TemporaryDirectory("qemu-gdbstub")
>>       socket_name = os.path.join(socket_dir.name, "gdbstub.socket")
>>   
>> -    # Launch QEMU with binary
>> -    if "system" in args.qemu:
>> -        if args.no_suspend:
>> -            suspend = ''
>> -        else:
>> -            suspend = ' -S'
>> -        cmd = f'{args.qemu} {args.qargs} {args.binary}' \
>> -            f'{suspend} -gdb unix:path={socket_name},server=on'
>> -    else:
>> -        if args.no_suspend:
>> -            suspend = ',suspend=n'
>> -        else:
>> -            suspend = ''
>> -        cmd = f'{args.qemu} {args.qargs} -g {socket_name}{suspend}' \
>> -            f' {args.binary}'
>> -
>> -    log(output, "QEMU CMD: %s" % (cmd))
>> -    inferior = subprocess.Popen(shlex.split(cmd))
>> +    if args.qemu and not args.binary:
>> +        print("QEMU needs a binary to run, but no binary provided")
>> +        exit(-1)
> 
> then we can avoid this.

Sure, thanks for the suggestion. I'm fixing it in v2.


>>   
>> -    # Now launch gdb with our test and collect the result
>> -    gdb_cmd = "%s %s" % (args.gdb, args.binary)
>> +    if args.qemu:
>> +        # Launch QEMU with binary.
>> +        if "system" in args.qemu:
>> +            if args.no_suspend:
>> +                suspend = ''
>> +            else:
>> +                suspend = ' -S'
>> +            cmd = f'{args.qemu} {args.qargs} {args.binary}' \
>> +                f'{suspend} -gdb unix:path={socket_name},server=on'
>> +        else:
>> +            if args.no_suspend:
>> +                suspend = ',suspend=n'
>> +            else:
>> +                suspend = ''
>> +            cmd = f'{args.qemu} {args.qargs} -g {socket_name}{suspend}' \
>> +                f' {args.binary}'
>> +
>> +        log(output, "QEMU CMD: %s" % (cmd))
>> +        inferior = subprocess.Popen(shlex.split(cmd))
>> +
>> +    # Now launch gdb with our test and collect the result.
>> +    gdb_cmd = args.gdb
>> +    if args.binary:
>> +        gdb_cmd += " %s" % (args.binary)
>>       if args.gdb_args:
>>           gdb_cmd += " %s" % (args.gdb_args)
>> -    # run quietly and ignore .gdbinit
>> +    # Run quietly and ignore .gdbinit.
>>       gdb_cmd += " -q -n -batch"
>> -    # disable pagination
>> +    # Disable pagination.
>>       gdb_cmd += " -ex 'set pagination off'"
>> -    # disable prompts in case of crash
>> +    # Disable prompts in case of crash.
>>       gdb_cmd += " -ex 'set confirm off'"
> 
> The re-formatting makes the diffs very noisy. If you want to clean up
> the captilization of stuff do that in another commit.

OK.


>> -    # connect to remote
>> -    gdb_cmd += " -ex 'target remote %s'" % (socket_name)
>> -    # finally the test script itself
>> +    # Connect automatically to remote only if QEMU is launched.
>> +    if args.qemu:
>> +        gdb_cmd += " -ex 'target remote %s'" % (socket_name)
>> +    # Finally the test script itself.
>>       if args.test:
>> -        if args.test_args:
>> -            gdb_cmd += f" -ex \"py sys.argv={args.test_args}\""
>> +        argv = [args.test] + args.test_args
>> +        gdb_cmd += f" -ex \"py sys.argv={argv}\""
>>           gdb_cmd += " -x %s" % (args.test)
> 
> I can see this echoes from:
> 
>     env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/meson test --suite thorough func-aarch64-aarch64_reverse_debug --verbose
> 
> Shows:
> 
>    GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py']" -x /home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py
> 
> But trying to piece that together on my the command line:
> 
>    env PYTHONPATH=/home/alex/lsrc/qemu.git/python:/home/alex/lsrc/qemu.git/tests/functional /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py']" -x /home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py
> Python Exception <class 'ModuleNotFoundError'>: No module named 'pycotap'
> Error occurred in Python: No module named 'pycotap'
> 
> What am I missing?

meson.build in tests/functional is properly setting PYTHONPATH in the env
when meson runs this command. libpython called from gdb binary will inspect
PYTHONPATH later. It's meson that knows where site-packages from the pyvenv
is located, so that's why meson is setting PYTHONPATH and that's why it works
when run by meson.

We should never need to put pieces together to run QEMU tests, I really
hate it (see my previous reply to Thomas about why using Avocado GDB for
test like this is not a good idea IMO).

That said, try to add the site-packages from your pyvenv set in your
build dir to PYTHONPATH:

gromero@gromero0:/mnt/git/qemu_/build$ ls -l ./pyvenv/lib/python3.10/site-packages
total 16
drwxrwxr-x  3 gromero gromero 4096 Aug 25 12:47 meson-1.8.1.dist-info
drwxrwxr-x 17 gromero gromero 4096 Aug 25 12:47 mesonbuild
drwxrwxr-x  3 gromero gromero 4096 Aug 25 12:47 pycotap
drwxrwxr-x  2 gromero gromero 4096 Aug 25 12:47 pycotap-1.3.1.dist-info

This is where the pycotap and other potential modules reside.


Cheers,
Gustavo

>>   
>>   
>> @@ -129,10 +137,11 @@ def log(output, msg):
>>           log(output, "GDB crashed? (%d, %d) SKIPPING" % (result, result - 128))
>>           exit(0)
>>   
>> -    try:
>> -        inferior.wait(2)
>> -    except subprocess.TimeoutExpired:
>> -        log(output, "GDB never connected? Killed guest")
>> -        inferior.kill()
>> +    if args.qemu:
>> +        try:
>> +            inferior.wait(2)
>> +        except subprocess.TimeoutExpired:
>> +            log(output, "GDB never connected? Killed guest")
>> +            inferior.kill()
>>   
>>       exit(result)
> 

Re: [PATCH 1/4] tests/guest-debug: Make QEMU optional in run-test.py

[Gustavo Romero]

Hi Alex,

On 8/25/25 14:26, Gustavo Romero wrote:
> Hi Alex!
> 
> On 8/25/25 14:01, Alex Bennée wrote:
>> Gustavo Romero <gustavo.romero@linaro.org> writes:
>>
>>> This commit makes QEMU optional in run-test.py, allowing it to be used
>>> as a GDB runner, i.e., to call GDB and pass a test script to it without
>>> launching QEMU. In this configuration, it is the test script’s duty to
>>> configure and run the VMs that GDB connects to.
>>>
>>> The --binary option continues to be required when --qemu is passed.
>>> sys.argv now includes the full path to the test script in addition to
>>> the script’s arguments, which allows unittest introspection to work
>>> properly in case it is used in the test script.
>>>
>>> Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
>>> ---
>>>   tests/guest-debug/run-test.py | 81 +++++++++++++++++++----------------
>>>   1 file changed, 45 insertions(+), 36 deletions(-)
>>>
>>> diff --git a/tests/guest-debug/run-test.py b/tests/guest-debug/run-test.py
>>> index 75e9c92e03..7fa17aedca 100755
>>> --- a/tests/guest-debug/run-test.py
>>> +++ b/tests/guest-debug/run-test.py
>>> @@ -22,10 +22,10 @@
>>>   def get_args():
>>>       parser = argparse.ArgumentParser(description="A gdbstub test runner")
>>>       parser.add_argument("--qemu", help="Qemu binary for test",
>>> -                        required=True)
>>> +                        required=False)
>>>       parser.add_argument("--qargs", help="Qemu arguments for test")
>>>       parser.add_argument("--binary", help="Binary to debug",
>>> -                        required=True)
>>> +                        required=False)
>>
>>      parser.add_argument("--binary", help="Binary to debug",
>>                          required='--qemu' in sys.argv)
>>
>>>       parser.add_argument("--test", help="GDB test script")
>>>       parser.add_argument('test_args', nargs='*',
>>>                           help="Additional args for GDB test script. "
>>> @@ -53,7 +53,7 @@ def log(output, msg):
>>>   if __name__ == '__main__':
>>>       args = get_args()
>>> -    # Search for a gdb we can use
>>> +    # Search for a gdb we can use.
>>>       if not args.gdb:
>>>           args.gdb = shutil.which("gdb-multiarch")
>>>       if not args.gdb:
>>> @@ -73,41 +73,49 @@ def log(output, msg):
>>>       socket_dir = TemporaryDirectory("qemu-gdbstub")
>>>       socket_name = os.path.join(socket_dir.name, "gdbstub.socket")
>>> -    # Launch QEMU with binary
>>> -    if "system" in args.qemu:
>>> -        if args.no_suspend:
>>> -            suspend = ''
>>> -        else:
>>> -            suspend = ' -S'
>>> -        cmd = f'{args.qemu} {args.qargs} {args.binary}' \
>>> -            f'{suspend} -gdb unix:path={socket_name},server=on'
>>> -    else:
>>> -        if args.no_suspend:
>>> -            suspend = ',suspend=n'
>>> -        else:
>>> -            suspend = ''
>>> -        cmd = f'{args.qemu} {args.qargs} -g {socket_name}{suspend}' \
>>> -            f' {args.binary}'
>>> -
>>> -    log(output, "QEMU CMD: %s" % (cmd))
>>> -    inferior = subprocess.Popen(shlex.split(cmd))
>>> +    if args.qemu and not args.binary:
>>> +        print("QEMU needs a binary to run, but no binary provided")
>>> +        exit(-1)
>>
>> then we can avoid this.
> 
> Sure, thanks for the suggestion. I'm fixing it in v2.
> 
> 
>>> -    # Now launch gdb with our test and collect the result
>>> -    gdb_cmd = "%s %s" % (args.gdb, args.binary)
>>> +    if args.qemu:
>>> +        # Launch QEMU with binary.
>>> +        if "system" in args.qemu:
>>> +            if args.no_suspend:
>>> +                suspend = ''
>>> +            else:
>>> +                suspend = ' -S'
>>> +            cmd = f'{args.qemu} {args.qargs} {args.binary}' \
>>> +                f'{suspend} -gdb unix:path={socket_name},server=on'
>>> +        else:
>>> +            if args.no_suspend:
>>> +                suspend = ',suspend=n'
>>> +            else:
>>> +                suspend = ''
>>> +            cmd = f'{args.qemu} {args.qargs} -g {socket_name}{suspend}' \
>>> +                f' {args.binary}'
>>> +
>>> +        log(output, "QEMU CMD: %s" % (cmd))
>>> +        inferior = subprocess.Popen(shlex.split(cmd))
>>> +
>>> +    # Now launch gdb with our test and collect the result.
>>> +    gdb_cmd = args.gdb
>>> +    if args.binary:
>>> +        gdb_cmd += " %s" % (args.binary)
>>>       if args.gdb_args:
>>>           gdb_cmd += " %s" % (args.gdb_args)
>>> -    # run quietly and ignore .gdbinit
>>> +    # Run quietly and ignore .gdbinit.
>>>       gdb_cmd += " -q -n -batch"
>>> -    # disable pagination
>>> +    # Disable pagination.
>>>       gdb_cmd += " -ex 'set pagination off'"
>>> -    # disable prompts in case of crash
>>> +    # Disable prompts in case of crash.
>>>       gdb_cmd += " -ex 'set confirm off'"
>>
>> The re-formatting makes the diffs very noisy. If you want to clean up
>> the captilization of stuff do that in another commit.
> 
> OK.
> 
> 
>>> -    # connect to remote
>>> -    gdb_cmd += " -ex 'target remote %s'" % (socket_name)
>>> -    # finally the test script itself
>>> +    # Connect automatically to remote only if QEMU is launched.
>>> +    if args.qemu:
>>> +        gdb_cmd += " -ex 'target remote %s'" % (socket_name)
>>> +    # Finally the test script itself.
>>>       if args.test:
>>> -        if args.test_args:
>>> -            gdb_cmd += f" -ex \"py sys.argv={args.test_args}\""
>>> +        argv = [args.test] + args.test_args
>>> +        gdb_cmd += f" -ex \"py sys.argv={argv}\""
>>>           gdb_cmd += " -x %s" % (args.test)
>>
>> I can see this echoes from:
>>
>>     env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/meson test --suite thorough func-aarch64-aarch64_reverse_debug --verbose
>>
>> Shows:
>>
>>    GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py']" -x /home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py
>>
>> But trying to piece that together on my the command line:
>>
>>    env PYTHONPATH=/home/alex/lsrc/qemu.git/python:/home/alex/lsrc/qemu.git/tests/functional /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py']" -x /home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py
>> Python Exception <class 'ModuleNotFoundError'>: No module named 'pycotap'
>> Error occurred in Python: No module named 'pycotap'
>>
>> What am I missing?
> 
> meson.build in tests/functional is properly setting PYTHONPATH in the env
> when meson runs this command. libpython called from gdb binary will inspect
> PYTHONPATH later. It's meson that knows where site-packages from the pyvenv
> is located, so that's why meson is setting PYTHONPATH and that's why it works
> when run by meson.
> 
> We should never need to put pieces together to run QEMU tests, I really
> hate it (see my previous reply to Thomas about why using Avocado GDB for
> test like this is not a good idea IMO).

ah, and I can think of some way to avoid having to put command pieces
together to re-run the test after we agree in the other more fundamental
aspects of this series, e.g., ok to kept it in tests/functional, ok to
use GDB Python API, etc.


Cheers,
Gustavo

> That said, try to add the site-packages from your pyvenv set in your
> build dir to PYTHONPATH:
> 
> gromero@gromero0:/mnt/git/qemu_/build$ ls -l ./pyvenv/lib/python3.10/site-packages
> total 16
> drwxrwxr-x  3 gromero gromero 4096 Aug 25 12:47 meson-1.8.1.dist-info
> drwxrwxr-x 17 gromero gromero 4096 Aug 25 12:47 mesonbuild
> drwxrwxr-x  3 gromero gromero 4096 Aug 25 12:47 pycotap
> drwxrwxr-x  2 gromero gromero 4096 Aug 25 12:47 pycotap-1.3.1.dist-info
> 
> This is where the pycotap and other potential modules reside.
> 
> 
> Cheers,
> Gustavo
> 
>>> @@ -129,10 +137,11 @@ def log(output, msg):
>>>           log(output, "GDB crashed? (%d, %d) SKIPPING" % (result, result - 128))
>>>           exit(0)
>>> -    try:
>>> -        inferior.wait(2)
>>> -    except subprocess.TimeoutExpired:
>>> -        log(output, "GDB never connected? Killed guest")
>>> -        inferior.kill()
>>> +    if args.qemu:
>>> +        try:
>>> +            inferior.wait(2)
>>> +        except subprocess.TimeoutExpired:
>>> +            log(output, "GDB never connected? Killed guest")
>>> +            inferior.kill()
>>>       exit(result)
>>
> 

Re: [PATCH 1/4] tests/guest-debug: Make QEMU optional in run-test.py

[Alex Bennée]

Gustavo Romero <gustavo.romero@linaro.org> writes:

> Hi Alex,
>
> On 8/25/25 14:26, Gustavo Romero wrote:
>> Hi Alex!
>> On 8/25/25 14:01, Alex Bennée wrote:
>>> Gustavo Romero <gustavo.romero@linaro.org> writes:
>>>
>>>> This commit makes QEMU optional in run-test.py, allowing it to be used
>>>> as a GDB runner, i.e., to call GDB and pass a test script to it without
>>>> launching QEMU. In this configuration, it is the test script’s duty to
>>>> configure and run the VMs that GDB connects to.
>>>>
>>>> The --binary option continues to be required when --qemu is passed.
>>>> sys.argv now includes the full path to the test script in addition to
>>>> the script’s arguments, which allows unittest introspection to work
>>>> properly in case it is used in the test script.
>>>>
>>>> Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
>>>> ---
>>>>   tests/guest-debug/run-test.py | 81 +++++++++++++++++++----------------
>>>>   1 file changed, 45 insertions(+), 36 deletions(-)
>>>>
>>>> diff --git a/tests/guest-debug/run-test.py b/tests/guest-debug/run-test.py
>>>> index 75e9c92e03..7fa17aedca 100755
>>>> --- a/tests/guest-debug/run-test.py
>>>> +++ b/tests/guest-debug/run-test.py
>>>> @@ -22,10 +22,10 @@
>>>>   def get_args():
>>>>       parser = argparse.ArgumentParser(description="A gdbstub test runner")
>>>>       parser.add_argument("--qemu", help="Qemu binary for test",
>>>> -                        required=True)
>>>> +                        required=False)
>>>>       parser.add_argument("--qargs", help="Qemu arguments for test")
>>>>       parser.add_argument("--binary", help="Binary to debug",
>>>> -                        required=True)
>>>> +                        required=False)
>>>
>>>      parser.add_argument("--binary", help="Binary to debug",
>>>                          required='--qemu' in sys.argv)
>>>
>>>>       parser.add_argument("--test", help="GDB test script")
>>>>       parser.add_argument('test_args', nargs='*',
>>>>                           help="Additional args for GDB test script. "
>>>> @@ -53,7 +53,7 @@ def log(output, msg):
>>>>   if __name__ == '__main__':
>>>>       args = get_args()
>>>> -    # Search for a gdb we can use
>>>> +    # Search for a gdb we can use.
>>>>       if not args.gdb:
>>>>           args.gdb = shutil.which("gdb-multiarch")
>>>>       if not args.gdb:
>>>> @@ -73,41 +73,49 @@ def log(output, msg):
>>>>       socket_dir = TemporaryDirectory("qemu-gdbstub")
>>>>       socket_name = os.path.join(socket_dir.name, "gdbstub.socket")
>>>> -    # Launch QEMU with binary
>>>> -    if "system" in args.qemu:
>>>> -        if args.no_suspend:
>>>> -            suspend = ''
>>>> -        else:
>>>> -            suspend = ' -S'
>>>> -        cmd = f'{args.qemu} {args.qargs} {args.binary}' \
>>>> -            f'{suspend} -gdb unix:path={socket_name},server=on'
>>>> -    else:
>>>> -        if args.no_suspend:
>>>> -            suspend = ',suspend=n'
>>>> -        else:
>>>> -            suspend = ''
>>>> -        cmd = f'{args.qemu} {args.qargs} -g {socket_name}{suspend}' \
>>>> -            f' {args.binary}'
>>>> -
>>>> -    log(output, "QEMU CMD: %s" % (cmd))
>>>> -    inferior = subprocess.Popen(shlex.split(cmd))
>>>> +    if args.qemu and not args.binary:
>>>> +        print("QEMU needs a binary to run, but no binary provided")
>>>> +        exit(-1)
>>>
>>> then we can avoid this.
>> Sure, thanks for the suggestion. I'm fixing it in v2.
>> 
>>>> -    # Now launch gdb with our test and collect the result
>>>> -    gdb_cmd = "%s %s" % (args.gdb, args.binary)
>>>> +    if args.qemu:
>>>> +        # Launch QEMU with binary.
>>>> +        if "system" in args.qemu:
>>>> +            if args.no_suspend:
>>>> +                suspend = ''
>>>> +            else:
>>>> +                suspend = ' -S'
>>>> +            cmd = f'{args.qemu} {args.qargs} {args.binary}' \
>>>> +                f'{suspend} -gdb unix:path={socket_name},server=on'
>>>> +        else:
>>>> +            if args.no_suspend:
>>>> +                suspend = ',suspend=n'
>>>> +            else:
>>>> +                suspend = ''
>>>> +            cmd = f'{args.qemu} {args.qargs} -g {socket_name}{suspend}' \
>>>> +                f' {args.binary}'
>>>> +
>>>> +        log(output, "QEMU CMD: %s" % (cmd))
>>>> +        inferior = subprocess.Popen(shlex.split(cmd))
>>>> +
>>>> +    # Now launch gdb with our test and collect the result.
>>>> +    gdb_cmd = args.gdb
>>>> +    if args.binary:
>>>> +        gdb_cmd += " %s" % (args.binary)
>>>>       if args.gdb_args:
>>>>           gdb_cmd += " %s" % (args.gdb_args)
>>>> -    # run quietly and ignore .gdbinit
>>>> +    # Run quietly and ignore .gdbinit.
>>>>       gdb_cmd += " -q -n -batch"
>>>> -    # disable pagination
>>>> +    # Disable pagination.
>>>>       gdb_cmd += " -ex 'set pagination off'"
>>>> -    # disable prompts in case of crash
>>>> +    # Disable prompts in case of crash.
>>>>       gdb_cmd += " -ex 'set confirm off'"
>>>
>>> The re-formatting makes the diffs very noisy. If you want to clean up
>>> the captilization of stuff do that in another commit.
>> OK.
>> 
>>>> -    # connect to remote
>>>> -    gdb_cmd += " -ex 'target remote %s'" % (socket_name)
>>>> -    # finally the test script itself
>>>> +    # Connect automatically to remote only if QEMU is launched.
>>>> +    if args.qemu:
>>>> +        gdb_cmd += " -ex 'target remote %s'" % (socket_name)
>>>> +    # Finally the test script itself.
>>>>       if args.test:
>>>> -        if args.test_args:
>>>> -            gdb_cmd += f" -ex \"py sys.argv={args.test_args}\""
>>>> +        argv = [args.test] + args.test_args
>>>> +        gdb_cmd += f" -ex \"py sys.argv={argv}\""
>>>>           gdb_cmd += " -x %s" % (args.test)
>>>
>>> I can see this echoes from:
>>>
>>>     env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/meson test --suite thorough func-aarch64-aarch64_reverse_debug --verbose
>>>
>>> Shows:
>>>
>>>    GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination
>>> off' -ex 'set confirm off' -ex "py
>>> sys.argv=['/home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py']"
>>> -x
>>> /home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py
>>>
>>> But trying to piece that together on my the command line:
>>>
>>>    env
>>> PYTHONPATH=/home/alex/lsrc/qemu.git/python:/home/alex/lsrc/qemu.git/tests/functional
>>> /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex
>>> 'set confirm off' -ex "py
>>> sys.argv=['/home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py']"
>>> -x
>>> /home/alex/lsrc/qemu.git/tests/functional/test_aarch64_reverse_debug.py
>>> Python Exception <class 'ModuleNotFoundError'>: No module named 'pycotap'
>>> Error occurred in Python: No module named 'pycotap'
>>>
>>> What am I missing?
>> meson.build in tests/functional is properly setting PYTHONPATH in
>> the env
>> when meson runs this command. libpython called from gdb binary will inspect
>> PYTHONPATH later. It's meson that knows where site-packages from the pyvenv
>> is located, so that's why meson is setting PYTHONPATH and that's why it works
>> when run by meson.
>> We should never need to put pieces together to run QEMU tests, I
>> really
>> hate it (see my previous reply to Thomas about why using Avocado GDB for
>> test like this is not a good idea IMO).
>
> ah, and I can think of some way to avoid having to put command pieces
> together to re-run the test after we agree in the other more fundamental
> aspects of this series, e.g., ok to kept it in tests/functional, ok to
> use GDB Python API, etc.

Hmm could meson devenv help?

>
> Cheers,
> Gustavo
>
>> That said, try to add the site-packages from your pyvenv set in your
>> build dir to PYTHONPATH:
>> gromero@gromero0:/mnt/git/qemu_/build$ ls -l
>> ./pyvenv/lib/python3.10/site-packages
>> total 16
>> drwxrwxr-x  3 gromero gromero 4096 Aug 25 12:47 meson-1.8.1.dist-info
>> drwxrwxr-x 17 gromero gromero 4096 Aug 25 12:47 mesonbuild
>> drwxrwxr-x  3 gromero gromero 4096 Aug 25 12:47 pycotap
>> drwxrwxr-x  2 gromero gromero 4096 Aug 25 12:47 pycotap-1.3.1.dist-info
>> This is where the pycotap and other potential modules reside.
>> Cheers,
>> Gustavo
>> 
>>>> @@ -129,10 +137,11 @@ def log(output, msg):
>>>>           log(output, "GDB crashed? (%d, %d) SKIPPING" % (result, result - 128))
>>>>           exit(0)
>>>> -    try:
>>>> -        inferior.wait(2)
>>>> -    except subprocess.TimeoutExpired:
>>>> -        log(output, "GDB never connected? Killed guest")
>>>> -        inferior.kill()
>>>> +    if args.qemu:
>>>> +        try:
>>>> +            inferior.wait(2)
>>>> +        except subprocess.TimeoutExpired:
>>>> +            log(output, "GDB never connected? Killed guest")
>>>> +            inferior.kill()
>>>>       exit(result)
>>>
>> 

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

[PATCH 2/4] tests/functional: Support tests that require a runner

[Gustavo Romero]

Add support for running tests that require a specific runner.

The test is specified via a tuple (name, runner, protocol), where name
is the test name as found in the tests/functional directory without the
'test_' prefix and the .py extension, runner is an array containing the
runner and any arguments required by the runner, and protocol is
the test protocol used by Meson to determine whether the test passed or
failed.

The test tuples are added to arrays that follow the current naming
logic but with the suffix '_with_runner' appended to their names. In
Meson it's not easy to select an element in an array at runtime based on
its type, so it's simpler to have a new array for these new test types
than use the current ones from the tests that don't require a runner,
and so avoid mixing strings and tuples in the same array.

Currently there is only one runner, the GDB runner, but more runners can
be defined and associated to a test via the tuple.

Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
---
 meson.build                  |  4 +++
 tests/functional/meson.build | 62 ++++++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+)

diff --git a/meson.build b/meson.build
index 50c774a195..8d482f0809 100644
--- a/meson.build
+++ b/meson.build
@@ -75,6 +75,10 @@ have_user = have_linux_user or have_bsd_user
 
 sh = find_program('sh')
 python = import('python').find_installation()
+# Meson python.get_path() on 'purelib' or 'platlib' doesn't properly return the
+# site-packages dir in pyvenv, so it is built manually.
+python_ver = python.language_version()
+python_site_packages = meson.build_root() / 'pyvenv/lib/python' + python_ver / 'site-packages'
 
 cc = meson.get_compiler('c')
 all_languages = ['c']
diff --git a/tests/functional/meson.build b/tests/functional/meson.build
index 311c6f1806..1f70b70fd4 100644
--- a/tests/functional/meson.build
+++ b/tests/functional/meson.build
@@ -349,6 +349,23 @@ tests_xtensa_system_thorough = [
   'xtensa_replay',
 ]
 
+# Tests that require a specific runner.
+gdb = find_program('gdb-multiarch', required: false)
+if gdb.found()
+    gdb_runner_script =  meson.project_source_root() + '/tests/guest-debug/run-test.py'
+    gdb_runner = [gdb_runner_script, '--gdb', gdb, '--test']
+
+    # A test with a runner is a tuple (name, runner, protocol).
+    # The tests must be elements of an array named like:
+    #
+    # test_<arch>_<mode=[system|linuxuser|bsduser]>_<speed=[quick|thorough]>_with_runner = [
+    #      ['test0', gdb_runner, 'exitcode'],
+    #      ...
+    # ]
+else
+    message('GDB multiarch not found, skipping functional tests that rely on it.')
+endif
+
 precache_all = []
 foreach speed : ['quick', 'thorough']
   foreach dir : target_dirs
@@ -372,9 +389,11 @@ foreach speed : ['quick', 'thorough']
       suites = ['func-quick', 'func-' + target_base]
       target_tests = get_variable('tests_' + target_base + '_' + sysmode + '_quick', []) \
                      + get_variable('tests_generic_' + sysmode)
+      target_tests_r = get_variable('tests_' + target_base + '_' + sysmode + '_quick_with_runner', [])
     else
       suites = ['func-' + speed, 'func-' + target_base + '-' + speed, speed]
       target_tests = get_variable('tests_' + target_base + '_' + sysmode + '_' + speed, [])
+      target_tests_r = get_variable('tests_' + target_base + '_' + sysmode + '_' + speed + '_with_runner', [])
     endif
 
     test_deps = [roms, keymap_targets]
@@ -423,6 +442,49 @@ foreach speed : ['quick', 'thorough']
            priority: test_timeouts.get(test, 90),
            suite: suites)
     endforeach
+
+    # Prepare tests that require a specific runner.
+    foreach test : target_tests_r
+      testname = '@0@-@1@'.format(target_base, test[0])
+      testfile = 'test_' + test[0] + '.py'
+      testpath = meson.current_source_dir() / testfile
+      teststamp = testname + '.tstamp'
+      testrunner  = test[1]
+      testproto = test[2]
+
+      test_precache_env = environment()
+      test_precache_env.set('QEMU_TEST_PRECACHE', meson.current_build_dir() / teststamp)
+      # python_site_packages, i.e., site packages from Python in pyvenv, is
+      # added to PYTHONPATH because some runners can run a program that has its
+      # own Python hooks that, by its turn, will search for modules based on
+      # PYTHONPATH independently of the Python used by the runner, like, for
+      # example, GDB using libpython.
+      test_precache_env.set('PYTHONPATH', meson.project_source_root() / 'python' +
+                                          ':' + meson.current_source_dir() +
+                                          ':' + python_site_packages)
+      precache = custom_target('func-precache-' + testname,
+                               output: teststamp,
+                               command: [testrunner, testpath],
+                               depend_files: files(testpath),
+                               build_by_default: false,
+                               env: test_precache_env)
+      precache_all += precache
+
+      # See comment above about python_site_packages in test_precache_env.
+      # Don't append to test_env since it will affect previous uses of it.
+      test_r_env = test_env
+      test_r_env.append('PYTHONPATH', python_site_packages)
+
+      test('func-' + testname,
+           python,
+           depends: [test_deps, test_emulator, emulator_modules, plugin_modules],
+           env: test_r_env,
+           args: [testrunner, testpath],
+           protocol: testproto,
+           timeout: test_timeouts.get(test[0], 90),
+           priority: test_timeouts.get(test[0], 90),
+           suite: suites)
+    endforeach
   endforeach
 endforeach
 
-- 
2.34.1

Re: [PATCH 2/4] tests/functional: Support tests that require a runner

[Alex Bennée]

Gustavo Romero <gustavo.romero@linaro.org> writes:

> Add support for running tests that require a specific runner.
>
> The test is specified via a tuple (name, runner, protocol), where name
> is the test name as found in the tests/functional directory without the
> 'test_' prefix and the .py extension, runner is an array containing the
> runner and any arguments required by the runner, and protocol is
> the test protocol used by Meson to determine whether the test passed or
> failed.
>
> The test tuples are added to arrays that follow the current naming
> logic but with the suffix '_with_runner' appended to their names. In
> Meson it's not easy to select an element in an array at runtime based on
> its type, so it's simpler to have a new array for these new test types
> than use the current ones from the tests that don't require a runner,
> and so avoid mixing strings and tuples in the same array.
>
> Currently there is only one runner, the GDB runner, but more runners can
> be defined and associated to a test via the tuple.
>
> Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
> ---
>  meson.build                  |  4 +++
>  tests/functional/meson.build | 62 ++++++++++++++++++++++++++++++++++++
>  2 files changed, 66 insertions(+)
>
> diff --git a/meson.build b/meson.build
> index 50c774a195..8d482f0809 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -75,6 +75,10 @@ have_user = have_linux_user or have_bsd_user
>  
>  sh = find_program('sh')
>  python = import('python').find_installation()
> +# Meson python.get_path() on 'purelib' or 'platlib' doesn't properly return the
> +# site-packages dir in pyvenv, so it is built manually.
> +python_ver = python.language_version()
> +python_site_packages = meson.build_root() / 'pyvenv/lib/python' + python_ver / 'site-packages'
>  
>  cc = meson.get_compiler('c')
>  all_languages = ['c']
> diff --git a/tests/functional/meson.build b/tests/functional/meson.build
> index 311c6f1806..1f70b70fd4 100644
> --- a/tests/functional/meson.build
> +++ b/tests/functional/meson.build
> @@ -349,6 +349,23 @@ tests_xtensa_system_thorough = [
>    'xtensa_replay',
>  ]
>  
> +# Tests that require a specific runner.
> +gdb = find_program('gdb-multiarch', required: false)
> +if gdb.found()
> +    gdb_runner_script =  meson.project_source_root() + '/tests/guest-debug/run-test.py'
> +    gdb_runner = [gdb_runner_script, '--gdb', gdb, '--test']
> +
> +    # A test with a runner is a tuple (name, runner, protocol).
> +    # The tests must be elements of an array named like:
> +    #
> +    # test_<arch>_<mode=[system|linuxuser|bsduser]>_<speed=[quick|thorough]>_with_runner = [
> +    #      ['test0', gdb_runner, 'exitcode'],
> +    #      ...
> +    # ]
> +else
> +    message('GDB multiarch not found, skipping functional tests that rely on it.')
> +endif
> +

We already have a --gdb option in configure which sets GDB= for
check-tcg - we should use the same configuration for these tests. You
may need to convert the configure probe to a meson-option.

>  precache_all = []
>  foreach speed : ['quick', 'thorough']
>    foreach dir : target_dirs
> @@ -372,9 +389,11 @@ foreach speed : ['quick', 'thorough']
>        suites = ['func-quick', 'func-' + target_base]
>        target_tests = get_variable('tests_' + target_base + '_' + sysmode + '_quick', []) \
>                       + get_variable('tests_generic_' + sysmode)
> +      target_tests_r = get_variable('tests_' + target_base + '_' + sysmode + '_quick_with_runner', [])
>      else
>        suites = ['func-' + speed, 'func-' + target_base + '-' + speed, speed]
>        target_tests = get_variable('tests_' + target_base + '_' + sysmode + '_' + speed, [])
> +      target_tests_r = get_variable('tests_' + target_base + '_' + sysmode + '_' + speed + '_with_runner', [])
>      endif
>  
>      test_deps = [roms, keymap_targets]
> @@ -423,6 +442,49 @@ foreach speed : ['quick', 'thorough']
>             priority: test_timeouts.get(test, 90),
>             suite: suites)
>      endforeach
> +
> +    # Prepare tests that require a specific runner.
> +    foreach test : target_tests_r
> +      testname = '@0@-@1@'.format(target_base, test[0])
> +      testfile = 'test_' + test[0] + '.py'
> +      testpath = meson.current_source_dir() / testfile
> +      teststamp = testname + '.tstamp'
> +      testrunner  = test[1]
> +      testproto = test[2]
> +
> +      test_precache_env = environment()
> +      test_precache_env.set('QEMU_TEST_PRECACHE', meson.current_build_dir() / teststamp)
> +      # python_site_packages, i.e., site packages from Python in pyvenv, is
> +      # added to PYTHONPATH because some runners can run a program that has its
> +      # own Python hooks that, by its turn, will search for modules based on
> +      # PYTHONPATH independently of the Python used by the runner, like, for
> +      # example, GDB using libpython.
> +      test_precache_env.set('PYTHONPATH', meson.project_source_root() / 'python' +
> +                                          ':' + meson.current_source_dir() +
> +                                          ':' + python_site_packages)
> +      precache = custom_target('func-precache-' + testname,
> +                               output: teststamp,
> +                               command: [testrunner, testpath],
> +                               depend_files: files(testpath),
> +                               build_by_default: false,
> +                               env: test_precache_env)
> +      precache_all += precache
> +
> +      # See comment above about python_site_packages in test_precache_env.
> +      # Don't append to test_env since it will affect previous uses of it.
> +      test_r_env = test_env
> +      test_r_env.append('PYTHONPATH', python_site_packages)
> +
> +      test('func-' + testname,
> +           python,
> +           depends: [test_deps, test_emulator, emulator_modules, plugin_modules],
> +           env: test_r_env,
> +           args: [testrunner, testpath],
> +           protocol: testproto,
> +           timeout: test_timeouts.get(test[0], 90),
> +           priority: test_timeouts.get(test[0], 90),
> +           suite: suites)
> +    endforeach
>    endforeach
>  endforeach

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 2/4] tests/functional: Support tests that require a runner

[Gustavo Romero]

Hi Alex,

On 8/25/25 13:50, Alex Bennée wrote:
> Gustavo Romero <gustavo.romero@linaro.org> writes:
> 
>> Add support for running tests that require a specific runner.
>>
>> The test is specified via a tuple (name, runner, protocol), where name
>> is the test name as found in the tests/functional directory without the
>> 'test_' prefix and the .py extension, runner is an array containing the
>> runner and any arguments required by the runner, and protocol is
>> the test protocol used by Meson to determine whether the test passed or
>> failed.
>>
>> The test tuples are added to arrays that follow the current naming
>> logic but with the suffix '_with_runner' appended to their names. In
>> Meson it's not easy to select an element in an array at runtime based on
>> its type, so it's simpler to have a new array for these new test types
>> than use the current ones from the tests that don't require a runner,
>> and so avoid mixing strings and tuples in the same array.
>>
>> Currently there is only one runner, the GDB runner, but more runners can
>> be defined and associated to a test via the tuple.
>>
>> Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
>> ---
>>   meson.build                  |  4 +++
>>   tests/functional/meson.build | 62 ++++++++++++++++++++++++++++++++++++
>>   2 files changed, 66 insertions(+)
>>
>> diff --git a/meson.build b/meson.build
>> index 50c774a195..8d482f0809 100644
>> --- a/meson.build
>> +++ b/meson.build
>> @@ -75,6 +75,10 @@ have_user = have_linux_user or have_bsd_user
>>   
>>   sh = find_program('sh')
>>   python = import('python').find_installation()
>> +# Meson python.get_path() on 'purelib' or 'platlib' doesn't properly return the
>> +# site-packages dir in pyvenv, so it is built manually.
>> +python_ver = python.language_version()
>> +python_site_packages = meson.build_root() / 'pyvenv/lib/python' + python_ver / 'site-packages'
>>   
>>   cc = meson.get_compiler('c')
>>   all_languages = ['c']
>> diff --git a/tests/functional/meson.build b/tests/functional/meson.build
>> index 311c6f1806..1f70b70fd4 100644
>> --- a/tests/functional/meson.build
>> +++ b/tests/functional/meson.build
>> @@ -349,6 +349,23 @@ tests_xtensa_system_thorough = [
>>     'xtensa_replay',
>>   ]
>>   
>> +# Tests that require a specific runner.
>> +gdb = find_program('gdb-multiarch', required: false)
>> +if gdb.found()
>> +    gdb_runner_script =  meson.project_source_root() + '/tests/guest-debug/run-test.py'
>> +    gdb_runner = [gdb_runner_script, '--gdb', gdb, '--test']
>> +
>> +    # A test with a runner is a tuple (name, runner, protocol).
>> +    # The tests must be elements of an array named like:
>> +    #
>> +    # test_<arch>_<mode=[system|linuxuser|bsduser]>_<speed=[quick|thorough]>_with_runner = [
>> +    #      ['test0', gdb_runner, 'exitcode'],
>> +    #      ...
>> +    # ]
>> +else
>> +    message('GDB multiarch not found, skipping functional tests that rely on it.')
>> +endif
>> +
> 
> We already have a --gdb option in configure which sets GDB= for
> check-tcg - we should use the same configuration for these tests. You
> may need to convert the configure probe to a meson-option.

True. I'll sort it out in v2. Afaics it will solve the issue Thomas found on
Fedora. As Daniel says gdb in Fedora already has all the arch built-in, the
detection in configure will work. Thanks.


Cheers,
Gustavo

>>   precache_all = []
>>   foreach speed : ['quick', 'thorough']
>>     foreach dir : target_dirs
>> @@ -372,9 +389,11 @@ foreach speed : ['quick', 'thorough']
>>         suites = ['func-quick', 'func-' + target_base]
>>         target_tests = get_variable('tests_' + target_base + '_' + sysmode + '_quick', []) \
>>                        + get_variable('tests_generic_' + sysmode)
>> +      target_tests_r = get_variable('tests_' + target_base + '_' + sysmode + '_quick_with_runner', [])
>>       else
>>         suites = ['func-' + speed, 'func-' + target_base + '-' + speed, speed]
>>         target_tests = get_variable('tests_' + target_base + '_' + sysmode + '_' + speed, [])
>> +      target_tests_r = get_variable('tests_' + target_base + '_' + sysmode + '_' + speed + '_with_runner', [])
>>       endif
>>   
>>       test_deps = [roms, keymap_targets]
>> @@ -423,6 +442,49 @@ foreach speed : ['quick', 'thorough']
>>              priority: test_timeouts.get(test, 90),
>>              suite: suites)
>>       endforeach
>> +
>> +    # Prepare tests that require a specific runner.
>> +    foreach test : target_tests_r
>> +      testname = '@0@-@1@'.format(target_base, test[0])
>> +      testfile = 'test_' + test[0] + '.py'
>> +      testpath = meson.current_source_dir() / testfile
>> +      teststamp = testname + '.tstamp'
>> +      testrunner  = test[1]
>> +      testproto = test[2]
>> +
>> +      test_precache_env = environment()
>> +      test_precache_env.set('QEMU_TEST_PRECACHE', meson.current_build_dir() / teststamp)
>> +      # python_site_packages, i.e., site packages from Python in pyvenv, is
>> +      # added to PYTHONPATH because some runners can run a program that has its
>> +      # own Python hooks that, by its turn, will search for modules based on
>> +      # PYTHONPATH independently of the Python used by the runner, like, for
>> +      # example, GDB using libpython.
>> +      test_precache_env.set('PYTHONPATH', meson.project_source_root() / 'python' +
>> +                                          ':' + meson.current_source_dir() +
>> +                                          ':' + python_site_packages)
>> +      precache = custom_target('func-precache-' + testname,
>> +                               output: teststamp,
>> +                               command: [testrunner, testpath],
>> +                               depend_files: files(testpath),
>> +                               build_by_default: false,
>> +                               env: test_precache_env)
>> +      precache_all += precache
>> +
>> +      # See comment above about python_site_packages in test_precache_env.
>> +      # Don't append to test_env since it will affect previous uses of it.
>> +      test_r_env = test_env
>> +      test_r_env.append('PYTHONPATH', python_site_packages)
>> +
>> +      test('func-' + testname,
>> +           python,
>> +           depends: [test_deps, test_emulator, emulator_modules, plugin_modules],
>> +           env: test_r_env,
>> +           args: [testrunner, testpath],
>> +           protocol: testproto,
>> +           timeout: test_timeouts.get(test[0], 90),
>> +           priority: test_timeouts.get(test[0], 90),
>> +           suite: suites)
>> +    endforeach
>>     endforeach
>>   endforeach
> 

[PATCH 3/4] tests/functional: Mark main in QemuBaseTest class as a static method

[Gustavo Romero]

The main() method in the QemuBaseTest class has no parameters but is
defined as a regular method. Currently, this does not cause any issues
because in the functional tests main() is always called directly from
QemuBaseTest (never from instances), but the way this method is defined
makes its signature wrong, implying a 'self'. Hence, it's best practice
to define such a method as a static method, so decorate it with
@staticmethod.

Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
---
 tests/functional/qemu_test/testcase.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
index 5caf7b13fe..fbeb171058 100644
--- a/tests/functional/qemu_test/testcase.py
+++ b/tests/functional/qemu_test/testcase.py
@@ -235,6 +235,7 @@ def tearDown(self):
         self.log.removeHandler(self._log_fh)
         self._log_fh.close()
 
+    @staticmethod
     def main():
         warnings.simplefilter("default")
         os.environ["PYTHONWARNINGS"] = "default"
-- 
2.34.1

Re: [PATCH 3/4] tests/functional: Mark main in QemuBaseTest class as a static method

[Daniel P. Berrangé]

On Tue, Aug 19, 2025 at 02:39:15PM +0000, Gustavo Romero wrote:
> The main() method in the QemuBaseTest class has no parameters but is
> defined as a regular method. Currently, this does not cause any issues
> because in the functional tests main() is always called directly from
> QemuBaseTest (never from instances), but the way this method is defined
> makes its signature wrong, implying a 'self'. Hence, it's best practice
> to define such a method as a static method, so decorate it with
> @staticmethod.
> 
> Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
> ---
>  tests/functional/qemu_test/testcase.py | 1 +
>  1 file changed, 1 insertion(+)

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH 4/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Gustavo Romero]

This commit removes Avocado as a dependency for running the
reverse_debugging test.

The main benefit, beyond eliminating an extra dependency, is that there
is no longer any need to handle GDB packets manually. This removes the
need for ad-hoc functions dealing with endianness and arch-specific
register numbers, making the test easier to read. The timeout variable
is also removed, since Meson now manages timeouts automatically.

The reverse_debugging test is now executed through a runner, because it
requires GDB in addition to QMP. The runner is responsible for invoking
GDB with the appropriate arguments and for passing the test script to
GDB.

Since the test now runs inside GDB, its output, particularly from
'stepi' commands, which cannot be disabled, interleaves with the TAP
output from unittest. To avoid this conflict, the test no longer uses
Meson’s 'tap' protocol and instead relies on the simpler 'exitcode'
protocol.

reverse_debugging is kept "skipped" for aarch64, ppc64, and x86_64, so
won't run unless QEMU_TEST_FLAKY_TESTS=1 is set in the test environment,
before running 'make check-functional' or 'meson test [...]'.

Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
---
 tests/functional/meson.build                  |  15 +-
 tests/functional/reverse_debugging.py         | 158 +++++++++---------
 .../functional/test_aarch64_reverse_debug.py  |  19 +--
 tests/functional/test_ppc64_reverse_debug.py  |  17 +-
 tests/functional/test_x86_64_reverse_debug.py |  21 +--
 5 files changed, 111 insertions(+), 119 deletions(-)

diff --git a/tests/functional/meson.build b/tests/functional/meson.build
index 1f70b70fd4..9eb11dad9f 100644
--- a/tests/functional/meson.build
+++ b/tests/functional/meson.build
@@ -93,7 +93,6 @@ tests_aarch64_system_thorough = [
   'aarch64_raspi3',
   'aarch64_raspi4',
   'aarch64_replay',
-  'aarch64_reverse_debug',
   'aarch64_rme_virt',
   'aarch64_rme_sbsaref',
   'aarch64_sbsaref',
@@ -254,7 +253,6 @@ tests_ppc64_system_thorough = [
   'ppc64_powernv',
   'ppc64_pseries',
   'ppc64_replay',
-  'ppc64_reverse_debug',
   'ppc64_tuxrun',
   'ppc64_mac99',
 ]
@@ -340,7 +338,6 @@ tests_x86_64_system_thorough = [
   'x86_64_hotplug_cpu',
   'x86_64_kvm_xen',
   'x86_64_replay',
-  'x86_64_reverse_debug',
   'x86_64_tuxrun',
 ]
 
@@ -362,6 +359,18 @@ if gdb.found()
     #      ['test0', gdb_runner, 'exitcode'],
     #      ...
     # ]
+
+    tests_aarch64_system_thorough_with_runner = [
+        ['aarch64_reverse_debug', gdb_runner, 'exitcode'],
+    ]
+
+    tests_ppc64_system_thorough_with_runner = [
+        ['ppc64_reverse_debug', gdb_runner, 'exitcode'],
+    ]
+
+    tests_x86_64_system_thorough_with_runner = [
+        ['x86_64_reverse_debug', gdb_runner, 'exitcode'],
+    ]
 else
     message('GDB multiarch not found, skipping functional tests that rely on it.')
 endif
diff --git a/tests/functional/reverse_debugging.py b/tests/functional/reverse_debugging.py
index f9a1d395f1..f4ee9a68bd 100644
--- a/tests/functional/reverse_debugging.py
+++ b/tests/functional/reverse_debugging.py
@@ -1,16 +1,26 @@
-# Reverse debugging test
-#
 # SPDX-License-Identifier: GPL-2.0-or-later
 #
+# Reverse debugging test
+#
 # Copyright (c) 2020 ISP RAS
+# Copyright (c) 2025 Linaro Limited
 #
 # Author:
 #  Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
+#  Gustavo Romero <gustavo.romero@linaro.org> (Run without Avocado)
 #
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
-import os
+
+try:
+    import gdb
+except ModuleNotFoundError:
+    from sys import exit
+    exit("This script must be launched via tests/guest-debug/run-test.py!")
 import logging
+import os
+import subprocess
+
 
 from qemu_test import LinuxKernelTest, get_qemu_img
 from qemu_test.ports import Ports
@@ -28,13 +38,9 @@ class ReverseDebugging(LinuxKernelTest):
     that the execution is stopped at the last of them.
     """
 
-    timeout = 10
     STEPS = 10
-    endian_is_le = True
 
     def run_vm(self, record, shift, args, replay_path, image_path, port):
-        from avocado.utils import datadrainer
-
         logger = logging.getLogger('replay')
         vm = self.get_vm(name='record' if record else 'replay')
         vm.set_console()
@@ -52,59 +58,46 @@ def run_vm(self, record, shift, args, replay_path, image_path, port):
         if args:
             vm.add_args(*args)
         vm.launch()
-        console_drainer = datadrainer.LineLogger(vm.console_socket.fileno(),
-                                    logger=self.log.getChild('console'),
-                                    stop_check=(lambda : not vm.is_running()))
-        console_drainer.start()
+
         return vm
 
     @staticmethod
-    def get_reg_le(g, reg):
-        res = g.cmd(b'p%x' % reg)
-        num = 0
-        for i in range(len(res))[-2::-2]:
-            num = 0x100 * num + int(res[i:i + 2], 16)
-        return num
+    def gdb_connect(host, port):
+        # Set debug on connection to get the qSupport string.
+        gdb.execute("set debug remote 1")
+        r = gdb.execute(f"target remote {host}:{port}", False, True)
+        gdb.execute("set debug remote 0")
+
+        return r
 
     @staticmethod
-    def get_reg_be(g, reg):
-        res = g.cmd(b'p%x' % reg)
-        return int(res, 16)
-
-    def get_reg(self, g, reg):
-        # value may be encoded in BE or LE order
-        if self.endian_is_le:
-            return self.get_reg_le(g, reg)
-        else:
-            return self.get_reg_be(g, reg)
+    def get_pc():
+        val = gdb.parse_and_eval("$pc")
+        pc = int(val)
 
-    def get_pc(self, g):
-        return self.get_reg(g, self.REG_PC)
+        return pc
 
-    def check_pc(self, g, addr):
-        pc = self.get_pc(g)
+    def check_pc(self, addr):
+        pc = self.get_pc()
         if pc != addr:
             self.fail('Invalid PC (read %x instead of %x)' % (pc, addr))
 
     @staticmethod
-    def gdb_step(g):
-        g.cmd(b's', b'T05thread:01;')
+    def gdb_step():
+        gdb.execute("stepi")
 
     @staticmethod
-    def gdb_bstep(g):
-        g.cmd(b'bs', b'T05thread:01;')
+    def gdb_bstep():
+        gdb.execute("reverse-stepi")
 
     @staticmethod
     def vm_get_icount(vm):
         return vm.qmp('query-replay')['return']['icount']
 
     def reverse_debugging(self, shift=7, args=None):
-        from avocado.utils import gdb
-        from avocado.utils import process
-
         logger = logging.getLogger('replay')
 
-        # create qcow2 for snapshots
+        # Create qcow2 for snapshots
         logger.info('creating qcow2 image for VM snapshots')
         image_path = os.path.join(self.workdir, 'disk.qcow2')
         qemu_img = get_qemu_img(self)
@@ -112,11 +105,11 @@ def reverse_debugging(self, shift=7, args=None):
             self.skipTest('Could not find "qemu-img", which is required to '
                           'create the temporary qcow2 image')
         cmd = '%s create -f qcow2 %s 128M' % (qemu_img, image_path)
-        process.run(cmd)
+        subprocess.run(cmd, shell=True)
 
         replay_path = os.path.join(self.workdir, 'replay.bin')
 
-        # record the log
+        # Record the log.
         vm = self.run_vm(True, shift, args, replay_path, image_path, -1)
         while self.vm_get_icount(vm) <= self.STEPS:
             pass
@@ -125,72 +118,71 @@ def reverse_debugging(self, shift=7, args=None):
 
         logger.info("recorded log with %s+ steps" % last_icount)
 
-        # replay and run debug commands
+        # Replay and run debug commands.
         with Ports() as ports:
             port = ports.find_free_port()
             vm = self.run_vm(False, shift, args, replay_path, image_path, port)
-        logger.info('connecting to gdbstub')
-        g = gdb.GDBRemote('127.0.0.1', port, False, False)
-        g.connect()
-        r = g.cmd(b'qSupported')
-        if b'qXfer:features:read+' in r:
-            g.cmd(b'qXfer:features:read:target.xml:0,ffb')
-        if b'ReverseStep+' not in r:
+        logger.info('Connecting to gdbstub')
+        r = self.gdb_connect('127.0.0.1', port)
+        if 'ReverseStep+' not in r:
             self.fail('Reverse step is not supported by QEMU')
-        if b'ReverseContinue+' not in r:
+        if 'ReverseContinue+' not in r:
             self.fail('Reverse continue is not supported by QEMU')
 
-        logger.info('stepping forward')
+        logger.info('Stepping forward')
         steps = []
-        # record first instruction addresses
+        # Record first instruction addresses.
         for _ in range(self.STEPS):
-            pc = self.get_pc(g)
-            logger.info('saving position %x' % pc)
+            pc = self.get_pc()
+            logger.info('Saving position %x' % pc)
             steps.append(pc)
-            self.gdb_step(g)
+            self.gdb_step()
 
-        # visit the recorded instruction in reverse order
-        logger.info('stepping backward')
+        # Visit the recorded instruction in reverse order.
+        logger.info('Stepping backward')
         for addr in steps[::-1]:
-            self.gdb_bstep(g)
-            self.check_pc(g, addr)
-            logger.info('found position %x' % addr)
+            self.gdb_bstep()
+            self.check_pc(addr)
+            logger.info('Found position %x' % addr)
 
-        # visit the recorded instruction in forward order
-        logger.info('stepping forward')
+        # Visit the recorded instruction in forward order.
+        logger.info('Stepping forward')
         for addr in steps:
-            self.check_pc(g, addr)
-            self.gdb_step(g)
-            logger.info('found position %x' % addr)
+            self.check_pc(addr)
+            self.gdb_step()
+            logger.info('Found position %x' % addr)
 
-        # set breakpoints for the instructions just stepped over
-        logger.info('setting breakpoints')
+        # Set breakpoints for the instructions just stepped over.
+        logger.info('Setting breakpoints')
         for addr in steps:
             # hardware breakpoint at addr with len=1
-            g.cmd(b'Z1,%x,1' % addr, b'OK')
+            gdb.execute(f"break *{hex(addr)}")
 
-        # this may hit a breakpoint if first instructions are executed
-        # again
-        logger.info('continuing execution')
+        # This may hit a breakpoint if first instructions are executed again.
+        logger.info('Continuing execution')
         vm.qmp('replay-break', icount=last_icount - 1)
-        # continue - will return after pausing
+        # continue - will return after pausing.
         # This could stop at the end and get a T02 return, or by
         # re-executing one of the breakpoints and get a T05 return.
-        g.cmd(b'c')
+        gdb.execute("continue")
         if self.vm_get_icount(vm) == last_icount - 1:
-            logger.info('reached the end (icount %s)' % (last_icount - 1))
+            logger.info('Reached the end (icount %s)' % (last_icount - 1))
         else:
-            logger.info('hit a breakpoint again at %x (icount %s)' %
+            logger.info('Hit a breakpoint again at %x (icount %s)' %
                         (self.get_pc(g), self.vm_get_icount(vm)))
 
-        logger.info('running reverse continue to reach %x' % steps[-1])
-        # reverse continue - will return after stopping at the breakpoint
-        g.cmd(b'bc', b'T05thread:01;')
+        logger.info('Running reverse continue to reach %x' % steps[-1])
+        # reverse continue - will return after stopping at the breakpoint.
+        gdb.execute("reverse-continue")
+
+        # Assume that none of the first instructions is executed again
+        # breaking the order of the breakpoints.
+        # steps[-1] is the first saved $pc in reverse order.
+        self.check_pc(steps[-1])
+        logger.info('Successfully reached %x' % steps[-1])
 
-        # assume that none of the first instructions is executed again
-        # breaking the order of the breakpoints
-        self.check_pc(g, steps[-1])
-        logger.info('successfully reached %x' % steps[-1])
+        logger.info('Exiting GDB and QEMU')
 
-        logger.info('exiting gdb and qemu')
         vm.shutdown()
+
+        self.assertTrue(True, "Test passed")
diff --git a/tests/functional/test_aarch64_reverse_debug.py b/tests/functional/test_aarch64_reverse_debug.py
index 58d4532835..8b6f82c227 100755
--- a/tests/functional/test_aarch64_reverse_debug.py
+++ b/tests/functional/test_aarch64_reverse_debug.py
@@ -1,27 +1,26 @@
-#!/usr/bin/env python3
-#
 # SPDX-License-Identifier: GPL-2.0-or-later
 #
-# Reverse debugging test
+# Reverse debugging test for aarch64
 #
 # Copyright (c) 2020 ISP RAS
+# Copyright (c) 2025 Linaro Limited
 #
 # Author:
 #  Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
+#  Gustavo Romero <gustavo.romero@linaro.org> (Run without Avocado)
 #
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from qemu_test import Asset, skipIfMissingImports, skipFlakyTest
+# ReverseDebugging must be imported always first because of the check
+# in it for not running this test without the GDB runner.
 from reverse_debugging import ReverseDebugging
+from qemu_test import Asset, skipFlakyTest
 
 
-@skipIfMissingImports('avocado.utils')
 class ReverseDebugging_AArch64(ReverseDebugging):
 
-    REG_PC = 32
-
-    KERNEL_ASSET = Asset(
+    ASSET_KERNEL = Asset(
         ('https://archives.fedoraproject.org/pub/archive/fedora/linux/'
          'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz'),
         '7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7')
@@ -30,9 +29,9 @@ class ReverseDebugging_AArch64(ReverseDebugging):
     def test_aarch64_virt(self):
         self.set_machine('virt')
         self.cpu = 'cortex-a53'
-        kernel_path = self.KERNEL_ASSET.fetch()
+        kernel_path = self.ASSET_KERNEL.fetch()
         self.reverse_debugging(args=('-kernel', kernel_path))
 
 
 if __name__ == '__main__':
-    ReverseDebugging.main()
+    ReverseDebugging_AArch64.main()
diff --git a/tests/functional/test_ppc64_reverse_debug.py b/tests/functional/test_ppc64_reverse_debug.py
index 5931adef5a..e70ca85d0a 100755
--- a/tests/functional/test_ppc64_reverse_debug.py
+++ b/tests/functional/test_ppc64_reverse_debug.py
@@ -1,41 +1,38 @@
-#!/usr/bin/env python3
-#
 # SPDX-License-Identifier: GPL-2.0-or-later
 #
-# Reverse debugging test
+# Reverse debugging test for ppc64
 #
 # Copyright (c) 2020 ISP RAS
+# Copyright (c) 2025 Linaro Limited
 #
 # Author:
 #  Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
+#  Gustavo Romero <gustavo.romero@linaro.org> (Run without Avocado)
 #
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from qemu_test import skipIfMissingImports, skipFlakyTest
+# ReverseDebugging must be imported always first because of the check
+# in it for not running this test without the GDB runner.
 from reverse_debugging import ReverseDebugging
+from qemu_test import skipFlakyTest
 
 
-@skipIfMissingImports('avocado.utils')
 class ReverseDebugging_ppc64(ReverseDebugging):
 
-    REG_PC = 0x40
-
     @skipFlakyTest("https://gitlab.com/qemu-project/qemu/-/issues/1992")
     def test_ppc64_pseries(self):
         self.set_machine('pseries')
         # SLOF branches back to its entry point, which causes this test
         # to take the 'hit a breakpoint again' path. That's not a problem,
         # just slightly different than the other machines.
-        self.endian_is_le = False
         self.reverse_debugging()
 
     @skipFlakyTest("https://gitlab.com/qemu-project/qemu/-/issues/1992")
     def test_ppc64_powernv(self):
         self.set_machine('powernv')
-        self.endian_is_le = False
         self.reverse_debugging()
 
 
 if __name__ == '__main__':
-    ReverseDebugging.main()
+    ReverseDebugging_ppc64.main()
diff --git a/tests/functional/test_x86_64_reverse_debug.py b/tests/functional/test_x86_64_reverse_debug.py
index d713e91e14..465f7e0abb 100755
--- a/tests/functional/test_x86_64_reverse_debug.py
+++ b/tests/functional/test_x86_64_reverse_debug.py
@@ -1,36 +1,31 @@
-#!/usr/bin/env python3
-#
 # SPDX-License-Identifier: GPL-2.0-or-later
 #
-# Reverse debugging test
+# Reverse debugging test for x86_64
 #
 # Copyright (c) 2020 ISP RAS
+# Copyright (c) 2025 Linaro Limited
 #
 # Author:
 #  Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
+#  Gustavo Romero <gustavo.romero@linaro.org> (Run without Avocado)
 #
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
-from qemu_test import skipIfMissingImports, skipFlakyTest
+# ReverseDebugging must be imported always first because of the check
+# in it for not running this test without the GDB runner.
 from reverse_debugging import ReverseDebugging
+from qemu_test import skipFlakyTest
 
 
-@skipIfMissingImports('avocado.utils')
 class ReverseDebugging_X86_64(ReverseDebugging):
 
-    REG_PC = 0x10
-    REG_CS = 0x12
-    def get_pc(self, g):
-        return self.get_reg_le(g, self.REG_PC) \
-            + self.get_reg_le(g, self.REG_CS) * 0x10
-
     @skipFlakyTest("https://gitlab.com/qemu-project/qemu/-/issues/2922")
     def test_x86_64_pc(self):
         self.set_machine('pc')
-        # start with BIOS only
+        # Start with BIOS only
         self.reverse_debugging()
 
 
 if __name__ == '__main__':
-    ReverseDebugging.main()
+    ReverseDebugging_X86_64.main()
-- 
2.34.1

Re: [PATCH 4/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Thomas Huth]

On 19/08/2025 16.39, Gustavo Romero wrote:
> This commit removes Avocado as a dependency for running the
> reverse_debugging test.
> 
> The main benefit, beyond eliminating an extra dependency, is that there
> is no longer any need to handle GDB packets manually. This removes the
> need for ad-hoc functions dealing with endianness and arch-specific
> register numbers, making the test easier to read. The timeout variable
> is also removed, since Meson now manages timeouts automatically.
> 
> The reverse_debugging test is now executed through a runner, because it
> requires GDB in addition to QMP. The runner is responsible for invoking
> GDB with the appropriate arguments and for passing the test script to
> GDB.
> 
> Since the test now runs inside GDB, its output, particularly from
> 'stepi' commands, which cannot be disabled, interleaves with the TAP
> output from unittest. To avoid this conflict, the test no longer uses
> Meson’s 'tap' protocol and instead relies on the simpler 'exitcode'
> protocol.
> 
> reverse_debugging is kept "skipped" for aarch64, ppc64, and x86_64, so
> won't run unless QEMU_TEST_FLAKY_TESTS=1 is set in the test environment,
> before running 'make check-functional' or 'meson test [...]'.
> 
> Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
> ---
...
> diff --git a/tests/functional/test_aarch64_reverse_debug.py b/tests/functional/test_aarch64_reverse_debug.py
> index 58d4532835..8b6f82c227 100755
> --- a/tests/functional/test_aarch64_reverse_debug.py
> +++ b/tests/functional/test_aarch64_reverse_debug.py
...
> -    KERNEL_ASSET = Asset(
> +    ASSET_KERNEL = Asset(
>           ('https://archives.fedoraproject.org/pub/archive/fedora/linux/'
>            'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz'),
>           '7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7')
> @@ -30,9 +29,9 @@ class ReverseDebugging_AArch64(ReverseDebugging):
>       def test_aarch64_virt(self):
>           self.set_machine('virt')
>           self.cpu = 'cortex-a53'
> -        kernel_path = self.KERNEL_ASSET.fetch()
> +        kernel_path = self.ASSET_KERNEL.fetch()
>           self.reverse_debugging(args=('-kernel', kernel_path))

Ouch, that "KERNEL_ASSET" is worth to be sent as a separate bugfix (without 
the proper "ASSET_" prefix, the asset won't be precached properly). Could 
you maybe send this as a separate patch, please?

  Thanks,
   Thomas

Re: [PATCH 4/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Gustavo Romero]

Hi Thomas,

On 8/25/25 07:34, Thomas Huth wrote:
> On 19/08/2025 16.39, Gustavo Romero wrote:
>> This commit removes Avocado as a dependency for running the
>> reverse_debugging test.
>>
>> The main benefit, beyond eliminating an extra dependency, is that there
>> is no longer any need to handle GDB packets manually. This removes the
>> need for ad-hoc functions dealing with endianness and arch-specific
>> register numbers, making the test easier to read. The timeout variable
>> is also removed, since Meson now manages timeouts automatically.
>>
>> The reverse_debugging test is now executed through a runner, because it
>> requires GDB in addition to QMP. The runner is responsible for invoking
>> GDB with the appropriate arguments and for passing the test script to
>> GDB.
>>
>> Since the test now runs inside GDB, its output, particularly from
>> 'stepi' commands, which cannot be disabled, interleaves with the TAP
>> output from unittest. To avoid this conflict, the test no longer uses
>> Meson’s 'tap' protocol and instead relies on the simpler 'exitcode'
>> protocol.
>>
>> reverse_debugging is kept "skipped" for aarch64, ppc64, and x86_64, so
>> won't run unless QEMU_TEST_FLAKY_TESTS=1 is set in the test environment,
>> before running 'make check-functional' or 'meson test [...]'.
>>
>> Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
>> ---
> ...
>> diff --git a/tests/functional/test_aarch64_reverse_debug.py b/tests/functional/test_aarch64_reverse_debug.py
>> index 58d4532835..8b6f82c227 100755
>> --- a/tests/functional/test_aarch64_reverse_debug.py
>> +++ b/tests/functional/test_aarch64_reverse_debug.py
> ...
>> -    KERNEL_ASSET = Asset(
>> +    ASSET_KERNEL = Asset(
>>           ('https://archives.fedoraproject.org/pub/archive/fedora/linux/'
>>            'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz'),
>>           '7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7')
>> @@ -30,9 +29,9 @@ class ReverseDebugging_AArch64(ReverseDebugging):
>>       def test_aarch64_virt(self):
>>           self.set_machine('virt')
>>           self.cpu = 'cortex-a53'
>> -        kernel_path = self.KERNEL_ASSET.fetch()
>> +        kernel_path = self.ASSET_KERNEL.fetch()
>>           self.reverse_debugging(args=('-kernel', kernel_path))
> 
> Ouch, that "KERNEL_ASSET" is worth to be sent as a separate bugfix (without the proper "ASSET_" prefix, the asset won't be precached properly). Could you maybe send this as a separate patch, please?

Sure!


Cheers,
Gustavo

Re: [PATCH 4/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Gustavo Romero]

Hi Thomas,

On 8/25/25 07:34, Thomas Huth wrote:
> On 19/08/2025 16.39, Gustavo Romero wrote:
>> This commit removes Avocado as a dependency for running the
>> reverse_debugging test.
>>
>> The main benefit, beyond eliminating an extra dependency, is that there
>> is no longer any need to handle GDB packets manually. This removes the
>> need for ad-hoc functions dealing with endianness and arch-specific
>> register numbers, making the test easier to read. The timeout variable
>> is also removed, since Meson now manages timeouts automatically.
>>
>> The reverse_debugging test is now executed through a runner, because it
>> requires GDB in addition to QMP. The runner is responsible for invoking
>> GDB with the appropriate arguments and for passing the test script to
>> GDB.
>>
>> Since the test now runs inside GDB, its output, particularly from
>> 'stepi' commands, which cannot be disabled, interleaves with the TAP
>> output from unittest. To avoid this conflict, the test no longer uses
>> Meson’s 'tap' protocol and instead relies on the simpler 'exitcode'
>> protocol.
>>
>> reverse_debugging is kept "skipped" for aarch64, ppc64, and x86_64, so
>> won't run unless QEMU_TEST_FLAKY_TESTS=1 is set in the test environment,
>> before running 'make check-functional' or 'meson test [...]'.
>>
>> Signed-off-by: Gustavo Romero <gustavo.romero@linaro.org>
>> ---
> ...
>> diff --git a/tests/functional/test_aarch64_reverse_debug.py b/tests/functional/test_aarch64_reverse_debug.py
>> index 58d4532835..8b6f82c227 100755
>> --- a/tests/functional/test_aarch64_reverse_debug.py
>> +++ b/tests/functional/test_aarch64_reverse_debug.py
> ...
>> -    KERNEL_ASSET = Asset(
>> +    ASSET_KERNEL = Asset(
>>           ('https://archives.fedoraproject.org/pub/archive/fedora/linux/'
>>            'releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz'),
>>           '7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7')
>> @@ -30,9 +29,9 @@ class ReverseDebugging_AArch64(ReverseDebugging):
>>       def test_aarch64_virt(self):
>>           self.set_machine('virt')
>>           self.cpu = 'cortex-a53'
>> -        kernel_path = self.KERNEL_ASSET.fetch()
>> +        kernel_path = self.ASSET_KERNEL.fetch()
>>           self.reverse_debugging(args=('-kernel', kernel_path))
> 
> Ouch, that "KERNEL_ASSET" is worth to be sent as a separate bugfix (without the proper "ASSET_" prefix, the asset won't be precached properly). Could you maybe send this as a separate patch, please?

Post here:

https://lists.gnu.org/archive/html/qemu-devel/2025-08/msg03763.html


Cheers,
Gustavo

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Thomas Huth]

On 19/08/2025 16.39, Gustavo Romero wrote:
> The goal of this series is to remove Avocado as a dependency for running
> the reverse_debugging functional test.
> 
> This test, the last one I’m aware of that relies on Avocado, requires it
> because of the need for GDB to test reverse stepping and continue.

  Hi!

Please note that there are currently also some efforts going on to extract 
the GDB part from avocado into a more self-contained python module called 
aautils, which might help here, too:

  https://github.com/avocado-framework/aautils/issues/82

> In this series, we leveraged the run-test.py script used in the
> check-tcg tests, making it a GDB runner capable of calling a test script
> without spawning any VMs. In this configuration, the test script can
> manage the VMs and also import gdb from the test script, making the
> Python GDB API available inside the functional test.
> 
> The test is kept “skipped” for aarch64, ppc64, and x86_64, so it is
> necessary to set QEMU_TEST_FLAKY_TESTS=1 in the test environment to
> effectively run the test on these archs.
> 
> On aarch64, the test is flaky, but there is a fix that I’ve tested while
> writing this series [0] that resolves it. On ppc64 and x86_64, the test
> always fails: on ppc64, GDB gets a bogus PC, and on x86_64, the last
> part of the test (reverse-continue) does not hit the last executed PC
> (as it should) but instead jumps to the beginning of the code (fist PC
> in forward order).

At least the x86 test used to work for me most of the time, it just fails 
occasionally. Same for the ppc64 test if I disable the powernv subtest there.

> Thus, to run the reverse_debugging test effectively on aarch64:
> 
> $ export QEMU_TEST_FLAKY_TESTS=1
> $ make check-functional

I gave it a try, but this did not work for me, the test was not run at all 
anymore. Are there any patches needed on top?

Anyway, shifting to a different test harness here makes me wonder whether 
the whole reverse_debug test should maybe be rather moved to tests/tcg 
instead, where we already have the basic support for the stuff from 
tests/guest-debug/ ?
The aarch64 would require a different guest payload, of course, in that 
case, so not sure whether it's feasible?

  Thomas

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Manos Pitsidianakis]

On Mon, Aug 25, 2025 at 1:31 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 19/08/2025 16.39, Gustavo Romero wrote:
> > The goal of this series is to remove Avocado as a dependency for running
> > the reverse_debugging functional test.
> >
> > This test, the last one I’m aware of that relies on Avocado, requires it
> > because of the need for GDB to test reverse stepping and continue.
>
>   Hi!
>
> Please note that there are currently also some efforts going on to extract
> the GDB part from avocado into a more self-contained python module called
> aautils, which might help here, too:
>
>   https://github.com/avocado-framework/aautils/issues/82
>

There's also https://github.com/cs01/pygdbmi which I have used before,
it seems like it does the same thing as avocado's gdb module but it
could be missing some functionality

> > In this series, we leveraged the run-test.py script used in the
> > check-tcg tests, making it a GDB runner capable of calling a test script
> > without spawning any VMs. In this configuration, the test script can
> > manage the VMs and also import gdb from the test script, making the
> > Python GDB API available inside the functional test.
> >
> > The test is kept “skipped” for aarch64, ppc64, and x86_64, so it is
> > necessary to set QEMU_TEST_FLAKY_TESTS=1 in the test environment to
> > effectively run the test on these archs.
> >
> > On aarch64, the test is flaky, but there is a fix that I’ve tested while
> > writing this series [0] that resolves it. On ppc64 and x86_64, the test
> > always fails: on ppc64, GDB gets a bogus PC, and on x86_64, the last
> > part of the test (reverse-continue) does not hit the last executed PC
> > (as it should) but instead jumps to the beginning of the code (fist PC
> > in forward order).
>
> At least the x86 test used to work for me most of the time, it just fails
> occasionally. Same for the ppc64 test if I disable the powernv subtest there.
>
> > Thus, to run the reverse_debugging test effectively on aarch64:
> >
> > $ export QEMU_TEST_FLAKY_TESTS=1
> > $ make check-functional
>
> I gave it a try, but this did not work for me, the test was not run at all
> anymore. Are there any patches needed on top?
>
> Anyway, shifting to a different test harness here makes me wonder whether
> the whole reverse_debug test should maybe be rather moved to tests/tcg
> instead, where we already have the basic support for the stuff from
> tests/guest-debug/ ?
> The aarch64 would require a different guest payload, of course, in that
> case, so not sure whether it's feasible?
>
>   Thomas
>
>

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Gustavo Romero]

Hi Manos!

On 8/25/25 08:00, Manos Pitsidianakis wrote:
> On Mon, Aug 25, 2025 at 1:31 PM Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 19/08/2025 16.39, Gustavo Romero wrote:
>>> The goal of this series is to remove Avocado as a dependency for running
>>> the reverse_debugging functional test.
>>>
>>> This test, the last one I’m aware of that relies on Avocado, requires it
>>> because of the need for GDB to test reverse stepping and continue.
>>
>>    Hi!
>>
>> Please note that there are currently also some efforts going on to extract
>> the GDB part from avocado into a more self-contained python module called
>> aautils, which might help here, too:
>>
>>    https://github.com/avocado-framework/aautils/issues/82
>>
> 
> There's also https://github.com/cs01/pygdbmi which I have used before,
> it seems like it does the same thing as avocado's gdb module but it
> could be missing some functionality

Thanks for the suggestion. I recall now that I've hastily looked at pygdbmi
when writing the mte.py test.

I don't think it misses any functionality for our current purposes here, but
it has its own runner so we'll be using two different GDB runners for the
tests. I´d like to avoid it and stick with the simplest runner which doesn't
add another module dependency. The same for the test interface.

In general, pygdbmi seems more than it would be necessary for the GDB tests
we have currently. I would like to keep the tests as easy as possible to
be executed manually and I think that adding another API rather than GDB's
own Python API, currently, will not help.


Cheers,
Gustavo

>>> In this series, we leveraged the run-test.py script used in the
>>> check-tcg tests, making it a GDB runner capable of calling a test script
>>> without spawning any VMs. In this configuration, the test script can
>>> manage the VMs and also import gdb from the test script, making the
>>> Python GDB API available inside the functional test.
>>>
>>> The test is kept “skipped” for aarch64, ppc64, and x86_64, so it is
>>> necessary to set QEMU_TEST_FLAKY_TESTS=1 in the test environment to
>>> effectively run the test on these archs.
>>>
>>> On aarch64, the test is flaky, but there is a fix that I’ve tested while
>>> writing this series [0] that resolves it. On ppc64 and x86_64, the test
>>> always fails: on ppc64, GDB gets a bogus PC, and on x86_64, the last
>>> part of the test (reverse-continue) does not hit the last executed PC
>>> (as it should) but instead jumps to the beginning of the code (fist PC
>>> in forward order).
>>
>> At least the x86 test used to work for me most of the time, it just fails
>> occasionally. Same for the ppc64 test if I disable the powernv subtest there.
>>
>>> Thus, to run the reverse_debugging test effectively on aarch64:
>>>
>>> $ export QEMU_TEST_FLAKY_TESTS=1
>>> $ make check-functional
>>
>> I gave it a try, but this did not work for me, the test was not run at all
>> anymore. Are there any patches needed on top?
>>
>> Anyway, shifting to a different test harness here makes me wonder whether
>> the whole reverse_debug test should maybe be rather moved to tests/tcg
>> instead, where we already have the basic support for the stuff from
>> tests/guest-debug/ ?
>> The aarch64 would require a different guest payload, of course, in that
>> case, so not sure whether it's feasible?
>>
>>    Thomas
>>
>>

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Gustavo Romero]

Hello, Thomas!

On 8/25/25 07:29, Thomas Huth wrote:
> On 19/08/2025 16.39, Gustavo Romero wrote:
>> The goal of this series is to remove Avocado as a dependency for running
>> the reverse_debugging functional test.
>>
>> This test, the last one I’m aware of that relies on Avocado, requires it
>> because of the need for GDB to test reverse stepping and continue.
> 
>   Hi!
> 
> Please note that there are currently also some efforts going on to extract the GDB part from avocado into a more self-contained python module called aautils, which might help here, too:
> 
>   https://github.com/avocado-framework/aautils/issues/82

Thanks, I was not aware of it. But note that the Avocado GDB API requires one
to deal with GDB packets directly, which is unnecessary for tests like reverse
debug. I think that in general, in QEMU, we should try to avoid using it as
much as possible because the tests became annoying to read without any strong
need for it.

In the commit message for 4/4 I mention that benefit of using the GDB Python
API instead.

I think we should aim to make the tests in QEMU _extremely_ easy to list, run,
and read. The original reverse_debugging test, using Avocado's GDB API, for
instance, is one example we should try to avoid. Say some developer is changing
something not related to RR but ends up breaking reverse_debugging. The developer
should at least be able to promptly run the QEMU, connect GDB, and reproduce the
issue manually following the exact commands in the test, without having to
consult the GDB protocol manual and to find out which GDB command is mapped to
which GDB packet.

In that sense I confess I'm not 100% happy with the output of reverse_debugging
yet, but one problem at a time :)


>> In this series, we leveraged the run-test.py script used in the
>> check-tcg tests, making it a GDB runner capable of calling a test script
>> without spawning any VMs. In this configuration, the test script can
>> manage the VMs and also import gdb from the test script, making the
>> Python GDB API available inside the functional test.
>>
>> The test is kept “skipped” for aarch64, ppc64, and x86_64, so it is
>> necessary to set QEMU_TEST_FLAKY_TESTS=1 in the test environment to
>> effectively run the test on these archs.
>>
>> On aarch64, the test is flaky, but there is a fix that I’ve tested while
>> writing this series [0] that resolves it. On ppc64 and x86_64, the test
>> always fails: on ppc64, GDB gets a bogus PC, and on x86_64, the last
>> part of the test (reverse-continue) does not hit the last executed PC
>> (as it should) but instead jumps to the beginning of the code (fist PC
>> in forward order).
> 
> At least the x86 test used to work for me most of the time, it just fails occasionally. Same for the ppc64 test if I disable the powernv subtest there.

ok, I'll double check the state on x86 in master. I understand we're not
testing reverse_debugging in our CI currently?

  >> Thus, to run the reverse_debugging test effectively on aarch64:
>>
>> $ export QEMU_TEST_FLAKY_TESTS=1
>> $ make check-functional
> 
> I gave it a try, but this did not work for me, the test was not run at all anymore. Are there any patches needed on top?

hmm that's odd. I'm able to run it with 'make check-functional' and with 'meson test'...

This is how I'm running it (let me know if I'm missing something):

gromero@gromero0:/mnt/git/qemu_$ b4 shazam 20250819143916.4138035-1-gustavo.romero@linaro.org
Grabbing thread from lore.kernel.org/all/20250819143916.4138035-1-gustavo.romero@linaro.org/t.mbox.gz
Checking for newer revisions
Grabbing search results from lore.kernel.org
Analyzing 9 messages in the thread
Looking for additional code-review trailers on lore.kernel.org
Analyzing 0 code-review messages
Checking attestation on all messages, may take a moment...
---
   ✓ [PATCH 1/4] tests/guest-debug: Make QEMU optional in run-test.py
   ✓ [PATCH 2/4] tests/functional: Support tests that require a runner
   ✓ [PATCH 3/4] tests/functional: Mark main in QemuBaseTest class as a static method
     + Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> (✓ DKIM/redhat.com)
   ✓ [PATCH 4/4] tests/functional: Adapt reverse_debugging to run w/o Avocado
   ---
   ✓ Signed: DKIM/linaro.org
---
Total patches: 4
---
Applying: tests/guest-debug: Make QEMU optional in run-test.py
Applying: tests/functional: Support tests that require a runner
Applying: tests/functional: Mark main in QemuBaseTest class as a static method
Applying: tests/functional: Adapt reverse_debugging to run w/o Avocado
gromero@gromero0:/mnt/git/qemu_$
gromero@gromero0:/mnt/git/qemu_/build$ ../configure --target-list=aarch64-softmmu --disable-docs
gromero@gromero0:/mnt/git/qemu_/build$ make -j 32
gromero@gromero0:/mnt/git/qemu_/build$ time make -j 15  check-functional
[1/2] Generating tests/functional/func-precache-aarch64-aarch64_reverse_debug with a custom command (wrapped by meson to set env)
2025-08-25 12:50:04,215 - qemu-test - INFO - Attempting to cache '/home/gromero/.cache/qemu/download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7'
2025-08-25 12:50:04,225 - qemu-test - DEBUG - Using cached asset /home/gromero/.cache/qemu/download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7 for https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/Everything/aarch64/os/images/pxeboot/vmlinuz
GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py']" -x /mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
[0/1] Running external command precache-functional (wrapped by meson to set env)
make[1]: Entering directory '/mnt/git/qemu_/build'
[1/6] Generating qemu-version.h with a custom command (wrapped by meson to capture output)
/mnt/git/qemu_/build/pyvenv/bin/meson test  --no-rebuild -t 1 --setup thorough  --num-processes 10 --print-errorlogs  --suite func  --suite func-quick  --suite func-thorough
  1/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_virt_gpu                   SKIP              1.95s   0 subtests passed
  2/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_smmu                       SKIP              4.17s   0 subtests passed
  3/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_raspi4                     OK               28.11s   2 subtests passed
  4/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_device_passthrough         OK               31.34s   1 subtests passed
  5/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_reverse_debug              OK               11.73s   <------ 'exitcode', not TAP, so no subtests-like output
  6/27 qemu:func-quick+func-aarch64 / func-aarch64-migration                                               OK                0.97s   2 subtests passed
  7/27 qemu:func-quick+func-aarch64 / func-aarch64-empty_cpu_model                                         OK                0.17s   1 subtests passed
[...]

gromero@gromero0:/mnt/git/qemu_/build$ grep -A 58 "= 5/27 =" /mnt/git/qemu_/build/meson-logs/testlog-thorough.txt
==================================== 5/27 ====================================
test:         qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_reverse_debug
start time:   13:44:21
duration:     11.73s
result:       exit status 0
command:      LD_LIBRARY_PATH=/mnt/git/qemu_/build/contrib/plugins:/mnt/git/qemu_/build/tests/tcg/plugins QEMU_BUILD_ROOT=/mnt/git/qemu_/build MALLOC_PERTURB_=138 G_TEST_SLOW=1 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 RUST_BACKTRACE=1 QEMU_TEST_QEMU_IMG=/mnt/git/qemu_/build/qemu-img MESON_TEST_ITERATION=1 PYTHONPATH=/mnt/git/qemu_/python:/mnt/git/qemu_/tests/functional:/mnt/git/qemu_/build/pyvenv/lib/python3.10/site-packages QEMU_TEST_QEMU_BINARY=/mnt/git/qemu_/build/qemu-system-aarch64 UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 /mnt/git/qemu_/build/pyvenv/bin/python3 /mnt/git/qemu_/tests/guest-debug/run-test.py --gdb /usr/bin/gdb-multiarch --test /mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
----------------------------------- stdout -----------------------------------
Formatting '/mnt/git/qemu_/build/tests/functional/aarch64/test_aarch64_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt/scratch/disk.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=134217728 lazy_refcounts=off refcount_bits=16
TAP version 13
0x0000000040000000 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000008 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000010 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040080004 in ?? ()
0x00000000410d0000 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0024 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0000 in ?? ()
0x0000000040080004 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040000010 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000008 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000000 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000008 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000010 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040080004 in ?? ()
0x00000000410d0000 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0024 in ?? ()
Breakpoint 1 at 0x40000000
Breakpoint 2 at 0x40000004
Breakpoint 3 at 0x40000008
Breakpoint 4 at 0x4000000c
Breakpoint 5 at 0x40000010
Breakpoint 6 at 0x40000014
Breakpoint 7 at 0x40080000
Breakpoint 8 at 0x40080004
Breakpoint 9 at 0x410d0000
Breakpoint 10 at 0x410d0020

Program received signal SIGINT, Interrupt.
0xffff3e25348a2050 in ?? ()

Breakpoint 10, 0x00000000410d0020 in ?? ()
ok 1 test_aarch64_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt
1..1
GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py']" -x /mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
==============================================================================
gromero@gromero0:/mnt/git/qemu_/build$


Also, using meson test:

gromero@gromero0:/mnt/git/qemu_/build$ cat /mnt/git/qemu_/build/meson-logs/testlog-thorough.txt
Log of Meson test suite run on 2025-08-25T13:54:13.544507

Inherited environment: SHELL=/bin/bash PYENV_SHELL=bash LC_ADDRESS=en_US.UTF-8 LC_NAME=en_US.UTF-8 LC_MONETARY=en_US.UTF-8 PWD=/mnt/git/qemu_/build LOGNAME=gromero XDG_SESSION_TYPE=tty TVM_HOME=/home/gromero/git/tvm MOTD_SHOWN=pam HOME=/home/gromero LANG=en_US.UTF-8 LC_PAPER=en_US.UTF-8 LS_COLORS='rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' SSH_CONNECTION='192.168.122.1 57300 192.168.122.158 22' LESSCLOSE='/usr/bin/lesspipe %s %s' XDG_SESSION_CLASS=user PYTHONPATH=/home/gromero/git/tvm/python: TERM=xterm-256color LC_IDENTIFICATION=en_US.UTF-8 LESSOPEN='| /usr/bin/lesspipe %s' QEMU_TEST_FLAKY_TESTS=1 USER=gromero SHLVL=1 LC_TELEPHONE=en_US.UTF-8 LC_MEASUREMENT=en_US.UTF-8 XDG_SESSION_ID=2738 XDG_RUNTIME_DIR=/run/user/1000 SSH_CLIENT='192.168.122.1 57300 22' PYENV_ROOT=/home/gromero/.pyenv LC_TIME=en_US.UTF-8 XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop PATH=/home/gromero/.pyenv/shims:/home/gromero/.pyenv/bin:/home/gromero/.local/bin:/home/gromero/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus SSH_TTY=/dev/pts/0 LC_NUMERIC=en_US.UTF-8 OLDPWD=/mnt/git/qemu_ _=/mnt/git/qemu_rr_fix/build/pyvenv/bin/meson

==================================== 1/1 =====================================
test:         qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-aarch64_reverse_debug
start time:   13:54:13
duration:     2.65s
result:       exit status 0
command:      PYTHONPATH=/mnt/git/qemu_/python:/mnt/git/qemu_/tests/functional:/mnt/git/qemu_/build/pyvenv/lib/python3.10/site-packages MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 LD_LIBRARY_PATH=/mnt/git/qemu_/build/contrib/plugins:/mnt/git/qemu_/build/tests/tcg/plugins RUST_BACKTRACE=1 MALLOC_PERTURB_=243 ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 QEMU_TEST_QEMU_BINARY=/mnt/git/qemu_/build/qemu-system-aarch64 QEMU_BUILD_ROOT=/mnt/git/qemu_/build UBSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 G_TEST_SLOW=1 MESON_TEST_ITERATION=1 QEMU_TEST_QEMU_IMG=/mnt/git/qemu_/build/qemu-img SPEED=thorough /mnt/git/qemu_/build/pyvenv/bin/python3 /mnt/git/qemu_/tests/guest-debug/run-test.py --gdb /usr/bin/gdb-multiarch --test /mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
----------------------------------- stdout -----------------------------------
Formatting '/mnt/git/qemu_/build/tests/functional/aarch64/test_aarch64_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt/scratch/disk.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=134217728 lazy_refcounts=off refcount_bits=16
TAP version 13
0x0000000040000000 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000008 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000010 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040080004 in ?? ()
0x00000000410d0000 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0024 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0000 in ?? ()
0x0000000040080004 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040000010 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000008 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000000 in ?? ()
0x0000000040000004 in ?? ()
0x0000000040000008 in ?? ()
0x000000004000000c in ?? ()
0x0000000040000010 in ?? ()
0x0000000040000014 in ?? ()
0x0000000040080000 in ?? ()
0x0000000040080004 in ?? ()
0x00000000410d0000 in ?? ()
0x00000000410d0020 in ?? ()
0x00000000410d0024 in ?? ()
Breakpoint 1 at 0x40000000
Breakpoint 2 at 0x40000004
Breakpoint 3 at 0x40000008
Breakpoint 4 at 0x4000000c
Breakpoint 5 at 0x40000010
Breakpoint 6 at 0x40000014
Breakpoint 7 at 0x40080000
Breakpoint 8 at 0x40080004
Breakpoint 9 at 0x410d0000
Breakpoint 10 at 0x410d0020

Program received signal SIGINT, Interrupt.
0xffff47bcc84a2050 in ?? ()

Breakpoint 10, 0x00000000410d0020 in ?? ()
ok 1 test_aarch64_reverse_debug.ReverseDebugging_AArch64.test_aarch64_virt
1..1
GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 'set confirm off' -ex "py sys.argv=['/mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py']" -x /mnt/git/qemu_/tests/functional/test_aarch64_reverse_debug.py
==============================================================================


Ok:                1
Fail:              0
gromero@gromero0:/mnt/git/qemu_/build$


Any chance you got confused by no subtests-like output caused by 'exitcode'
used instead of ´tap'?


> Anyway, shifting to a different test harness here makes me wonder whether the whole reverse_debug test should maybe be rather moved to tests/tcg instead, where we already have the basic support for the stuff from tests/guest-debug/ ?
> The aarch64 would require a different guest payload, of course, in that case, so not sure whether it's feasible?

I think reverse_debugging is really a functional test. It requires GDB, yes,
but also QMP and booting a whole kernel and the feature itself makes me think
it's a functional test. I wouldn't move it to tcg-check just for the sake of
no adding a new way to run test in meson.build in functional tests.


Cheers,
Gustavo

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Thomas Huth]

On 25/08/2025 16.04, Gustavo Romero wrote:
> Hello, Thomas!
> 
> On 8/25/25 07:29, Thomas Huth wrote:
>> On 19/08/2025 16.39, Gustavo Romero wrote:
>>> The goal of this series is to remove Avocado as a dependency for running
>>> the reverse_debugging functional test.
>>>
>>> This test, the last one I’m aware of that relies on Avocado, requires it
>>> because of the need for GDB to test reverse stepping and continue.
...
>> I gave it a try, but this did not work for me, the test was not run at all 
>> anymore. Are there any patches needed on top?
> 
> hmm that's odd. I'm able to run it with 'make check-functional' and with 
> 'meson test'...
> 
> This is how I'm running it (let me know if I'm missing something):
...
> gromero@gromero0:/mnt/git/qemu_$
> gromero@gromero0:/mnt/git/qemu_/build$ ../configure --target-list=aarch64- 
> softmmu --disable-docs
> gromero@gromero0:/mnt/git/qemu_/build$ make -j 32
> gromero@gromero0:/mnt/git/qemu_/build$ time make -j 15  check-functional
> [1/2] Generating tests/functional/func-precache-aarch64- 
> aarch64_reverse_debug with a custom command (wrapped by meson to set env)
> 2025-08-25 12:50:04,215 - qemu-test - INFO - Attempting to cache '/home/ 
> gromero/.cache/qemu/ 
> download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7'
> 2025-08-25 12:50:04,225 - qemu-test - DEBUG - Using cached asset /home/ 
> gromero/.cache/qemu/ 
> download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7 
> for https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/ 
> Everything/aarch64/os/images/pxeboot/vmlinuz
> GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off' -ex 
> 'set confirm off' -ex "py sys.argv=['/mnt/git/qemu_/tests/functional/ 
> test_aarch64_reverse_debug.py']" -x /mnt/git/qemu_/tests/functional/ 
> test_aarch64_reverse_debug.py
> [0/1] Running external command precache-functional (wrapped by meson to set 
> env)
> make[1]: Entering directory '/mnt/git/qemu_/build'
> [1/6] Generating qemu-version.h with a custom command (wrapped by meson to 
> capture output)
> /mnt/git/qemu_/build/pyvenv/bin/meson test  --no-rebuild -t 1 --setup 
> thorough  --num-processes 10 --print-errorlogs  --suite func  --suite func- 
> quick  --suite func-thorough
>   1/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64- 
> aarch64_virt_gpu                   SKIP              1.95s   0 subtests passed

I tried a couple of times now, and finally realized that it's the 
"gdb-multiarch" binary that is missing on Fedora. And as far as I can see, 
there is also no package that provides this on Fedora? So if we go ahead 
with your patches, this test will only run on certain distros that provide 
this binary.

  Thomas

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Alex Bennée]

Thomas Huth <thuth@redhat.com> writes:

> On 25/08/2025 16.04, Gustavo Romero wrote:
>> Hello, Thomas!
>> On 8/25/25 07:29, Thomas Huth wrote:
>>> On 19/08/2025 16.39, Gustavo Romero wrote:
>>>> The goal of this series is to remove Avocado as a dependency for running
>>>> the reverse_debugging functional test.
>>>>
>>>> This test, the last one I’m aware of that relies on Avocado, requires it
>>>> because of the need for GDB to test reverse stepping and continue.
> ...
>>> I gave it a try, but this did not work for me, the test was not run
>>> at all anymore. Are there any patches needed on top?
>> hmm that's odd. I'm able to run it with 'make check-functional' and
>> with 'meson test'...
>> This is how I'm running it (let me know if I'm missing something):
> ...
>> gromero@gromero0:/mnt/git/qemu_$
>> gromero@gromero0:/mnt/git/qemu_/build$ ../configure
>> --target-list=aarch64- softmmu --disable-docs
>> gromero@gromero0:/mnt/git/qemu_/build$ make -j 32
>> gromero@gromero0:/mnt/git/qemu_/build$ time make -j 15  check-functional
>> [1/2] Generating tests/functional/func-precache-aarch64-
>> aarch64_reverse_debug with a custom command (wrapped by meson to set
>> env)
>> 2025-08-25 12:50:04,215 - qemu-test - INFO - Attempting to cache
>> '/home/ gromero/.cache/qemu/
>> download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7'
>> 2025-08-25 12:50:04,225 - qemu-test - DEBUG - Using cached asset
>> /home/ gromero/.cache/qemu/
>> download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7
>> for
>> https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/
>> Everything/aarch64/os/images/pxeboot/vmlinuz
>> GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination
>> off' -ex 'set confirm off' -ex "py
>> sys.argv=['/mnt/git/qemu_/tests/functional/
>> test_aarch64_reverse_debug.py']" -x /mnt/git/qemu_/tests/functional/
>> test_aarch64_reverse_debug.py
>> [0/1] Running external command precache-functional (wrapped by meson
>> to set env)
>> make[1]: Entering directory '/mnt/git/qemu_/build'
>> [1/6] Generating qemu-version.h with a custom command (wrapped by
>> meson to capture output)
>> /mnt/git/qemu_/build/pyvenv/bin/meson test  --no-rebuild -t 1
>> --setup thorough  --num-processes 10 --print-errorlogs  --suite
>> func  --suite func- quick  --suite func-thorough
>>   1/27 qemu:func-thorough+func-aarch64-thorough+thorough /
>> func-aarch64- aarch64_virt_gpu                   SKIP             
>> 1.95s   0 subtests passed
>
> I tried a couple of times now, and finally realized that it's the
> "gdb-multiarch" binary that is missing on Fedora. And as far as I can
> see, there is also no package that provides this on Fedora? So if we
> go ahead with your patches, this test will only run on certain distros
> that provide this binary.

I've mentioned in another review comment we should use the same gdb as
set/detected from configure as we do for the check-tcg tests.

>
>  Thomas

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Manos Pitsidianakis]

On Tue, Aug 26, 2025 at 11:28 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Thomas Huth <thuth@redhat.com> writes:
>
> > On 25/08/2025 16.04, Gustavo Romero wrote:
> >> Hello, Thomas!
> >> On 8/25/25 07:29, Thomas Huth wrote:
> >>> On 19/08/2025 16.39, Gustavo Romero wrote:
> >>>> The goal of this series is to remove Avocado as a dependency for running
> >>>> the reverse_debugging functional test.
> >>>>
> >>>> This test, the last one I’m aware of that relies on Avocado, requires it
> >>>> because of the need for GDB to test reverse stepping and continue.
> > ...
> >>> I gave it a try, but this did not work for me, the test was not run
> >>> at all anymore. Are there any patches needed on top?
> >> hmm that's odd. I'm able to run it with 'make check-functional' and
> >> with 'meson test'...
> >> This is how I'm running it (let me know if I'm missing something):
> > ...
> >> gromero@gromero0:/mnt/git/qemu_$
> >> gromero@gromero0:/mnt/git/qemu_/build$ ../configure
> >> --target-list=aarch64- softmmu --disable-docs
> >> gromero@gromero0:/mnt/git/qemu_/build$ make -j 32
> >> gromero@gromero0:/mnt/git/qemu_/build$ time make -j 15  check-functional
> >> [1/2] Generating tests/functional/func-precache-aarch64-
> >> aarch64_reverse_debug with a custom command (wrapped by meson to set
> >> env)
> >> 2025-08-25 12:50:04,215 - qemu-test - INFO - Attempting to cache
> >> '/home/ gromero/.cache/qemu/
> >> download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7'
> >> 2025-08-25 12:50:04,225 - qemu-test - DEBUG - Using cached asset
> >> /home/ gromero/.cache/qemu/
> >> download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7
> >> for
> >> https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/
> >> Everything/aarch64/os/images/pxeboot/vmlinuz
> >> GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination
> >> off' -ex 'set confirm off' -ex "py
> >> sys.argv=['/mnt/git/qemu_/tests/functional/
> >> test_aarch64_reverse_debug.py']" -x /mnt/git/qemu_/tests/functional/
> >> test_aarch64_reverse_debug.py
> >> [0/1] Running external command precache-functional (wrapped by meson
> >> to set env)
> >> make[1]: Entering directory '/mnt/git/qemu_/build'
> >> [1/6] Generating qemu-version.h with a custom command (wrapped by
> >> meson to capture output)
> >> /mnt/git/qemu_/build/pyvenv/bin/meson test  --no-rebuild -t 1
> >> --setup thorough  --num-processes 10 --print-errorlogs  --suite
> >> func  --suite func- quick  --suite func-thorough
> >>   1/27 qemu:func-thorough+func-aarch64-thorough+thorough /
> >> func-aarch64- aarch64_virt_gpu                   SKIP
> >> 1.95s   0 subtests passed
> >
> > I tried a couple of times now, and finally realized that it's the
> > "gdb-multiarch" binary that is missing on Fedora. And as far as I can
> > see, there is also no package that provides this on Fedora? So if we
> > go ahead with your patches, this test will only run on certain distros
> > that provide this binary.
>
> I've mentioned in another review comment we should use the same gdb as
> set/detected from configure as we do for the check-tcg tests.


FWIW you can pass your own binary and command flags to
https://github.com/cs01/pygdbmi

It uses subprocess and wraps all stdout/stderr of gdb, like avocado.
So you can call it from the functional test directly.

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Daniel P. Berrangé]

On Tue, Aug 26, 2025 at 09:51:27AM +0200, Thomas Huth wrote:
> On 25/08/2025 16.04, Gustavo Romero wrote:
> > Hello, Thomas!
> > 
> > On 8/25/25 07:29, Thomas Huth wrote:
> > > On 19/08/2025 16.39, Gustavo Romero wrote:
> > > > The goal of this series is to remove Avocado as a dependency for running
> > > > the reverse_debugging functional test.
> > > > 
> > > > This test, the last one I’m aware of that relies on Avocado, requires it
> > > > because of the need for GDB to test reverse stepping and continue.
> ...
> > > I gave it a try, but this did not work for me, the test was not run
> > > at all anymore. Are there any patches needed on top?
> > 
> > hmm that's odd. I'm able to run it with 'make check-functional' and with
> > 'meson test'...
> > 
> > This is how I'm running it (let me know if I'm missing something):
> ...
> > gromero@gromero0:/mnt/git/qemu_$
> > gromero@gromero0:/mnt/git/qemu_/build$ ../configure
> > --target-list=aarch64- softmmu --disable-docs
> > gromero@gromero0:/mnt/git/qemu_/build$ make -j 32
> > gromero@gromero0:/mnt/git/qemu_/build$ time make -j 15  check-functional
> > [1/2] Generating tests/functional/func-precache-aarch64-
> > aarch64_reverse_debug with a custom command (wrapped by meson to set
> > env)
> > 2025-08-25 12:50:04,215 - qemu-test - INFO - Attempting to cache '/home/
> > gromero/.cache/qemu/
> > download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7'
> > 2025-08-25 12:50:04,225 - qemu-test - DEBUG - Using cached asset /home/
> > gromero/.cache/qemu/
> > download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7
> > for
> > https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/
> > Everything/aarch64/os/images/pxeboot/vmlinuz
> > GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off'
> > -ex 'set confirm off' -ex "py
> > sys.argv=['/mnt/git/qemu_/tests/functional/
> > test_aarch64_reverse_debug.py']" -x /mnt/git/qemu_/tests/functional/
> > test_aarch64_reverse_debug.py
> > [0/1] Running external command precache-functional (wrapped by meson to
> > set env)
> > make[1]: Entering directory '/mnt/git/qemu_/build'
> > [1/6] Generating qemu-version.h with a custom command (wrapped by meson
> > to capture output)
> > /mnt/git/qemu_/build/pyvenv/bin/meson test  --no-rebuild -t 1 --setup
> > thorough  --num-processes 10 --print-errorlogs  --suite func  --suite
> > func- quick  --suite func-thorough
> >   1/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-
> > aarch64_virt_gpu                   SKIP              1.95s   0 subtests
> > passed
> 
> I tried a couple of times now, and finally realized that it's the
> "gdb-multiarch" binary that is missing on Fedora. And as far as I can see,
> there is also no package that provides this on Fedora? So if we go ahead
> with your patches, this test will only run on certain distros that provide
> this binary.

'gdb-multiarch' is a command name invented by Debian.

On Fedora, the regular 'gdb' binary is built with support for
multiple architectures for the purpose of remote debugging.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Daniel P. Berrangé]

On Tue, Aug 26, 2025 at 03:10:42PM +0100, Daniel P. Berrangé wrote:
> On Tue, Aug 26, 2025 at 09:51:27AM +0200, Thomas Huth wrote:
> > On 25/08/2025 16.04, Gustavo Romero wrote:
> > > Hello, Thomas!
> > > 
> > > On 8/25/25 07:29, Thomas Huth wrote:
> > > > On 19/08/2025 16.39, Gustavo Romero wrote:
> > > > > The goal of this series is to remove Avocado as a dependency for running
> > > > > the reverse_debugging functional test.
> > > > > 
> > > > > This test, the last one I’m aware of that relies on Avocado, requires it
> > > > > because of the need for GDB to test reverse stepping and continue.
> > ...
> > > > I gave it a try, but this did not work for me, the test was not run
> > > > at all anymore. Are there any patches needed on top?
> > > 
> > > hmm that's odd. I'm able to run it with 'make check-functional' and with
> > > 'meson test'...
> > > 
> > > This is how I'm running it (let me know if I'm missing something):
> > ...
> > > gromero@gromero0:/mnt/git/qemu_$
> > > gromero@gromero0:/mnt/git/qemu_/build$ ../configure
> > > --target-list=aarch64- softmmu --disable-docs
> > > gromero@gromero0:/mnt/git/qemu_/build$ make -j 32
> > > gromero@gromero0:/mnt/git/qemu_/build$ time make -j 15  check-functional
> > > [1/2] Generating tests/functional/func-precache-aarch64-
> > > aarch64_reverse_debug with a custom command (wrapped by meson to set
> > > env)
> > > 2025-08-25 12:50:04,215 - qemu-test - INFO - Attempting to cache '/home/
> > > gromero/.cache/qemu/
> > > download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7'
> > > 2025-08-25 12:50:04,225 - qemu-test - DEBUG - Using cached asset /home/
> > > gromero/.cache/qemu/
> > > download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7
> > > for
> > > https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/
> > > Everything/aarch64/os/images/pxeboot/vmlinuz
> > > GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off'
> > > -ex 'set confirm off' -ex "py
> > > sys.argv=['/mnt/git/qemu_/tests/functional/
> > > test_aarch64_reverse_debug.py']" -x /mnt/git/qemu_/tests/functional/
> > > test_aarch64_reverse_debug.py
> > > [0/1] Running external command precache-functional (wrapped by meson to
> > > set env)
> > > make[1]: Entering directory '/mnt/git/qemu_/build'
> > > [1/6] Generating qemu-version.h with a custom command (wrapped by meson
> > > to capture output)
> > > /mnt/git/qemu_/build/pyvenv/bin/meson test  --no-rebuild -t 1 --setup
> > > thorough  --num-processes 10 --print-errorlogs  --suite func  --suite
> > > func- quick  --suite func-thorough
> > >   1/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-
> > > aarch64_virt_gpu                   SKIP              1.95s   0 subtests
> > > passed
> > 
> > I tried a couple of times now, and finally realized that it's the
> > "gdb-multiarch" binary that is missing on Fedora. And as far as I can see,
> > there is also no package that provides this on Fedora? So if we go ahead
> > with your patches, this test will only run on certain distros that provide
> > this binary.
> 
> 'gdb-multiarch' is a command name invented by Debian.
> 
> On Fedora, the regular 'gdb' binary is built with support for
> multiple architectures for the purpose of remote debugging.

The set of archs supported by GDB in Fedora is

  s390-linux-gnu
  powerpc-linux-gnu
  arm-linux-gnu
  aarch64-linux-gnu
  riscv64-linux-gnu
  x86_64-redhat-linux-gnu

you can query this with 'gdb --configuration' and parsing the
'--enable-targets' args  - I've not found a better way :-( 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Gustavo Romero]

Hi Daniel!

On 8/26/25 12:22, Daniel P. Berrangé wrote:
> On Tue, Aug 26, 2025 at 03:10:42PM +0100, Daniel P. Berrangé wrote:
>> On Tue, Aug 26, 2025 at 09:51:27AM +0200, Thomas Huth wrote:
>>> On 25/08/2025 16.04, Gustavo Romero wrote:
>>>> Hello, Thomas!
>>>>
>>>> On 8/25/25 07:29, Thomas Huth wrote:
>>>>> On 19/08/2025 16.39, Gustavo Romero wrote:
>>>>>> The goal of this series is to remove Avocado as a dependency for running
>>>>>> the reverse_debugging functional test.
>>>>>>
>>>>>> This test, the last one I’m aware of that relies on Avocado, requires it
>>>>>> because of the need for GDB to test reverse stepping and continue.
>>> ...
>>>>> I gave it a try, but this did not work for me, the test was not run
>>>>> at all anymore. Are there any patches needed on top?
>>>>
>>>> hmm that's odd. I'm able to run it with 'make check-functional' and with
>>>> 'meson test'...
>>>>
>>>> This is how I'm running it (let me know if I'm missing something):
>>> ...
>>>> gromero@gromero0:/mnt/git/qemu_$
>>>> gromero@gromero0:/mnt/git/qemu_/build$ ../configure
>>>> --target-list=aarch64- softmmu --disable-docs
>>>> gromero@gromero0:/mnt/git/qemu_/build$ make -j 32
>>>> gromero@gromero0:/mnt/git/qemu_/build$ time make -j 15  check-functional
>>>> [1/2] Generating tests/functional/func-precache-aarch64-
>>>> aarch64_reverse_debug with a custom command (wrapped by meson to set
>>>> env)
>>>> 2025-08-25 12:50:04,215 - qemu-test - INFO - Attempting to cache '/home/
>>>> gromero/.cache/qemu/
>>>> download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7'
>>>> 2025-08-25 12:50:04,225 - qemu-test - DEBUG - Using cached asset /home/
>>>> gromero/.cache/qemu/
>>>> download/7e1430b81c26bdd0da025eeb8fbd77b5dc961da4364af26e771bd39f379cbbf7
>>>> for
>>>> https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/29/
>>>> Everything/aarch64/os/images/pxeboot/vmlinuz
>>>> GDB CMD: /usr/bin/gdb-multiarch -q -n -batch -ex 'set pagination off'
>>>> -ex 'set confirm off' -ex "py
>>>> sys.argv=['/mnt/git/qemu_/tests/functional/
>>>> test_aarch64_reverse_debug.py']" -x /mnt/git/qemu_/tests/functional/
>>>> test_aarch64_reverse_debug.py
>>>> [0/1] Running external command precache-functional (wrapped by meson to
>>>> set env)
>>>> make[1]: Entering directory '/mnt/git/qemu_/build'
>>>> [1/6] Generating qemu-version.h with a custom command (wrapped by meson
>>>> to capture output)
>>>> /mnt/git/qemu_/build/pyvenv/bin/meson test  --no-rebuild -t 1 --setup
>>>> thorough  --num-processes 10 --print-errorlogs  --suite func  --suite
>>>> func- quick  --suite func-thorough
>>>>    1/27 qemu:func-thorough+func-aarch64-thorough+thorough / func-aarch64-
>>>> aarch64_virt_gpu                   SKIP              1.95s   0 subtests
>>>> passed
>>>
>>> I tried a couple of times now, and finally realized that it's the
>>> "gdb-multiarch" binary that is missing on Fedora. And as far as I can see,
>>> there is also no package that provides this on Fedora? So if we go ahead
>>> with your patches, this test will only run on certain distros that provide
>>> this binary.
>>
>> 'gdb-multiarch' is a command name invented by Debian.
>>
>> On Fedora, the regular 'gdb' binary is built with support for
>> multiple architectures for the purpose of remote debugging.
> 
> The set of archs supported by GDB in Fedora is
> 
>    s390-linux-gnu
>    powerpc-linux-gnu
>    arm-linux-gnu
>    aarch64-linux-gnu
>    riscv64-linux-gnu
>    x86_64-redhat-linux-gnu
> 
> you can query this with 'gdb --configuration' and parsing the
> '--enable-targets' args  - I've not found a better way :-(

Thanks a lot for checking it closely.

We already detect it via:

source_path/scripts/probe-gdb-support.py


in configure to detect the arches. That's what Alex is suggesting,
to use the gdb from configure in the test harness or scripts that
rely on GDB.


Cheers,
Gustavo

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Thomas Huth]

On 25/08/2025 16.04, Gustavo Romero wrote:
...
>> Anyway, shifting to a different test harness here makes me wonder whether 
>> the whole reverse_debug test should maybe be rather moved to tests/tcg 
>> instead, where we already have the basic support for the stuff from tests/ 
>> guest-debug/ ?
>> The aarch64 would require a different guest payload, of course, in that 
>> case, so not sure whether it's feasible?
> 
> I think reverse_debugging is really a functional test. It requires GDB, yes,
> but also QMP and booting a whole kernel and the feature itself makes me think
> it's a functional test. I wouldn't move it to tcg-check just for the sake of
> no adding a new way to run test in meson.build in functional tests.

Ok, fair point. But thinking about this twice, wouldn't it be nicer to keep 
the current test harness and simply launch the script from the test_* 
methods instead? You could redirect stdout of the script into a log file, so 
that it cannot mess with the TAP output of the test. Or is there something 
that I missed and this cannot work?

  Thomas

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Gustavo Romero]

Hi Thomas,

On 8/26/25 05:06, Thomas Huth wrote:
> On 25/08/2025 16.04, Gustavo Romero wrote:
> ...
>>> Anyway, shifting to a different test harness here makes me wonder whether the whole reverse_debug test should maybe be rather moved to tests/tcg instead, where we already have the basic support for the stuff from tests/ guest-debug/ ?
>>> The aarch64 would require a different guest payload, of course, in that case, so not sure whether it's feasible?
>>
>> I think reverse_debugging is really a functional test. It requires GDB, yes,
>> but also QMP and booting a whole kernel and the feature itself makes me think
>> it's a functional test. I wouldn't move it to tcg-check just for the sake of
>> no adding a new way to run test in meson.build in functional tests.
> 
> Ok, fair point. But thinking about this twice, wouldn't it be nicer to keep the current test harness and simply launch the script from the test_* methods instead?

If no additional module is used, then GDB needs to execute the test_* scripts via '-x' option
because only this way the script can do "import gdb", hence we need a runner that will call gdb
and pass the script to it.

Naturally, we can instead use a module like pygdbmi (as Manos suggested) or aautils (with the
caveats I mentioned about having to deal with GDB packets directly) that will simply wrap up
calling gdb and passing it the script with subprocess etc., so no need to augment the test harness.

The trade off I can see is: re-use the runner we use in check-tcg and augment the test
harness to accommodate it in the functional tests _vs_ use the "runner" from another module,
bring another python dependency and deal with another GDB API (I like using GDB native Python
API because it's simple, sufficient and self-contained, but I'm open to experiment with pygdbmi),
use another runner (other than the one in tests/guest-debug/test_gdbstub.py) and don't modify the
test harness.


> You could redirect stdout of the script into a log file, so that it cannot mess with the TAP output of the test. Or is there something that I missed and this cannot work?

Yeah, I really tried it but redirecting to a file using GDB Python
API didn't work :-(

I've tried to redirect GDB's output from the test script using
"set logging redirect on" but something is off. I even tried on GDB
master, but besides direct not taking effect (output is still printed
to the stdout) I got a GDB internal error when switching it off.

So, yes, it would work (even adding a # in the output messages so the
TAP protocol would simply ignore the lines at the same time we would
be able to see the output from GDB), but I was not able to make it work
with GDB native Python API, so turned to use the "exitcode".


Cheers,
Gustavo

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Daniel P. Berrangé]

On Mon, Aug 25, 2025 at 11:04:58AM -0300, Gustavo Romero wrote:
> Hello, Thomas!
> 
> On 8/25/25 07:29, Thomas Huth wrote:
> > On 19/08/2025 16.39, Gustavo Romero wrote:
> > > The goal of this series is to remove Avocado as a dependency for running
> > > the reverse_debugging functional test.
> > > 
> > > This test, the last one I’m aware of that relies on Avocado, requires it
> > > because of the need for GDB to test reverse stepping and continue.
> > 
> >   Hi!
> > 
> > Please note that there are currently also some efforts going on to extract the GDB part from avocado into a more self-contained python module called aautils, which might help here, too:
> > 
> >   https://github.com/avocado-framework/aautils/issues/82
> 
> Thanks, I was not aware of it. But note that the Avocado GDB API requires one
> to deal with GDB packets directly, which is unnecessary for tests like reverse
> debug. I think that in general, in QEMU, we should try to avoid using it as
> much as possible because the tests became annoying to read without any strong
> need for it.
> 
> In the commit message for 4/4 I mention that benefit of using the GDB Python
> API instead.
> 
> I think we should aim to make the tests in QEMU _extremely_ easy to list, run,
> and read.

Yes, I really like how your patch improves readability of the test
by using GDB commands rather than raw packets.




> > Anyway, shifting to a different test harness here makes me wonder whether the whole reverse_debug test should maybe be rather moved to tests/tcg instead, where we already have the basic support for the stuff from tests/guest-debug/ ?
> > The aarch64 would require a different guest payload, of course, in that case, so not sure whether it's feasible?
> 
> I think reverse_debugging is really a functional test. It requires GDB, yes,
> but also QMP and booting a whole kernel and the feature itself makes me think
> it's a functional test. I wouldn't move it to tcg-check just for the sake of
> no adding a new way to run test in meson.build in functional tests.

Agreed, this conceptually feels like it belongs as a functional test.



With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 0/4] tests/functional: Adapt reverse_debugging to run w/o Avocado

[Daniel P. Berrangé]

On Mon, Aug 25, 2025 at 12:29:41PM +0200, Thomas Huth wrote:
> On 19/08/2025 16.39, Gustavo Romero wrote:
> > The goal of this series is to remove Avocado as a dependency for running
> > the reverse_debugging functional test.
> > 
> > This test, the last one I’m aware of that relies on Avocado, requires it
> > because of the need for GDB to test reverse stepping and continue.
> 
>  Hi!
> 
> Please note that there are currently also some efforts going on to extract
> the GDB part from avocado into a more self-contained python module called
> aautils, which might help here, too:
> 
>  https://github.com/avocado-framework/aautils/issues/82

If we did go the route of wanting the pygdb code, I'd prefer we just
included it in tests/functional/qemu_test, rather than adding a dep
on aautils which doesn't seem broadly relevant. Or pygdb should just
be spun out as a standalone python project, as it doesn't need much
of a dep on anything else.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|