Async machine appliance for testing (python)

Async machine appliance for testing (python)

[John Snow]

WIth the upgrade to 3.12 imminent, now might be a good time for me to
revisit the work I started when I converted QMP from sync to an async
library. I have a draft of the library I wrote some time ago, but it's
probably in somewhat dodgy shape after a year or two of being ignored.

Before I dig in, I wanted to ask if this would be a helpful project.
The main selling points would be:

(1) Ability to split it out from the main QEMU tree, like QMP was, for
use in external testing
(2) Better stream management so console information is consumed and
logged reliably without deadlocks due to pipes hitting their buffer
limit
(3) Removing multiprocess and multithread processing of console and
stdout/stderr logging from test suites in favor of using async
user-threads for (hopefully) simpler, more straightforward processing
and customization.
(4) Ability to add listeners/event handlers to watch for specific
patterns and trigger asynchronously when they occur; i.e. ability to
handle and watch for specific patterns in an arbitrary order without
deadlocking your test
(5) Centralized timeout setting and handling.

I recall in the past that I heard some interest from various folks,
can you just confirm this is still interesting for you at this time?

Thanks,
--js

Re: Async machine appliance for testing (python)

[Gerd Hoffmann]

  Hi,

> Before I dig in, I wanted to ask if this would be a helpful project.
> The main selling points would be:
> 
> (1) Ability to split it out from the main QEMU tree, like QMP was, for
> use in external testing

I'd still love to see this happening.

> (4) Ability to add listeners/event handlers to watch for specific
> patterns and trigger asynchronously when they occur; i.e. ability to
> handle and watch for specific patterns in an arbitrary order without
> deadlocking your test

That sounds very useful to me.

take care,
  Gerd

Re: Async machine appliance for testing (python)

[John Snow]

On Wed, Sep 2, 2026 at 5:36 AM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>   Hi,
>
> > Before I dig in, I wanted to ask if this would be a helpful project.
> > The main selling points would be:
> >
> > (1) Ability to split it out from the main QEMU tree, like QMP was, for
> > use in external testing
>
> I'd still love to see this happening.
>
> > (4) Ability to add listeners/event handlers to watch for specific
> > patterns and trigger asynchronously when they occur; i.e. ability to
> > handle and watch for specific patterns in an arbitrary order without
> > deadlocking your test
>
> That sounds very useful to me.
>
> take care,
>   Gerd

Got it. When I handle some of the more obviously pressing issues with
our build system as a result of the py312 upgrade, I will work to
resuscitate this project and I will reach out to you (hopefully)
before long for some early feedback on the API.

One of the big challenges here is designing an API that allows us to
safely split it out from the QEMU tree without boxing ourselves into a
corner with options management. QMP is relatively stable and so was
easy to fork out, but the machine appliance makes a lot more
assumptions about how to manage the QEMU process and there's a bit of
design risk inherent to splitting it out. I think I have some good
ideas for how to do this without accidentally reinventing libvirt
again, but we'll see.

Thanks,
--js

Re: Async machine appliance for testing (python)

[Daniel P. Berrangé]

On Mon, Aug 31, 2026 at 04:51:36PM -0400, John Snow wrote:
> WIth the upgrade to 3.12 imminent, now might be a good time for me to
> revisit the work I started when I converted QMP from sync to an async
> library. I have a draft of the library I wrote some time ago, but it's
> probably in somewhat dodgy shape after a year or two of being ignored.
> 
> Before I dig in, I wanted to ask if this would be a helpful project.
> The main selling points would be:
> 
> (1) Ability to split it out from the main QEMU tree, like QMP was, for
> use in external testing
> (2) Better stream management so console information is consumed and
> logged reliably without deadlocks due to pipes hitting their buffer
> limit

I can't remember the specific details, but I do recall some
scenarios where it would have been valuable to have the console
output continuously streamed to the log file on disk, while
at the same time retainining the ability to do expected
output matching from the API.

> (3) Removing multiprocess and multithread processing of console and
> stdout/stderr logging from test suites in favor of using async
> user-threads for (hopefully) simpler, more straightforward processing
> and customization.

Which tests were you refering to here ?  I didn't think we had
python tests using multiprocess/multithread ?

Tangentially related, in debugging failures of the functional
test suite, and to a lesser extent I/O tests / qmp-shell, I've
often found myself heavily confused by the asyncio call paths
and control flow, making it really hard to debug.

In particular I've never fully figured out how to get qmp-shell
to have a clean exit path without stack traces being thrown.

I'm unclear if this is related to our having QEMUMachine class
expose a sync API on top of an async QMP class (square peg in
a round hole situation), or if this is just something inherent
to Python asyncio APIs, or probably a bit of both ?

If switching to a fully async API throughout the stack makes
debugging simpler, I'd welcome it.

> (4) Ability to add listeners/event handlers to watch for specific
> patterns and trigger asynchronously when they occur; i.e. ability to
> handle and watch for specific patterns in an arbitrary order without
> deadlocking your test
> (5) Centralized timeout setting and handling.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: Async machine appliance for testing (python)

[John Snow]

On Wed, Sep 2, 2026 at 1:50 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Mon, Aug 31, 2026 at 04:51:36PM -0400, John Snow wrote:
> > WIth the upgrade to 3.12 imminent, now might be a good time for me to
> > revisit the work I started when I converted QMP from sync to an async
> > library. I have a draft of the library I wrote some time ago, but it's
> > probably in somewhat dodgy shape after a year or two of being ignored.
> >
> > Before I dig in, I wanted to ask if this would be a helpful project.
> > The main selling points would be:
> >
> > (1) Ability to split it out from the main QEMU tree, like QMP was, for
> > use in external testing
> > (2) Better stream management so console information is consumed and
> > logged reliably without deadlocks due to pipes hitting their buffer
> > limit
>
> I can't remember the specific details, but I do recall some
> scenarios where it would have been valuable to have the console
> output continuously streamed to the log file on disk, while
> at the same time retainining the ability to do expected
> output matching from the API.
>
> > (3) Removing multiprocess and multithread processing of console and
> > stdout/stderr logging from test suites in favor of using async
> > user-threads for (hopefully) simpler, more straightforward processing
> > and customization.
>
> Which tests were you refering to here ?  I didn't think we had
> python tests using multiprocess/multithread ?

We use it for console log draining ... see
python/qemu/machine/console_socket.py which is quite the hack.

>
> Tangentially related, in debugging failures of the functional
> test suite, and to a lesser extent I/O tests / qmp-shell, I've
> often found myself heavily confused by the asyncio call paths
> and control flow, making it really hard to debug.
>
> In particular I've never fully figured out how to get qmp-shell
> to have a clean exit path without stack traces being thrown.

Send me an example / a reproducer and I will make it pretty or die trying.

Most teardown errors fall into two camps:

(1) EOF was encountered, and it gets reported up the stack as an
"Error" even though it's quite an ordinary part of the connection
lifecycle. I did this mostly to force iotests to be noisy when QEMU
died unexpectedly towards the end of a test, which happened more often
than you'd like. Not every last possible pathway to have QEMU exit or
shut down is always handled, and so EOF pops up as an "error" when it
may not be.

(I tried to handle them all, but I think there may be infinite ways to
intentionally kill QEMU.)

(2) We are using the sync frontend and an API call to terminate the
machine and/or the QMP class wasn't issued, which means we miss out on
the opportunity to gracefully exit the async loop in the backend. I
make an effort to rectify this whenever possible, but sometimes it
just isn't possible. The avalanche of stack traces you receive in this
cases is not ideal, but I have made efforts to give human readable
summaries when they occur so that the cause and fix are obvious. I
wouldn't be fundamentally shocked if I missed a case where this can
happen.

I was focused on the async qmp shell utility for a time, but stalled
because I couldn't design an appropriate command format I wanted to
commit to. I have probably neglected to make the "old" qmp-shell as
nice as a result... might be time to revisit this to make sure we have
a utility that is using all of the nice new features of the async
library.

(For instance, I never routed through async events up to the "old"
qmp-shell, so you still have to press enter on a blank line to trick
the utility into polling for new events to show. Not ideal.)

One of my hopes is that an async machine class can work in conjunction
with an async qmp class to corral concurrent errors better and produce
less output yapping in general when errors occur. I should be able to
condense them and report them more succinctly as a result.

>
> I'm unclear if this is related to our having QEMUMachine class
> expose a sync API on top of an async QMP class (square peg in
> a round hole situation), or if this is just something inherent
> to Python asyncio APIs, or probably a bit of both ?

A bit of both, but there are some mitigations possible with 3.12 that
I want to address.

One of them is the addition of ExceptionGroup which allows handling of
multiple error pathways and handling/reporting those errors in a sane
way.

Another new thing I have wanted to add is Exception notes, which
allows nice, human-readable error messages attached to unhandled stack
traces that can help give non-jargony explanations for why something
has crashed. It won't stop the unruly stack trace, but it will at
least give a nice summary that end users can latch on to without
needing to go cross-eyed trying to understand the async trace.

I also believe there's a few workarounds and hacks that got baked in
during the early days that can be dropped now, and that should
hopefully help to simplify error pathways, too. They are admittedly
complex.

>
> If switching to a fully async API throughout the stack makes
> debugging simpler, I'd welcome it.

That's the hope. If I can't make that true, it won't be worth it, I agree.

>
> > (4) Ability to add listeners/event handlers to watch for specific
> > patterns and trigger asynchronously when they occur; i.e. ability to
> > handle and watch for specific patterns in an arbitrary order without
> > deadlocking your test
> > (5) Centralized timeout setting and handling.
>
> With regards,
> Daniel
> --
> |: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
> |: https://libvirt.org          ~~          https://entangle-photo.org :|
> |: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|
>