Can we make QMP commands in Rust always be coroutine safe?

Can we make QMP commands in Rust always be coroutine safe?

[Markus Armbruster]

QMP commands that perform potentially blocking I/O can profit from
running in a coroutine.  Ideally, we'd run all commands in coroutine
context: most of them don't care, a few profit.  However, some existing
commands may need fixing to run safely there.  Since we don't know which
ones do, running in couroutine context is opt-in.  From
docs/devel/qapi-code-gen.rst section "Commands":

    Member 'coroutine' tells the QMP dispatcher whether the command handler
    is safe to be run in a coroutine.  It defaults to false.  If it is true,
    the command handler is called from coroutine context and may yield while
    waiting for an external event (such as I/O completion) in order to avoid
    blocking the guest and other background operations.

    Coroutine safety can be hard to prove, similar to thread safety.  Common
    pitfalls are:

    - The BQL isn't held across ``qemu_coroutine_yield()``, so
      operations that used to assume that they execute atomically may have
      to be more careful to protect against changes in the global state.

    - Nested event loops (``AIO_WAIT_WHILE()`` etc.) are problematic in
      coroutine context and can easily lead to deadlocks.  They should be
      replaced by yielding and reentering the coroutine when the condition
      becomes false.

    Since the command handler may assume coroutine context, any callers
    other than the QMP dispatcher must also call it in coroutine context.
    In particular, HMP commands calling such a QMP command handler must be
    marked ``.coroutine = true`` in hmp-commands.hx.

    It is an error to specify both ``'coroutine': true`` and ``'allow-oob': true``
    for a command.  We don't currently have a use case for both together and
    without a use case, it's not entirely clear what the semantics should
    be.

Can we make commands written in Rust always coroutine safe?

Re: Can we make QMP commands in Rust always be coroutine safe?

[Manos Pitsidianakis]

On Tue, May 5, 2026 at 11:45 AM Markus Armbruster <armbru@redhat.com> wrote:
>
> QMP commands that perform potentially blocking I/O can profit from
> running in a coroutine.  Ideally, we'd run all commands in coroutine
> context: most of them don't care, a few profit.  However, some existing
> commands may need fixing to run safely there.  Since we don't know which
> ones do, running in couroutine context is opt-in.  From
> docs/devel/qapi-code-gen.rst section "Commands":
>
>     Member 'coroutine' tells the QMP dispatcher whether the command handler
>     is safe to be run in a coroutine.  It defaults to false.  If it is true,
>     the command handler is called from coroutine context and may yield while
>     waiting for an external event (such as I/O completion) in order to avoid
>     blocking the guest and other background operations.
>
>     Coroutine safety can be hard to prove, similar to thread safety.  Common
>     pitfalls are:
>
>     - The BQL isn't held across ``qemu_coroutine_yield()``, so
>       operations that used to assume that they execute atomically may have
>       to be more careful to protect against changes in the global state.
>
>     - Nested event loops (``AIO_WAIT_WHILE()`` etc.) are problematic in
>       coroutine context and can easily lead to deadlocks.  They should be
>       replaced by yielding and reentering the coroutine when the condition
>       becomes false.
>
>     Since the command handler may assume coroutine context, any callers
>     other than the QMP dispatcher must also call it in coroutine context.
>     In particular, HMP commands calling such a QMP command handler must be
>     marked ``.coroutine = true`` in hmp-commands.hx.
>
>     It is an error to specify both ``'coroutine': true`` and ``'allow-oob': true``
>     for a command.  We don't currently have a use case for both together and
>     without a use case, it's not entirely clear what the semantics should
>     be.
>
> Can we make commands written in Rust always coroutine safe?

For async Rust, yes. Any type that doesn't implement the `Send` trait
cannot be held across an .await (i.e. yield). This'd make it
impossible to hold the BQL and yield, it would just not compile. You'd
need a simple executor (i.e. runtime) to run async code, possibly as
part of QEMU's existing event loop.

-- 
Manos Pitsidianakis
Emulation and Virtualization Engineer at Linaro Ltd

Re: Can we make QMP commands in Rust always be coroutine safe?

[Paolo Bonzini]

On 5/5/26 10:44, Markus Armbruster wrote:
>      Coroutine safety can be hard to prove, similar to thread safety.  Common
>      pitfalls are:
> 
>      - The BQL isn't held across ``qemu_coroutine_yield()``, so
>        operations that used to assume that they execute atomically may have
>        to be more careful to protect against changes in the global state.
> 
>      - Nested event loops (``AIO_WAIT_WHILE()`` etc.) are problematic in
>        coroutine context and can easily lead to deadlocks.  They should be
>        replaced by yielding and reentering the coroutine when the condition
>        becomes false.
> 
>      Since the command handler may assume coroutine context, any callers
>      other than the QMP dispatcher must also call it in coroutine context.
>      In particular, HMP commands calling such a QMP command handler must be
>      marked ``.coroutine = true`` in hmp-commands.hx.
> 
>      It is an error to specify both ``'coroutine': true`` and ``'allow-oob': true``
>      for a command.  We don't currently have a use case for both together and
>      without a use case, it's not entirely clear what the semantics should
>      be.
> 
> Can we make commands written in Rust always coroutine safe?

We won't *ever* have mixed coroutine/non-coroutine functions in Rust. 
Kevin's prototype used async functions (stackless coroutines) for Rust 
yielding functions, rather than qemu_coroutine_yield()[1].  Within sucg 
fubctuibs, yielding is much more explicit than in C (you have to write 
".await" explicitly at all levels of calling a yielding function).

But we have the BQL, and "coroutine: true" commands if I understand 
correctly are run outside it (they run in iothread context).  So any 
command that uses BQL-protected data cannot be coroutine safe, and that 
means it's likely that Rust would also have coroutine: true/false.

However, there are safeguards:

1) it will be impossible to write yielding code in a "coroutine: false" 
Rust command; it won't secretly start a nested event loop.

2) releasing the BQL while keeping a reference to a BqlRefCell is an 
instant panic;

This leaves nested event loops in "coroutine: true" commands as a 
potential pitfall.

Paolo

[1] suspension and resumption is represented respectively by futures (a 
discriminated union of a suspension state and a result after completion) 
and wakers.  To convert suspended async fns to qemu_coroutine_yield() 
calls, Kevin wrapped the async fns with a loop that continues until the 
future as a result, yielding across calls to the async fn; and to resume 
the suspended async fn, he wrote a waker that invokes 
qemu_coroutine_resume().

Re: Can we make QMP commands in Rust always be coroutine safe?

[Kevin Wolf]

Am 05.05.2026 um 12:51 hat Paolo Bonzini geschrieben:
> On 5/5/26 10:44, Markus Armbruster wrote:
> >      Coroutine safety can be hard to prove, similar to thread safety.  Common
> >      pitfalls are:
> > 
> >      - The BQL isn't held across ``qemu_coroutine_yield()``, so
> >        operations that used to assume that they execute atomically may have
> >        to be more careful to protect against changes in the global state.
> > 
> >      - Nested event loops (``AIO_WAIT_WHILE()`` etc.) are problematic in
> >        coroutine context and can easily lead to deadlocks.  They should be
> >        replaced by yielding and reentering the coroutine when the condition
> >        becomes false.
> > 
> >      Since the command handler may assume coroutine context, any callers
> >      other than the QMP dispatcher must also call it in coroutine context.
> >      In particular, HMP commands calling such a QMP command handler must be
> >      marked ``.coroutine = true`` in hmp-commands.hx.
> > 
> >      It is an error to specify both ``'coroutine': true`` and ``'allow-oob': true``
> >      for a command.  We don't currently have a use case for both together and
> >      without a use case, it's not entirely clear what the semantics should
> >      be.
> > 
> > Can we make commands written in Rust always coroutine safe?
> 
> We won't *ever* have mixed coroutine/non-coroutine functions in Rust.
> Kevin's prototype used async functions (stackless coroutines) for Rust
> yielding functions, rather than qemu_coroutine_yield()[1].  Within sucg
> fubctuibs, yielding is much more explicit than in C (you have to write
> ".await" explicitly at all levels of calling a yielding function).

By the way, one thing I realised only recently is that we probably don't
have a way to implement no_coroutine_fn in Rust (or specfically the Rust
bindings for C no_coroutine_fn functions).

> But we have the BQL, and "coroutine: true" commands if I understand
> correctly are run outside it (they run in iothread context).  So any command
> that uses BQL-protected data cannot be coroutine safe, and that means it's
> likely that Rust would also have coroutine: true/false.

A coroutine QMP command handler runs in the main loop initially, which
means that it holds the BQL. So I don't think you have a real reason to
have coroutine: false.

What the handler can do is move itself into a different thread later on
using something like aio_co_reschedule_self(), and then of course, it
can't rely on the BQL any more.

My understanding is that this would normally be handled by requiring the
Send trait for the Future in the function that hands the future off to
another thread. However, here we're not in the context of the
synchronous executor that works with the Future object, but in the async
fn itself. I'm not quite sure how to encode this requirement in the
binding for aio_co_reschedule_self() so that the calling async fn has to
be Send for it to compile. Any ideas?

> However, there are safeguards:
> 
> 1) it will be impossible to write yielding code in a "coroutine: false" Rust
> command; it won't secretly start a nested event loop.

Having a nested event loop in Rust isn't any harder than having it in C
(and I needed it in some block layer bindings because of synchronous
interfaces), so this depends on our discipline.

> 2) releasing the BQL while keeping a reference to a BqlRefCell is an instant
> panic;
> 
> This leaves nested event loops in "coroutine: true" commands as a potential
> pitfall.

I guess this is where a no_coroutine_fn that is understood by the
compiler would be useful.

> [1] suspension and resumption is represented respectively by futures (a
> discriminated union of a suspension state and a result after completion) and
> wakers.  To convert suspended async fns to qemu_coroutine_yield() calls,
> Kevin wrapped the async fns with a loop that continues until the future as a
> result, yielding across calls to the async fn; and to resume the suspended
> async fn, he wrote a waker that invokes qemu_coroutine_resume().

Kevin