Re: Best practice for issuing blocking calls in response to an event

[Miles Glenn <milesg@linux.ibm.com>]

On Thu, 2025-03-20 at 16:09 -0400, Stefan Hajnoczi wrote:
> On Thu, Mar 20, 2025 at 12:34 PM Miles Glenn <milesg@linux.ibm.com> wrote:
> > Hello,
> > 
> > I am attempting to simulate a system with multiple CPU
> > architectures.  To do this I am starting a unique QEMU process for each
> > CPU architecture that is needed. I'm also developing some QEMU code
> > that aids in transporting MMIO transactions across the process
> > boundaries using sockets.
> 
> I have CCed Phil. He has been working on heterogenous target emulation
> and might be interested.
> 
> > The design takes MMIO request messages off of a socket, services the
> > request by calling address_space_ldq_be(), then sends a response
> > message (containing the requested data) over the same
> > socket.  Currently, this is all done inside the socket IOReadHandler
> > callback function.
> 
> At a high level this is similar to the vfio-user feature where a PCI
> device is emulated in a separate process. This also involves sending
> messages describing QEMU's MemoryRegion accesses. See the "remote"
> machine type in QEMU to look at the code.
> 
> > This works as long as the targeted register exists in the same QEMU
> > process that received the request.  However, If the register exists in
> > another QEMU process, then the call to address_space_ldq_be() results
> > in another socket message being sent to that QEMU process, requesting
> > the data, and then waiting (blocking) for the response message
> > containing the data.  In other words, it ends up blocking inside the
> > event handler and even though the QEMU process containing the target
> > register was able to receive the request and send the response, the
> > originator of the request is unable to receive the response until it
> > eventually times out and stops blocking.  Once it times out and stops
> > blocking, it does receive the response, but now it is too late.
> > 
> > Here's a summary of the stack up to where the code blocks:
> > 
> > IOReadHandler callback
> >   calls address_space_ldq_be()
> >     resolves to mmio read op of a remote device
> >       sends request over socket and waits (blocks) for response
> > 
> > So, I'm looking for a way to handle the work of calling
> > address_space_ldq_be(), which might block when attempting to read a
> > register of a remote device, without blocking inside the IOReadHandler
> > callback context.
> > 
> > I've done a lot of searches and reading about how to do this on the web
> > and in the QEMU code but it's still not really clear to me how this
> > should be done in QEMU.  I've seen a lot about using coroutines to
> > handle cases like this. Is that what I should be using here?
> 
> The fundamental problem is that address_space_ldq_be() is synchronous,
> so there is no way to return back to the caller until the response has
> been received.
> 
> vfio-user didn't solve this problem. It simply blocks until the
> response is received, but it does drop the Big QEMU Lock during this
> time so that other vCPU threads can run. For example, see
> hw/remote/proxy.c:send_bar_access_msg() and
> mpqemu_msg_send_and_await_reply().
> 
> QEMU supports nested event loops, but they come with their own set of
> gotchas. The way a nested event loop might help here is to send the
> request and then call aio_poll() to receive the response in another
> IOReadHandler. This way other event loop processing can take place
> while waiting in address_space_ldq_be().
> 
> The second problem is that this approach where QEMU processes send
> requests to each other needs to be implemented carefully to avoid
> deadlocks. For example, devices that do DMA could load/store memory
> belonging to another device handled by another QEMU. Once there is an
> A -> B -> A situation it could deadlock.
> 
> Both vfio-user and vhost-user have similar issues with their
> bi-directional communication where a device emulation process can send
> a message to QEMU while processing a message from QEMU. Deadlock can
> be avoided if the code is structured so that QEMU is able to receive
> new requests during the time when it is waiting for a response.
> 
> Stefan

Stefan, Thank you for the quick response and great information!

I'm not sure if this is the best way, but I was able to get things
working today using the coroutine approach.

Now, the aforementioned stack looks like this:

IOReadHandler callback receives request
  enters coroutine
    calls address_space_ldq_be()
      resolves to mmio read op of a remote device
        sends request
over socket
        detects coroutine context and
        calls qemu_coroutine_yield() instead of blocking
  returns to callback 

<time passes>

IOReadHandler callback receives response
  re-enters coroutine
        mmio read op returns data received in response message
    address_space_ldq_be() returns
  coroutine completes and returns to callback

While this works, I couldn't help but notice that the coroutine concept
seems to be like a form of multithreading.  Is there some advantage to
using coroutines over doing the work in another thread?  Does QEMU
offer an interface that allows for a callback to queue up work that can
be handled by another thread or a pool of threads?

Thanks,

Glenn Miles