Re: virtio-msg inter-VM transport vs virtio-vhost-user

["Edgar E. Iglesias" via qemu development <qemu-devel@nongnu.org>]

On Tue, Mar 03, 2026 at 12:56:46PM -0500, Demi Marie Obenour wrote:
> Spectrum (https://spectrum-os.org) is going to be implementing
> virtio devices outside of the host.  One proposed method of doing
> this is virtio-vhost-user, which is a virtio device that allows a
> VM to expose a vhost-user device to another VM.  For instance, one
> could assign a NIC to one VM and have it provide a vhost-user-net
> device for use by a different VM.
> 
> I brought this up on the KVM/QEMU community call today.  Alex Bennée
> recommended using virtio-msg instead.  However, I have a few concerns
> with this:
> 
> 1. Virtio-msg buses are specific to a given hypervisor or (in the
>    case of FF-A) to a given CPU architecture.  None of the current
>    buses support KVM on platforms other than Arm64.  Therefore,
>    a brand-new bus would be needed.

Hi Demi,

The virtio-msg AMP PCI bus works on KVM both for ARM and x86 and
any other arch, it's generic. These are the patches we posted to
qemu-devel:

https://lore.kernel.org/qemu-devel/20260224155721.612314-1-edgar.iglesias@gmail.com/


> 
> 2. Virtio-msg requires not-yet-upstream drivers in both the frontend
>    (the VM using the device) and the backend (the VM providing the
>    device).  Vhost-user uses any of the existing transports, such as
>    PCI or MMIO.  This means that upstream drivers can be used in the
>    frontend, and also enables supports for Windows and other guests
>    that lack support for virtio-msg.

Fair point, a bit of chicken and egg...

> 
> 3. Vhost-user is already widely deployed, so frontend implementations
>    are quite well tested.  A KVM-specific virtio-msg transport would
>    serve only one purpose: driver VMs (with assigned devices) on
>    non-Arm64 platforms.  This is a quite niche use-case.  Therefore,
>    I'm concerned that the needed frontend code will be poorly tested
>    and bitrot.
> 
> Manos Pitsidianakis stated that vhost-user does not make sense in
> this case.  Why is that?  Would it make sense to use virtio-msg
> between VMM and its VM, and expose a vhost-user device to the
> outside world?  What about having the virtio-vhost-user guest driver
> emulate a virtio-msg transport, so that it can be used with any device
> implementation supporting virtio-msg?
> 
> I would greatly appreciate any and all suggestions here.  This is a
> serious project that is going to be used in production, but I want
> to ensure that the design is the best possible.

I've not looked in detail at virtio-vhost-user but it seems to be
a bit similar to virtio-msg over PCI AMP with some differences.

My take is:
vhost-user is an interface designed for splitting virtio transport
from virtio device backend in user-space on the same OS instance.
The virtio-vhost-user device, enables the use of vhost-user across
VM's streching vhost-user's intended use. This is a great step but
Since vhost-user is not a virtio transport, It is not end-to-end in
the sense that you need QEMU to translate virtio-pci to vhost-user and
tunnel it over the virtio-vhost-user device to the backend.

Virtio-msg is from the start a transport meant to work between
heterogenous systems, different OS, OS instances even different
SoCs. It's end-to-end in the sense that if you have a front-end
driver in the front-end VM with an appropriate virtio-msg bus you can
talk directly with the backend without intermediate proxies,
potentially without VM exits.

Cheers,
Edgar