Re: RFC: use VFIO over a UNIX domain socket to implement device offloading

["Daniel P. Berrangé" <berrange@redhat.com>]

On Thu, Apr 02, 2020 at 11:19:42AM +0100, Stefan Hajnoczi wrote:
> On Wed, Apr 01, 2020 at 06:58:20PM +0200, Marc-André Lureau wrote:
> > On Wed, Apr 1, 2020 at 5:51 PM Thanos Makatos
> > <thanos.makatos@nutanix.com> wrote:
> > > > > Bear in mind that since this is just a PoC lots of things can break, e.g. some
> > > > > system call not intercepted etc.
> > > >
> > > > Cool, I had a quick look at libvfio and how the transport integrates
> > > > into libmuser.  The integration on the libmuser side is nice and small.
> > > >
> > > > It seems likely that there will be several different implementations of
> > > > the vfio-over-socket device side (server):
> > > > 1. libmuser
> > > > 2. A Rust equivalent to libmuser
> > > > 3. Maybe a native QEMU implementation for multi-process QEMU (I think JJ
> > > >    has been investigating this?)
> > > >
> > > > In order to interoperate we'll need to maintain a protocol
> > > > specification.  Mayb You and JJ could put that together and CC the vfio,
> > > > rust-vmm, and QEMU communities for discussion?
> > >
> > > Sure, I can start by drafting a design doc and share it.
> > 
> > ok! I am quite amazed you went this far with a ldpreload hack. This
> > demonstrates some limits of gpl projects, if it was necessary.
> > 
> > I think with this work, and the muser experience, you have a pretty
> > good idea of what the protocol could look like. My approach, as I
> > remember, was a quite straightforward VFIO over socket translation,
> > while trying to see if it could share some aspects with vhost-user,
> > such as memory handling etc.
> > 
> > To contrast with the work done on qemu-mp series, I'd also prefer we
> > focus our work on a vfio-like protocol, before trying to see how qemu
> > code and interface could be changed over multiple binaries etc. We
> > will start with some limitations, similar to the one that apply to
> > VFIO: migration, introspection, managements etc are mostly left out
> > for now. (iow, qemu-mp is trying to do too many things simultaneously)
> 
> qemu-mp has been cut down significantly in order to make it
> non-invasive.  The model is now much cleaner:
> 1. No monitor command or command-line option forwarding.  The device
>    emulation program has its own command-line and monitor that QEMU
>    doesn't know about.
> 2. No per-device proxy objects.  A single RemotePCIDevice is added to
>    QEMU.  In the current patch series it only supports the LSI SCSI
>    controller but once the socket protocol is changed to
>    vfio-over-socket it will be possible to use any PCI device.
> 
> We recently agreed on dropping live migration to further reduce the
> patch series.  If you have specific suggestions, please post reviews on
> the latest patch series.

To clarify - the decision was to *temporarily* drop live migration, to
make the initial patch series smaller and thus easier to merge. It does
ultimately need live migration, so there would be followup patch series
to provide migration support, after the initial merge.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|