Hermetic virtio-vsock in QEMU

Hermetic virtio-vsock in QEMU

[Roman Kiryanov]

[-- Attachment #1: Type: text/plain, Size: 755 bytes --]

Hi Peter, Alex and QEMU,

I work in Android Studio Emulator and we use virtio-vsock to emulate
devices (e.g. sensors) which live in the Emulator binary. We need to run on
Windows and in environments without CONFIG_VHOST_VSOCK, that is why we
cannot use vhost-vsock and invented our implementation. I tried to grep the
QEMU8 sources and I believe virtio-vsock is not available there.

Do you think it is a good idea to implement virtio-vsock in QEMU (e.g. to
run on Windows)? If the answer is yes, could you please point where I could
start to build an upstreamable solution (not Android Emulator specific)? It
is not clear to me how we should make the device available for clients
(sensors, modem, adb, etc) in a generic way.

Thank you.

Regards,
Roman.

[-- Attachment #2: Type: text/html, Size: 889 bytes --]

Re: Hermetic virtio-vsock in QEMU

[Daniel P. Berrangé]

On Wed, Apr 03, 2024 at 02:30:33PM -0700, Roman Kiryanov wrote:
> Hi Peter, Alex and QEMU,
> 
> I work in Android Studio Emulator and we use virtio-vsock to emulate
> devices (e.g. sensors) which live in the Emulator binary. We need to run on
> Windows and in environments without CONFIG_VHOST_VSOCK, that is why we
> cannot use vhost-vsock and invented our implementation. I tried to grep the
> QEMU8 sources and I believe virtio-vsock is not available there.
> 
> Do you think it is a good idea to implement virtio-vsock in QEMU (e.g. to
> run on Windows)? If the answer is yes, could you please point where I could
> start to build an upstreamable solution (not Android Emulator specific)? It
> is not clear to me how we should make the device available for clients
> (sensors, modem, adb, etc) in a generic way.

This issue is proposing the idea of exposing VSOCK using AF_UNIX as the
host backend, in a manner that's compatible with that used by firecracker
and cloud-hypervisor:

  https://gitlab.com/qemu-project/qemu/-/issues/2095

Recent versions of Windows support AF_UNIX these days, so hopefully that
would be satisfactory as an approach ?

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Hermetic virtio-vsock in QEMU

[Roman Kiryanov]

[-- Attachment #1: Type: text/plain, Size: 2282 bytes --]

Hi Daniel,

thank you for looking into this. I checked how VHOST_USER_VSOCK and it
refers to the vhost-user protocol. It is implemented in the
subprojects/libvhost-user library, but this library depends on poll.h
and linux/vhost.h files. Do you know if it builds/works on Windows?

I checked how poll.h is used and the only reference I found is "VU_WATCH_IN
= POLLIN", no other VU_WATCH_* items seem to be used (the same applies to
G_IO_* in -glib.c). Maybe retire VU_WATCH_ and poll.h dependency?

For linux/vhost.h, I see it is gated by defined(__linux__)
in libvhost-user.c but not in libvhost-user.h and I think it is not used
in libvhost-user.

Regards,
Roman.


On Mon, Apr 15, 2024 at 4:20 AM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Wed, Apr 03, 2024 at 02:30:33PM -0700, Roman Kiryanov wrote:
> > Hi Peter, Alex and QEMU,
> >
> > I work in Android Studio Emulator and we use virtio-vsock to emulate
> > devices (e.g. sensors) which live in the Emulator binary. We need to run
> on
> > Windows and in environments without CONFIG_VHOST_VSOCK, that is why we
> > cannot use vhost-vsock and invented our implementation. I tried to grep
> the
> > QEMU8 sources and I believe virtio-vsock is not available there.
> >
> > Do you think it is a good idea to implement virtio-vsock in QEMU (e.g. to
> > run on Windows)? If the answer is yes, could you please point where I
> could
> > start to build an upstreamable solution (not Android Emulator specific)?
> It
> > is not clear to me how we should make the device available for clients
> > (sensors, modem, adb, etc) in a generic way.
>
> This issue is proposing the idea of exposing VSOCK using AF_UNIX as the
> host backend, in a manner that's compatible with that used by firecracker
> and cloud-hypervisor:
>
>   https://gitlab.com/qemu-project/qemu/-/issues/2095
>
> Recent versions of Windows support AF_UNIX these days, so hopefully that
> would be satisfactory as an approach ?
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>
>

[-- Attachment #2: Type: text/html, Size: 3377 bytes --]

Re: Hermetic virtio-vsock in QEMU

[Daniel P. Berrangé]

On Wed, Apr 17, 2024 at 12:31:41PM -0700, Roman Kiryanov wrote:
> Hi Daniel,
> 
> thank you for looking into this. I checked how VHOST_USER_VSOCK and it
> refers to the vhost-user protocol. It is implemented in the
> subprojects/libvhost-user library, but this library depends on poll.h
> and linux/vhost.h files. Do you know if it builds/works on Windows?

Stefano recently ported it to work on any POSIX platform,
but obviously that still excludes Windows:

  https://lists.nongnu.org/archive/html/qemu-devel/2024-04/msg00388.html

I'm unclear what blockers there are wrt Windows, but perhaps
Stefano (CC'd) can advise.

Personally, while I think vhost-user is a good option for the
vast majority of cases, I believe QEMU ought to natively
implement a simple self contained AF_UNIX backend for vsock,
avoiding need to depend on 3rd party software for its use.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Hermetic virtio-vsock in QEMU

[Stefano Garzarella]

Hi,
sorry for the delay, but I have been away most of last month.

On Thu, Apr 18, 2024 at 10:34:43AM GMT, Daniel P. Berrangé wrote:
>On Wed, Apr 17, 2024 at 12:31:41PM -0700, Roman Kiryanov wrote:
>> Hi Daniel,
>>
>> thank you for looking into this. I checked how VHOST_USER_VSOCK and it
>> refers to the vhost-user protocol. It is implemented in the
>> subprojects/libvhost-user library, but this library depends on poll.h
>> and linux/vhost.h files. Do you know if it builds/works on Windows?
>
>Stefano recently ported it to work on any POSIX platform,
>but obviously that still excludes Windows:
>
>  https://lists.nongnu.org/archive/html/qemu-devel/2024-04/msg00388.html
>
>I'm unclear what blockers there are wrt Windows, but perhaps
>Stefano (CC'd) can advise.

I have no experience with Windows, but what we need for vhost-user is:

- AF_UNIX and be able to send file descriptors using ancillary data
   (i.e. SCM_RIGHTS)
- share memory via an fd associated with it (i.e. shm_open(),
   memfd_create(), etc.)
- eventfd or pipe/pipe2 to exchange events (irq and kick)

>
>Personally, while I think vhost-user is a good option for the
>vast majority of cases, I believe QEMU ought to natively
>implement a simple self contained AF_UNIX backend for vsock,
>avoiding need to depend on 3rd party software for its use.

I was already planning to extrapolate the code that handles this from
vhost-user-vsock to implement a library, this might be a good reason.
Next GSoC?

Thanks,
Stefano

Re: Hermetic virtio-vsock in QEMU

[Roman Kiryanov]

Hi Stefano,

On Tue, May 7, 2024 at 1:10 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
> I have no experience with Windows, but what we need for vhost-user is:
>
> - AF_UNIX and be able to send file descriptors using ancillary data
>    (i.e. SCM_RIGHTS)

As far as I understand, Windows does NOT support SCM_RIGHTS over AF_UNIX.

Regards,
Roman.

Re: Hermetic virtio-vsock in QEMU

[Stefano Garzarella]

Hi Roman,

On Tue, May 07, 2024 at 11:20:50PM GMT, Roman Kiryanov wrote:
>Hi Stefano,
>
>On Tue, May 7, 2024 at 1:10 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>> I have no experience with Windows, but what we need for vhost-user is:
>>
>> - AF_UNIX and be able to send file descriptors using ancillary data
>>    (i.e. SCM_RIGHTS)
>
>As far as I understand, Windows does NOT support SCM_RIGHTS over AF_UNIX.

Thank you for the information. This is unfortunate and does not allow
us to use vhost-user as it is on Windows.

Thanks,
Stefano

Re: Hermetic virtio-vsock in QEMU

[Marc-André Lureau]

Hi

On Wed, May 8, 2024 at 11:50 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>
> Hi Roman,
>
> On Tue, May 07, 2024 at 11:20:50PM GMT, Roman Kiryanov wrote:
> >Hi Stefano,
> >
> >On Tue, May 7, 2024 at 1:10 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
> >> I have no experience with Windows, but what we need for vhost-user is:
> >>
> >> - AF_UNIX and be able to send file descriptors using ancillary data
> >>    (i.e. SCM_RIGHTS)
> >
> >As far as I understand, Windows does NOT support SCM_RIGHTS over AF_UNIX.
>
> Thank you for the information. This is unfortunate and does not allow
> us to use vhost-user as it is on Windows.
>

fwiw, Windows has other mechanisms to share resources between processes.

To share/pass sockets, you can use WSADuplicateSocket. For shared
memory and other resources, DuplicateHandle API.

(you can also share other things like D3D textures etc. those
mechanisms are used to enable -display dbus on Windows efficiently)


-- 
Marc-André Lureau

Re: Hermetic virtio-vsock in QEMU

[Stefano Garzarella]

On Wed, May 08, 2024 at 01:13:09PM GMT, Marc-André Lureau wrote:
>Hi
>
>On Wed, May 8, 2024 at 11:50 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>>
>> Hi Roman,
>>
>> On Tue, May 07, 2024 at 11:20:50PM GMT, Roman Kiryanov wrote:
>> >Hi Stefano,
>> >
>> >On Tue, May 7, 2024 at 1:10 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>> >> I have no experience with Windows, but what we need for vhost-user is:
>> >>
>> >> - AF_UNIX and be able to send file descriptors using ancillary data
>> >>    (i.e. SCM_RIGHTS)
>> >
>> >As far as I understand, Windows does NOT support SCM_RIGHTS over AF_UNIX.
>>
>> Thank you for the information. This is unfortunate and does not allow
>> us to use vhost-user as it is on Windows.
>>
>
>fwiw, Windows has other mechanisms to share resources between processes.
>
>To share/pass sockets, you can use WSADuplicateSocket. For shared
>memory and other resources, DuplicateHandle API.

Cool, thanks for sharing that. So it could be done, but I think we need
to extend the vhost-user protocol to work with Windows.

Thanks,
Stefano

>
>(you can also share other things like D3D textures etc. those
>mechanisms are used to enable -display dbus on Windows efficiently)
>
>
>-- 
>Marc-André Lureau
>

Re: Hermetic virtio-vsock in QEMU

[Paolo Bonzini]

On 5/8/24 11:38, Stefano Garzarella wrote:
> On Wed, May 08, 2024 at 01:13:09PM GMT, Marc-André Lureau wrote:
>> Hi
>>
>> On Wed, May 8, 2024 at 11:50 AM Stefano Garzarella 
>> <sgarzare@redhat.com> wrote:
>>>
>>> Hi Roman,
>>>
>>> On Tue, May 07, 2024 at 11:20:50PM GMT, Roman Kiryanov wrote:
>>> >Hi Stefano,
>>> >
>>> >On Tue, May 7, 2024 at 1:10 AM Stefano Garzarella 
>>> <sgarzare@redhat.com> wrote:
>>> >> I have no experience with Windows, but what we need for vhost-user 
>>> is:
>>> >>
>>> >> - AF_UNIX and be able to send file descriptors using ancillary data
>>> >>    (i.e. SCM_RIGHTS)
>>> >
>>> >As far as I understand, Windows does NOT support SCM_RIGHTS over 
>>> AF_UNIX.
>>>
>>> Thank you for the information. This is unfortunate and does not allow
>>> us to use vhost-user as it is on Windows.
>>>
>>
>> fwiw, Windows has other mechanisms to share resources between processes.
>>
>> To share/pass sockets, you can use WSADuplicateSocket. For shared
>> memory and other resources, DuplicateHandle API.
> 
> Cool, thanks for sharing that. So it could be done, but I think we need
> to extend the vhost-user protocol to work with Windows.

It would be possible to implement the memfd backend for Windows, using 
the CreateFileMapping() API.

However, the vhost-user protocol's VHOST_USER_SET_MEM_TABLE requests do 
not have any padding that can be used to pass the handle to the target. 
An extended version would be necessary.

One difference between Unix and Windows is that, if the vhost-server 
messes up the handling of messages from the socket, and therefore it 
does not close the handle, it is leaked forever.  This is not a huge 
deal per se, but I think it means that QEMU is not allowed to "open" a 
privileged vhost-user server process with PROCESS_DUP_HANDLE rights 
(translation: QEMU cannot provide duplicate handles to a privileged 
vhost-user server process).

Also I'm not sure what the cost of DuplicateHandle() is, and whether 
it's a good idea to do it for every region on every 
VHOST_USER_SET_MEM_TABLE request.  But VHOST_USER_SET_MEM_TABLE is not a 
fast path, so perhaps it's okay.

I think a virtio-vsock implementation in QEMU would be easier, lacking 
another usecase for vhost-user on Windows.

The main design question is whether multiple virtio-vsock devices for 
the same guest should share the CID space or not (I think it should, but 
I'm not 100% sure).  To connect host<->guest you could have a QOM 
object, here I am naming it vsock-forward as an example:

   -object vsock-forward,socket=SocketAddress,port=uint32,mode=connect

      takes the address of a listening socket on the host, and all
      connections to the given vsock port would be forwarded to the
      socket.

   -object vsock-forward,socket=SocketAddress,port=uint32,mode=listen

      creates a listening socket on the host, and all connections to that
      socket would be forwarded to the vsock port, where the guest should
      set up a server.

i.e. "mode" is what QEMU does, whether connect to the address or listen 
on it.

Another possibility for the names is to copy "hostfwd" and "guestfwd" 
names that are in netdev -netdev, i.e. respectively:

   -object vsock-guestfwd,socket=SocketAddress,port=uint32
   -object vsock-hostfwd,socket=SocketAddress,port=uint32

but I think it's worse.

Paolo

Re: Hermetic virtio-vsock in QEMU

[Daniel P. Berrangé]

On Wed, May 08, 2024 at 02:37:52PM +0200, Paolo Bonzini wrote:
> On 5/8/24 11:38, Stefano Garzarella wrote:
> > On Wed, May 08, 2024 at 01:13:09PM GMT, Marc-André Lureau wrote:
> > > Hi
> > > 
> > > On Wed, May 8, 2024 at 11:50 AM Stefano Garzarella
> > > <sgarzare@redhat.com> wrote:
> > > > 
> > > > Hi Roman,
> > > > 
> > > > On Tue, May 07, 2024 at 11:20:50PM GMT, Roman Kiryanov wrote:
> > > > >Hi Stefano,
> > > > >
> > > > >On Tue, May 7, 2024 at 1:10 AM Stefano Garzarella
> > > > <sgarzare@redhat.com> wrote:
> > > > >> I have no experience with Windows, but what we need for
> > > > vhost-user is:
> > > > >>
> > > > >> - AF_UNIX and be able to send file descriptors using ancillary data
> > > > >>    (i.e. SCM_RIGHTS)
> > > > >
> > > > >As far as I understand, Windows does NOT support SCM_RIGHTS
> > > > over AF_UNIX.
> > > > 
> > > > Thank you for the information. This is unfortunate and does not allow
> > > > us to use vhost-user as it is on Windows.
> > > > 
> > > 
> > > fwiw, Windows has other mechanisms to share resources between processes.
> > > 
> > > To share/pass sockets, you can use WSADuplicateSocket. For shared
> > > memory and other resources, DuplicateHandle API.
> > 
> > Cool, thanks for sharing that. So it could be done, but I think we need
> > to extend the vhost-user protocol to work with Windows.
> 
> It would be possible to implement the memfd backend for Windows, using the
> CreateFileMapping() API.
> 
> However, the vhost-user protocol's VHOST_USER_SET_MEM_TABLE requests do not
> have any padding that can be used to pass the handle to the target. An
> extended version would be necessary.
> 
> One difference between Unix and Windows is that, if the vhost-server messes
> up the handling of messages from the socket, and therefore it does not close
> the handle, it is leaked forever.  This is not a huge deal per se, but I
> think it means that QEMU is not allowed to "open" a privileged vhost-user
> server process with PROCESS_DUP_HANDLE rights (translation: QEMU cannot
> provide duplicate handles to a privileged vhost-user server process).
> 
> Also I'm not sure what the cost of DuplicateHandle() is, and whether it's a
> good idea to do it for every region on every VHOST_USER_SET_MEM_TABLE
> request.  But VHOST_USER_SET_MEM_TABLE is not a fast path, so perhaps it's
> okay.
> 
> I think a virtio-vsock implementation in QEMU would be easier, lacking
> another usecase for vhost-user on Windows.
> 
> The main design question is whether multiple virtio-vsock devices for the
> same guest should share the CID space or not (I think it should, but I'm not
> 100% sure).  To connect host<->guest you could have a QOM object, here I am
> naming it vsock-forward as an example:

Designwise, a native VSOCK backend in QEMU really should implement the
same approach defined by firecracker, so that we have interoperability
with systemd, firecracker and cloud-hypervisor. See

  https://gitlab.com/qemu-project/qemu/-/issues/2095
  https://github.com/firecracker-microvm/firecracker/blob/main/docs/vsock.md#firecracker-virtio-vsock-design

This involves multiple UNIX sockets on the host

  1 * /some/path   - QEMU listens on this, and accepts connections
                     from other host processes. The client sends
                     "PORT <num>" to indicate that guest port it
                     is connecting to

  n * /some/path_$PORT - QEMU connect to this for outgoing connections
                         from the guest. Other host processes need
                         to listen on whatever path_$PORT need to be
                         serviced

IOW, from a CLI pov, QEMU should need nothing more than

    -object vsock-forward,prefix=/some/path


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Hermetic virtio-vsock in QEMU

[Paolo Bonzini]

On Wed, May 8, 2024 at 2:51 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> Designwise, a native VSOCK backend in QEMU really should implement the
> same approach defined by firecracker, so that we have interoperability
> with systemd, firecracker and cloud-hypervisor. See
>
>   https://gitlab.com/qemu-project/qemu/-/issues/2095
>   https://github.com/firecracker-microvm/firecracker/blob/main/docs/vsock.md#firecracker-virtio-vsock-design
>
> This involves multiple UNIX sockets on the host
>
>   1 * /some/path   - QEMU listens on this, and accepts connections
>                      from other host processes. The client sends
>                      "PORT <num>" to indicate that guest port it
>                      is connecting to
>
>   n * /some/path_$PORT - QEMU connect to this for outgoing connections
>                          from the guest. Other host processes need
>                          to listen on whatever path_$PORT need to be
>                          serviced

The former is quite horrible, but okay. Out of curiosity, which
clients are using that convention?

> IOW, from a CLI pov, QEMU should need nothing more than
>
>     -object vsock-forward,prefix=/some/path

I would call this a "vsock-namespace" object with a matching namespace
property on the virtio-vsock device.  The id on the vsock-namespace
object would also allow 1) adding vsock-forward objects hanging from
the namespace 2) omitting the prefix to only allow specifying
connections via vsock-forward.

I agree that allowing interoperability with existing implementations
is nice to have, at least as a principle.

Paolo

Re: Hermetic virtio-vsock in QEMU

[Daniel P. Berrangé]

On Wed, May 08, 2024 at 03:00:30PM +0200, Paolo Bonzini wrote:
> On Wed, May 8, 2024 at 2:51 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> > Designwise, a native VSOCK backend in QEMU really should implement the
> > same approach defined by firecracker, so that we have interoperability
> > with systemd, firecracker and cloud-hypervisor. See
> >
> >   https://gitlab.com/qemu-project/qemu/-/issues/2095
> >   https://github.com/firecracker-microvm/firecracker/blob/main/docs/vsock.md#firecracker-virtio-vsock-design
> >
> > This involves multiple UNIX sockets on the host
> >
> >   1 * /some/path   - QEMU listens on this, and accepts connections
> >                      from other host processes. The client sends
> >                      "PORT <num>" to indicate that guest port it
> >                      is connecting to
> >
> >   n * /some/path_$PORT - QEMU connect to this for outgoing connections
> >                          from the guest. Other host processes need
> >                          to listen on whatever path_$PORT need to be
> >                          serviced
> 
> The former is quite horrible, but okay. Out of curiosity, which
> clients are using that convention?

I've no idea. The RFE above came out of a discussion with systemd
upstream when they were implementing their SSH-over-vsock host SSH
plugin

https://github.com/systemd/systemd/pull/30777

where it was noted that their code for using AF_VSOCK natively
wouldn't work with firecracker or cloud-hypervisor. Although
they've got native AF_VSOCK support there merged, conceptually
it looks desirable to have a common way of exposing AF_VSOCK
on the host.

Also, aside from cross-OS portability, this has the other benefit
that arbitarary users on the same host can make use of AF_VSOCK,
without worrying about CID clashes in the host kernel between
guests they're not aware of. All guests can just use the same
fixed CID and the UNIX socket exposure on the host namespaces
them.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|