VHOST_USER_PROTOCOL_F_VDPA

VHOST_USER_PROTOCOL_F_VDPA

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2510 bytes --]

The first vDPA ioctls have been added to the vhost-user protocol and I
wonder if it's time to fully change the vhost-user protocol's focus to
providing a full VIRTIO device model like vDPA does.

Initially vhost-user was just used for vhost-net. As a result it didn't
need the full VIRTIO device model including the configuration space and
device status register.

Over the years device-specific messages were added to extend vhost-user
to cover more of the VIRTIO device model. vhost-user-blk needed
configuration space support, for example.

The problem for VMMs and device backend implementors is that the
protocol is currently positioned halfway between the original vhost-net
approach and the full VIRTIO device model. Even if a VMM implements
VHOST_USER_GET_CONFIG, it can only expect it to work with
vhost-user-blk, not vhost-user-net. Similarly, a vhost-user-net device
backend cannot implement VHOST_USER_GET_CONFIG and expect all VMMs to
allow it to participate in configuration space emulation because
existing VMMs won't send that message.

The current approach where each device type uses a undocumented subset
of vhost-user messages is really messy. VMM and device backend
implementors have to look at existing implementations to know what is
expected for a given device type. It would be nice to switch to the
VIRTIO device model so that the VIRTIO specification can be used as the
reference for how device types work.

Now that vDPA is here and extends the kernel vhost ioctls with a full
VIRTIO device model, it might be a good time to revise the vhost-user
protocol.

A vdpa-user protocol (or vhost-user 2.0) would replace the current mess
with a full VIRTIO device model. Both VMMs and device backends would
require changes to support this, but it would be a cleaner path forward
for the vhost-user protocol.

One way of doing this would be a new VHOST_USER_PROTOCOL_F_VDPA feature
bit that indicates all the currently existing Linux vDPA ioctl messages
are available. Legacy vhost-user messages with overlapping functionality
must not be used when this bit is set. Most importantly, device backends
need to implement the full VIRTIO device model, regardless of device
type (net, blk, scsi, etc).

The device type most affected by this change would be virtio-net. The
other device types are already closer to the full VIRTIO device model.

I wanted to share this idea in case someone is currently trying to
figure out how to add more VIRTIO device model functionality to
vhost-user.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: VHOST_USER_PROTOCOL_F_VDPA

[Jason Wang]

On 2020/8/21 下午7:08, Stefan Hajnoczi wrote:
> The first vDPA ioctls have been added to the vhost-user protocol and I
> wonder if it's time to fully change the vhost-user protocol's focus to
> providing a full VIRTIO device model like vDPA does.
>
> Initially vhost-user was just used for vhost-net. As a result it didn't
> need the full VIRTIO device model including the configuration space and
> device status register.
>
> Over the years device-specific messages were added to extend vhost-user
> to cover more of the VIRTIO device model. vhost-user-blk needed
> configuration space support, for example.
>
> The problem for VMMs and device backend implementors is that the
> protocol is currently positioned halfway between the original vhost-net
> approach and the full VIRTIO device model. Even if a VMM implements
> VHOST_USER_GET_CONFIG, it can only expect it to work with
> vhost-user-blk, not vhost-user-net. Similarly, a vhost-user-net device
> backend cannot implement VHOST_USER_GET_CONFIG and expect all VMMs to
> allow it to participate in configuration space emulation because
> existing VMMs won't send that message.
>
> The current approach where each device type uses a undocumented subset
> of vhost-user messages is really messy. VMM and device backend
> implementors have to look at existing implementations to know what is
> expected for a given device type. It would be nice to switch to the
> VIRTIO device model so that the VIRTIO specification can be used as the
> reference for how device types work.
>
> Now that vDPA is here and extends the kernel vhost ioctls with a full
> VIRTIO device model, it might be a good time to revise the vhost-user
> protocol.
>
> A vdpa-user protocol (or vhost-user 2.0) would replace the current mess
> with a full VIRTIO device model. Both VMMs and device backends would
> require changes to support this, but it would be a cleaner path forward
> for the vhost-user protocol.
>
> One way of doing this would be a new VHOST_USER_PROTOCOL_F_VDPA feature
> bit that indicates all the currently existing Linux vDPA ioctl messages
> are available. Legacy vhost-user messages with overlapping functionality
> must not be used when this bit is set. Most importantly, device backends
> need to implement the full VIRTIO device model, regardless of device
> type (net, blk, scsi, etc).
>
> The device type most affected by this change would be virtio-net. The
> other device types are already closer to the full VIRTIO device model.
>
> I wanted to share this idea in case someone is currently trying to
> figure out how to add more VIRTIO device model functionality to
> vhost-user.


Cc Adrian.

I think this makes sense.

Thanks


>
> Stefan

Re: VHOST_USER_PROTOCOL_F_VDPA

[Maxime Coquelin]

[-- Attachment #1.1: Type: text/plain, Size: 2784 bytes --]

Hi Stefan,

On 8/21/20 1:08 PM, Stefan Hajnoczi wrote:
> The first vDPA ioctls have been added to the vhost-user protocol and I
> wonder if it's time to fully change the vhost-user protocol's focus to
> providing a full VIRTIO device model like vDPA does.
> 
> Initially vhost-user was just used for vhost-net. As a result it didn't
> need the full VIRTIO device model including the configuration space and
> device status register.
> 
> Over the years device-specific messages were added to extend vhost-user
> to cover more of the VIRTIO device model. vhost-user-blk needed
> configuration space support, for example.
> 
> The problem for VMMs and device backend implementors is that the
> protocol is currently positioned halfway between the original vhost-net
> approach and the full VIRTIO device model. Even if a VMM implements
> VHOST_USER_GET_CONFIG, it can only expect it to work with
> vhost-user-blk, not vhost-user-net. Similarly, a vhost-user-net device
> backend cannot implement VHOST_USER_GET_CONFIG and expect all VMMs to
> allow it to participate in configuration space emulation because
> existing VMMs won't send that message.
> 
> The current approach where each device type uses a undocumented subset
> of vhost-user messages is really messy. VMM and device backend
> implementors have to look at existing implementations to know what is
> expected for a given device type. It would be nice to switch to the
> VIRTIO device model so that the VIRTIO specification can be used as the
> reference for how device types work.
> 
> Now that vDPA is here and extends the kernel vhost ioctls with a full
> VIRTIO device model, it might be a good time to revise the vhost-user
> protocol.
> 
> A vdpa-user protocol (or vhost-user 2.0) would replace the current mess
> with a full VIRTIO device model. Both VMMs and device backends would
> require changes to support this, but it would be a cleaner path forward
> for the vhost-user protocol.
> 
> One way of doing this would be a new VHOST_USER_PROTOCOL_F_VDPA feature
> bit that indicates all the currently existing Linux vDPA ioctl messages
> are available. Legacy vhost-user messages with overlapping functionality
> must not be used when this bit is set. Most importantly, device backends
> need to implement the full VIRTIO device model, regardless of device
> type (net, blk, scsi, etc).
> 
> The device type most affected by this change would be virtio-net. The
> other device types are already closer to the full VIRTIO device model.
> 
> I wanted to share this idea in case someone is currently trying to
> figure out how to add more VIRTIO device model functionality to
> vhost-user.
> 
> Stefan
> 

I understand the need and like the idea.

Maxime


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: VHOST_USER_PROTOCOL_F_VDPA

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1945 bytes --]

On Thu, Sep 10, 2020 at 10:55:00AM +0200, Maxime Coquelin wrote:
> I understand the need and like the idea.

If no one plans to work on this within the next 6 months it could be an
internship project since the Outreachy December-March round is coming
up.

The time requirement is typically ~5 hours per week from the mentor.

Jason, Maxime, Cindy, Michael: Are you interested in mentoring this
project idea?

Internships focus mostly implementing existing designs since the intern
doesn't have enough background knowledge or time to design some of this
complexity from scratch.

As a starting point, the vhost-user protocol can be extended with all
the vDPA messages. Redundant vhost-user messages from the existing
protocol can be dropped/deprecated (i.e. device-specific messages that
were introduced to work around the lack of the VIRTIO device lifecycle).

Fully embracing the vDPA/VIRTIO lifecycle eliminates the need for
per-type QEMU devices (vhost-user-net, vhost-user-blk, etc). Instead
per-transport devices are needed (-device vdpa-user-{pci,ccw,mmio}).
This means new device types can be added later without writing all the
boilerplate -device vhost-user-foo-{pci,ccw,mmio} code!

The internship could focus on implementing -device vdpa-user-pci and a
libvdpa-user test/example. The vhost-user spec changes should probably
be agreed by the community beforehand so the intern doesn't need to
worry about the protocol design (there's plenty of implementation work
to do!).

This is one way to break this down into an internship project, but there
are other approaches. Maybe you prefer to extend vhost-user more
gradually (e.g. no -device vdpa-user-pci and the existing per-type
device approach is continued) while still adding the vDPA protocol
messages...

Anyway, if you'd like to mentor this project in Outreachy December-March
please add a project idea to the QEMU wiki:
https://wiki.qemu.org/Outreachy_2020_DecemberMarch

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]