[virtio-dev] Backend libraries for VirtIO device emulation

[virtio-dev] Backend libraries for VirtIO device emulation

[Alex Bennée]

Hi,

So the context of my question is what sort of common software layer is
required to implement a virtio backend entirely in userspace?

Currently most virtio backends are embedded directly in various VMMs
which emulate a number of devices as well as deal with handling devices
that are vhost aware and link with the host kernel. However there seems
to be a growing interest in having backends implemented in separate
processes, potentially even hosted in other guest VMs.

As far as I can tell there is a lot of duplicated effort in handling the
low level navigation of virt queues and buffers. QEMU has code in
hw/virtio as well as contrib/libvhost-user which is used by the recent
virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a
similar set of functionality for it's emulation. The Rust-vmm project
has libraries for implementing the device traits.

Another aspect to this is the growing interest in carrying virtio over
other hypervisors. I'm wondering if there is enough abstraction possible
to have a common library that is hypervisor agnostic? Can a device
backend be emulated purely with some shared memory and some sockets for
passing messages/kicks from/to the VMM which then deals with the hypervisor
specifics of the virtio-transport?

Thoughts?

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Matias Ezequiel Vara Larsen]

Hello Alex,
On Fri, Mar 06, 2020 at 06:33:57PM +0000, Alex Bennée wrote:
> Hi,
> 
> So the context of my question is what sort of common software layer is
> required to implement a virtio backend entirely in userspace?
> 
> Currently most virtio backends are embedded directly in various VMMs
> which emulate a number of devices as well as deal with handling devices
> that are vhost aware and link with the host kernel. However there seems
> to be a growing interest in having backends implemented in separate
> processes, potentially even hosted in other guest VMs.
> 
> As far as I can tell there is a lot of duplicated effort in handling the
> low level navigation of virt queues and buffers. QEMU has code in
> hw/virtio as well as contrib/libvhost-user which is used by the recent
> virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a
> similar set of functionality for it's emulation. The Rust-vmm project
> has libraries for implementing the device traits.
> 
> Another aspect to this is the growing interest in carrying virtio over
> other hypervisors. I'm wondering if there is enough abstraction possible
> to have a common library that is hypervisor agnostic? Can a device
> backend be emulated purely with some shared memory and some sockets for
> passing messages/kicks from/to the VMM which then deals with the hypervisor
> specifics of the virtio-transport?
> 
> Thoughts?

I am currently investigating the use of metamodeling to capture virtio
concepts. The work is still a draft but you can give a look at
https://github.com/MatiasVara/virtioml. One of the ideas is to base on a
metamodel to generate the code for different targets, e.g., rust,
c, etc. For the moment, I just played by generating the c header. This
could be extended to other languages. 

Regards, Matias.
> 
> -- 
> Alex Bennée
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Alex Bennée]

Matias Ezequiel Vara Larsen <matiasevara@gmail.com> writes:

> Hello Alex,
> On Fri, Mar 06, 2020 at 06:33:57PM +0000, Alex Bennée wrote:
>> Hi,
>> 
>> So the context of my question is what sort of common software layer is
>> required to implement a virtio backend entirely in userspace?
<snip>
>> 
>> Thoughts?
>
> I am currently investigating the use of metamodeling to capture virtio
> concepts. The work is still a draft but you can give a look at
> https://github.com/MatiasVara/virtioml. One of the ideas is to base on a
> metamodel to generate the code for different targets, e.g., rust,
> c, etc. For the moment, I just played by generating the c header. This
> could be extended to other languages.

Thanks for the pointer, I shall have a look.

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Dr. David Alan Gilbert]

* Alex Bennée (alex.bennee@linaro.org) wrote:
> Hi,
> 
> So the context of my question is what sort of common software layer is
> required to implement a virtio backend entirely in userspace?
> 
> Currently most virtio backends are embedded directly in various VMMs
> which emulate a number of devices as well as deal with handling devices
> that are vhost aware and link with the host kernel. However there seems
> to be a growing interest in having backends implemented in separate
> processes, potentially even hosted in other guest VMs.
> 
> As far as I can tell there is a lot of duplicated effort in handling the
> low level navigation of virt queues and buffers. QEMU has code in
> hw/virtio as well as contrib/libvhost-user which is used by the recent
> virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a
> similar set of functionality for it's emulation. The Rust-vmm project
> has libraries for implementing the device traits.
> 
> Another aspect to this is the growing interest in carrying virtio over
> other hypervisors. I'm wondering if there is enough abstraction possible
> to have a common library that is hypervisor agnostic? Can a device
> backend be emulated purely with some shared memory and some sockets for
> passing messages/kicks from/to the VMM which then deals with the hypervisor
> specifics of the virtio-transport?

It's a little tricky because it has to interface tightly with the way
that the memory-mapping works for the hypervisor, so that the external
process can access the memory of the queues.
QEMU's vhost-user has a fair amount of code for handling the mappings,
dirty logging for migration, iommu's and things like reset (which is
pretty hairy, and probably needs more work).

Dave

> Thoughts?
> 
> -- 
> Alex Bennée
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Alex Bennée]

Dr. David Alan Gilbert <dgilbert@redhat.com> writes:

> * Alex Bennée (alex.bennee@linaro.org) wrote:
>> Hi,
>> 
>> So the context of my question is what sort of common software layer is
>> required to implement a virtio backend entirely in userspace?
>> 
>> Currently most virtio backends are embedded directly in various VMMs
>> which emulate a number of devices as well as deal with handling devices
>> that are vhost aware and link with the host kernel. However there seems
>> to be a growing interest in having backends implemented in separate
>> processes, potentially even hosted in other guest VMs.
>> 
>> As far as I can tell there is a lot of duplicated effort in handling the
>> low level navigation of virt queues and buffers. QEMU has code in
>> hw/virtio as well as contrib/libvhost-user which is used by the recent
>> virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a
>> similar set of functionality for it's emulation. The Rust-vmm project
>> has libraries for implementing the device traits.
>> 
>> Another aspect to this is the growing interest in carrying virtio over
>> other hypervisors. I'm wondering if there is enough abstraction possible
>> to have a common library that is hypervisor agnostic? Can a device
>> backend be emulated purely with some shared memory and some sockets for
>> passing messages/kicks from/to the VMM which then deals with the hypervisor
>> specifics of the virtio-transport?
>
> It's a little tricky because it has to interface tightly with the way
> that the memory-mapping works for the hypervisor, so that the external
> process can access the memory of the queues.

I suspect the problem space can at least be reduced to at least a
POSIX-like environment - if that makes things simpler. The setting up of
memory-mappings should be the problem of the VMM, which would possibly
be hypervisor specific. After that it is simply(?) a question of sharing
the appropriate bit of memory between the VMM and the device process.

The other model would be the device process runs inside another guest -
most likely a Linux VM. Here the guest kernel can be told an area of
memory is special in some way and provide a device node that can be
mmaped in more or less the same way. In this configuration it can't even
be aware of what the underlying hypervisor is - just a block of memory
and a way to receive message queue events.

> QEMU's vhost-user has a fair amount of code for handling the mappings,
> dirty logging for migration, iommu's and things like reset (which is
> pretty hairy, and probably needs more work).

I suspect all of these multi-process models just hand wave away details
like migration because that really does benefit from a single process
with total awareness of the state of the system. That said I wonder how
robust a guest can be if the device emulation may go away at any time?

I guess in virtio if you never signal the consumption of a virt-queue it
will still be there waiting until you restart the emulation process and
pick up from where you left off?

>
> Dave
>
>> Thoughts?
>> 
>> -- 
>> Alex Bennée
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>> 


-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Jan Kiszka]

On 06.03.20 21:24, Alex Bennée wrote:
> I suspect all of these multi-process models just hand wave away details
> like migration because that really does benefit from a single process
> with total awareness of the state of the system. That said I wonder how
> robust a guest can be if the device emulation may go away at any time?
> 
> I guess in virtio if you never signal the consumption of a virt-queue it
> will still be there waiting until you restart the emulation process and
> pick up from where you left off?

The virtio spec seems be fine with individual backends dying. Recovery 
depends on the hypervisor maintaining state for the backends - not sure 
if that is easily doable, though.

But the challenge starts earlier: virtio-over-ivshmem is designed to 
support independent backends lifecycles (and has no state except for 
guest lifeness in the hypervisor). But while playing with that, I 
immediately ran into Linux frontend code assuming that a virtio device 
never dies (virtio console, IIRC). I suspect that there are more cases 
in Linux, not to speak of other guests.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Dr. David Alan Gilbert]

* Alex Bennée (alex.bennee@linaro.org) wrote:
> 
> Dr. David Alan Gilbert <dgilbert@redhat.com> writes:
> 
> > * Alex Bennée (alex.bennee@linaro.org) wrote:
> >> Hi,
> >> 
> >> So the context of my question is what sort of common software layer is
> >> required to implement a virtio backend entirely in userspace?
> >> 
> >> Currently most virtio backends are embedded directly in various VMMs
> >> which emulate a number of devices as well as deal with handling devices
> >> that are vhost aware and link with the host kernel. However there seems
> >> to be a growing interest in having backends implemented in separate
> >> processes, potentially even hosted in other guest VMs.
> >> 
> >> As far as I can tell there is a lot of duplicated effort in handling the
> >> low level navigation of virt queues and buffers. QEMU has code in
> >> hw/virtio as well as contrib/libvhost-user which is used by the recent
> >> virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a
> >> similar set of functionality for it's emulation. The Rust-vmm project
> >> has libraries for implementing the device traits.
> >> 
> >> Another aspect to this is the growing interest in carrying virtio over
> >> other hypervisors. I'm wondering if there is enough abstraction possible
> >> to have a common library that is hypervisor agnostic? Can a device
> >> backend be emulated purely with some shared memory and some sockets for
> >> passing messages/kicks from/to the VMM which then deals with the hypervisor
> >> specifics of the virtio-transport?
> >
> > It's a little tricky because it has to interface tightly with the way
> > that the memory-mapping works for the hypervisor, so that the external
> > process can access the memory of the queues.
> 
> I suspect the problem space can at least be reduced to at least a
> POSIX-like environment - if that makes things simpler. The setting up of
> memory-mappings should be the problem of the VMM, which would possibly
> be hypervisor specific. After that it is simply(?) a question of sharing
> the appropriate bit of memory between the VMM and the device process.

The 'simply(?)' is actually pretty tricky.
You have to share the mapping of all the RAM blocks in whcih the virtio
queues or the data to which they point might reside, and you also have
to let the other process know where in Guest physical address space they
live.  That mapping is also not constant, either with hotplug, or with
architecture specific things that cause physical address mapping to
change.

> The other model would be the device process runs inside another guest -
> most likely a Linux VM. Here the guest kernel can be told an area of
> memory is special in some way and provide a device node that can be
> mmaped in more or less the same way. In this configuration it can't even
> be aware of what the underlying hypervisor is - just a block of memory
> and a way to receive message queue events.

Doing it between VMs works in my mind; but again you still need to
handle that mapping.

> > QEMU's vhost-user has a fair amount of code for handling the mappings,
> > dirty logging for migration, iommu's and things like reset (which is
> > pretty hairy, and probably needs more work).
> 
> I suspect all of these multi-process models just hand wave away details
> like migration because that really does benefit from a single process
> with total awareness of the state of the system.

Vhost-user has it pretty well defined; it works - as long as the user
process does dirty map update.  Postcopy can also be made to work.

> That said I wonder how
> robust a guest can be if the device emulation may go away at any time?

That one I've not thought too much about, but the opposite case; making
the separate process survive even when the guest behaves
badly/resets/etc is quite nasty.

Dave

> I guess in virtio if you never signal the consumption of a virt-queue it
> will still be there waiting until you restart the emulation process and
> pick up from where you left off?
> 
> >
> > Dave
> >
> >> Thoughts?
> >> 
> >> -- 
> >> Alex Bennée
> >> 
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> >> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >> 
> 
> 
> -- 
> Alex Bennée
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Alex Bennée]

Dr. David Alan Gilbert <dgilbert@redhat.com> writes:

> * Alex Bennée (alex.bennee@linaro.org) wrote:
>> 
>> Dr. David Alan Gilbert <dgilbert@redhat.com> writes:
>> 
>> > * Alex Bennée (alex.bennee@linaro.org) wrote:
>> >> Hi,
>> >> 
>> >> So the context of my question is what sort of common software layer is
>> >> required to implement a virtio backend entirely in userspace?
>> >> 
>> >> Currently most virtio backends are embedded directly in various VMMs
>> >> which emulate a number of devices as well as deal with handling devices
>> >> that are vhost aware and link with the host kernel. However there seems
>> >> to be a growing interest in having backends implemented in separate
>> >> processes, potentially even hosted in other guest VMs.
>> >> 
>> >> As far as I can tell there is a lot of duplicated effort in handling the
>> >> low level navigation of virt queues and buffers. QEMU has code in
>> >> hw/virtio as well as contrib/libvhost-user which is used by the recent
>> >> virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a
>> >> similar set of functionality for it's emulation. The Rust-vmm project
>> >> has libraries for implementing the device traits.
>> >> 
>> >> Another aspect to this is the growing interest in carrying virtio over
>> >> other hypervisors. I'm wondering if there is enough abstraction possible
>> >> to have a common library that is hypervisor agnostic? Can a device
>> >> backend be emulated purely with some shared memory and some sockets for
>> >> passing messages/kicks from/to the VMM which then deals with the hypervisor
>> >> specifics of the virtio-transport?
>> >
>> > It's a little tricky because it has to interface tightly with the way
>> > that the memory-mapping works for the hypervisor, so that the external
>> > process can access the memory of the queues.
>> 
>> I suspect the problem space can at least be reduced to at least a
>> POSIX-like environment - if that makes things simpler. The setting up of
>> memory-mappings should be the problem of the VMM, which would possibly
>> be hypervisor specific. After that it is simply(?) a question of sharing
>> the appropriate bit of memory between the VMM and the device process.
>
> The 'simply(?)' is actually pretty tricky.

Well I am at the start of this journey so may be hand waving a bit ;-)

Lets drill down:

> You have to share the mapping of all the RAM blocks in whcih the virtio
> queues or the data to which they point might reside

Aren't all the queues in one section of memory?

As for where the data is doesn't this depend on the structure of the
device. As I understand the behaviour of virtfs there is a direct
relationship between the guest page cache and the host page cache to
take advantage of DAX. This by definition means the backend needs access
to the entire address space of the guest.

Is this also the case for other devices?

> and you also have
> to let the other process know where in Guest physical address space they
> live.  That mapping is also not constant, either with hotplug, or with
> architecture specific things that cause physical address mapping to
> change.

It sounds like the solution here would be to have bounce buffers as part
of the virtio spec which could be part of the virtio memory block? I
guess another option is for guests to keep their internal data (as
referenced by virtio drivers) in a fixed guest physical address but that
gets real complicated quick and I suspect is harder to audit from a
security point of view.

>> The other model would be the device process runs inside another guest -
>> most likely a Linux VM. Here the guest kernel can be told an area of
>> memory is special in some way and provide a device node that can be
>> mmaped in more or less the same way. In this configuration it can't even
>> be aware of what the underlying hypervisor is - just a block of memory
>> and a way to receive message queue events.
>
> Doing it between VMs works in my mind; but again you still need to
> handle that mapping.
>
>> > QEMU's vhost-user has a fair amount of code for handling the mappings,
>> > dirty logging for migration, iommu's and things like reset (which is
>> > pretty hairy, and probably needs more work).
>> 
>> I suspect all of these multi-process models just hand wave away details
>> like migration because that really does benefit from a single process
>> with total awareness of the state of the system.
>
> Vhost-user has it pretty well defined; it works - as long as the user
> process does dirty map update.  Postcopy can also be made to work.
>
>> That said I wonder how
>> robust a guest can be if the device emulation may go away at any time?
>
> That one I've not thought too much about, but the opposite case; making
> the separate process survive even when the guest behaves
> badly/resets/etc is quite nasty.

I guess whatever orchestrates the start-up of the VMs has to worry about
that. Some of the models I've been looking at have very simplistic
setups where the guest VMs described in the platform data to be spawned
directly by the hypervisor. I guess in those cases you need to restart
everything.

>
> Dave
>
>> I guess in virtio if you never signal the consumption of a virt-queue it
>> will still be there waiting until you restart the emulation process and
>> pick up from where you left off?
>> 
>> >
>> > Dave
>> >
>> >> Thoughts?
>> >> 
>> >> -- 
>> >> Alex Bennée
>> >> 
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>> >> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>> >> 
>> 
>> 
>> -- 
>> Alex Bennée
>> 


-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Dr. David Alan Gilbert]

* Alex Bennée (alex.bennee@linaro.org) wrote:
> 
> Dr. David Alan Gilbert <dgilbert@redhat.com> writes:
> 
> > * Alex Bennée (alex.bennee@linaro.org) wrote:
> >> 
> >> Dr. David Alan Gilbert <dgilbert@redhat.com> writes:
> >> 
> >> > * Alex Bennée (alex.bennee@linaro.org) wrote:
> >> >> Hi,
> >> >> 
> >> >> So the context of my question is what sort of common software layer is
> >> >> required to implement a virtio backend entirely in userspace?
> >> >> 
> >> >> Currently most virtio backends are embedded directly in various VMMs
> >> >> which emulate a number of devices as well as deal with handling devices
> >> >> that are vhost aware and link with the host kernel. However there seems
> >> >> to be a growing interest in having backends implemented in separate
> >> >> processes, potentially even hosted in other guest VMs.
> >> >> 
> >> >> As far as I can tell there is a lot of duplicated effort in handling the
> >> >> low level navigation of virt queues and buffers. QEMU has code in
> >> >> hw/virtio as well as contrib/libvhost-user which is used by the recent
> >> >> virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a
> >> >> similar set of functionality for it's emulation. The Rust-vmm project
> >> >> has libraries for implementing the device traits.
> >> >> 
> >> >> Another aspect to this is the growing interest in carrying virtio over
> >> >> other hypervisors. I'm wondering if there is enough abstraction possible
> >> >> to have a common library that is hypervisor agnostic? Can a device
> >> >> backend be emulated purely with some shared memory and some sockets for
> >> >> passing messages/kicks from/to the VMM which then deals with the hypervisor
> >> >> specifics of the virtio-transport?
> >> >
> >> > It's a little tricky because it has to interface tightly with the way
> >> > that the memory-mapping works for the hypervisor, so that the external
> >> > process can access the memory of the queues.
> >> 
> >> I suspect the problem space can at least be reduced to at least a
> >> POSIX-like environment - if that makes things simpler. The setting up of
> >> memory-mappings should be the problem of the VMM, which would possibly
> >> be hypervisor specific. After that it is simply(?) a question of sharing
> >> the appropriate bit of memory between the VMM and the device process.
> >
> > The 'simply(?)' is actually pretty tricky.
> 
> Well I am at the start of this journey so may be hand waving a bit ;-)
> 
> Lets drill down:
> 
> > You have to share the mapping of all the RAM blocks in whcih the virtio
> > queues or the data to which they point might reside
> 
> Aren't all the queues in one section of memory?

I don't think so; it's just allocated in guest RAM which can be split
into multiple blocks due to NUMA etc.

> As for where the data is doesn't this depend on the structure of the
> device. As I understand the behaviour of virtfs there is a direct
> relationship between the guest page cache and the host page cache to
> take advantage of DAX. This by definition means the backend needs access
> to the entire address space of the guest.
> 
> Is this also the case for other devices?

virtiofs's DAX shared memory is a bit different from normal virtio
queues and data.  Normal queues and data just live in normal guest RAM
and their location is chosen by the guest.


> > and you also have
> > to let the other process know where in Guest physical address space they
> > live.  That mapping is also not constant, either with hotplug, or with
> > architecture specific things that cause physical address mapping to
> > change.
> 
> It sounds like the solution here would be to have bounce buffers as part
> of the virtio spec which could be part of the virtio memory block? I
> guess another option is for guests to keep their internal data (as
> referenced by virtio drivers) in a fixed guest physical address but that
> gets real complicated quick and I suspect is harder to audit from a
> security point of view.

Bounce buffers are expensive - they're used in some things (like SEV
encrypted memory).

> >> The other model would be the device process runs inside another guest -
> >> most likely a Linux VM. Here the guest kernel can be told an area of
> >> memory is special in some way and provide a device node that can be
> >> mmaped in more or less the same way. In this configuration it can't even
> >> be aware of what the underlying hypervisor is - just a block of memory
> >> and a way to receive message queue events.
> >
> > Doing it between VMs works in my mind; but again you still need to
> > handle that mapping.
> >
> >> > QEMU's vhost-user has a fair amount of code for handling the mappings,
> >> > dirty logging for migration, iommu's and things like reset (which is
> >> > pretty hairy, and probably needs more work).
> >> 
> >> I suspect all of these multi-process models just hand wave away details
> >> like migration because that really does benefit from a single process
> >> with total awareness of the state of the system.
> >
> > Vhost-user has it pretty well defined; it works - as long as the user
> > process does dirty map update.  Postcopy can also be made to work.
> >
> >> That said I wonder how
> >> robust a guest can be if the device emulation may go away at any time?
> >
> > That one I've not thought too much about, but the opposite case; making
> > the separate process survive even when the guest behaves
> > badly/resets/etc is quite nasty.
> 
> I guess whatever orchestrates the start-up of the VMs has to worry about
> that. Some of the models I've been looking at have very simplistic
> setups where the guest VMs described in the platform data to be spawned
> directly by the hypervisor. I guess in those cases you need to restart
> everything.

That depends; the orchestrator doesn't normally see a guest reset - even
a nasty one.

Dave

> >
> > Dave
> >
> >> I guess in virtio if you never signal the consumption of a virt-queue it
> >> will still be there waiting until you restart the emulation process and
> >> pick up from where you left off?
> >> 
> >> >
> >> > Dave
> >> >
> >> >> Thoughts?
> >> >> 
> >> >> -- 
> >> >> Alex Bennée
> >> >> 
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> >> >> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >> >> 
> >> 
> >> 
> >> -- 
> >> Alex Bennée
> >> 
> 
> 
> -- 
> Alex Bennée
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2628 bytes --]

On Fri, Mar 06, 2020 at 06:33:57PM +0000, Alex Bennée wrote:
> So the context of my question is what sort of common software layer is
> required to implement a virtio backend entirely in userspace?

Yes and no.

vhost-user is currently the most widely-used (cross-VMM) interface for
out-of-process VIRTIO device emulation.

However, it's not a library, just a protocol specification involving
UNIX domain sockets.  You mentioned the various implementations below:

> Currently most virtio backends are embedded directly in various VMMs
> which emulate a number of devices as well as deal with handling devices
> that are vhost aware and link with the host kernel. However there seems
> to be a growing interest in having backends implemented in separate
> processes, potentially even hosted in other guest VMs.
> 
> As far as I can tell there is a lot of duplicated effort in handling the
> low level navigation of virt queues and buffers. QEMU has code in
> hw/virtio as well as contrib/libvhost-user which is used by the recent
> virtiofsd daemon. kvm-tool has a virtio subdirectory that implements a
> similar set of functionality for it's emulation. The Rust-vmm project
> has libraries for implementing the device traits.

You missed DPDK/SPDK.

The issue is that no single library would work for all of these users.

DPDK/SPDK does poll-mode device emulation.  Therefore the details are
slightly different from the eventfd-based approach that event-driven
implementations will use.

rust-vmm crates are written in Rust, so direct code sharing is not
possible.  Bindings for a C library are possible but is that desirable
(it defeats the point of using a safe language, it may be harder to add
features for developers mainly using Rust).

Unifying everything does not seem realistic for these sorts of reasons.
What we have today is that the Linux kernel uapi headers are shared, so
at least the struct and constant definitions don't need to be duplicated
by C/C++ projects.

> Another aspect to this is the growing interest in carrying virtio over
> other hypervisors. I'm wondering if there is enough abstraction possible
> to have a common library that is hypervisor agnostic? Can a device

That already exists today.  Pick a vhost-user device implementation from
QEMU, DPDK/SPDK, or rust-vmm and run it with your VMM of choice.

> backend be emulated purely with some shared memory and some sockets for
> passing messages/kicks from/to the VMM which then deals with the hypervisor
> specifics of the virtio-transport?

Yes, that is what vhost-user does.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Alex Bennée]

Stefan Hajnoczi <stefanha@redhat.com> writes:

> On Fri, Mar 06, 2020 at 06:33:57PM +0000, Alex Bennée wrote:
>> So the context of my question is what sort of common software layer is
>> required to implement a virtio backend entirely in userspace?
>
> Yes and no.
>
> vhost-user is currently the most widely-used (cross-VMM) interface for
> out-of-process VIRTIO device emulation.
>
> However, it's not a library, just a protocol specification involving
> UNIX domain sockets.  You mentioned the various implementations below:
>
<snip>
>
>> backend be emulated purely with some shared memory and some sockets for
>> passing messages/kicks from/to the VMM which then deals with the hypervisor
>> specifics of the virtio-transport?
>
> Yes, that is what vhost-user does.

I thought so - but does any vhost-user implementation assume it has
access to the entire of the guests memory space? I can see why that
might be seen as undesirable from a security point of view.

So aside from virtiofsd and crosvm's various pieces for crostini
emulation what other out-of-VMM device emulation daemons are you aware
of?

-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2225 bytes --]

On Mon, Mar 09, 2020 at 04:43:31PM +0000, Alex Bennée wrote:
> 
> Stefan Hajnoczi <stefanha@redhat.com> writes:
> 
> > On Fri, Mar 06, 2020 at 06:33:57PM +0000, Alex Bennée wrote:
> >> So the context of my question is what sort of common software layer is
> >> required to implement a virtio backend entirely in userspace?
> >
> > Yes and no.
> >
> > vhost-user is currently the most widely-used (cross-VMM) interface for
> > out-of-process VIRTIO device emulation.
> >
> > However, it's not a library, just a protocol specification involving
> > UNIX domain sockets.  You mentioned the various implementations below:
> >
> <snip>
> >
> >> backend be emulated purely with some shared memory and some sockets for
> >> passing messages/kicks from/to the VMM which then deals with the hypervisor
> >> specifics of the virtio-transport?
> >
> > Yes, that is what vhost-user does.
> 
> I thought so - but does any vhost-user implementation assume it has
> access to the entire of the guests memory space? I can see why that
> might be seen as undesirable from a security point of view.

VMMs typically give the vhost-user device program access to the entirety
of guest RAM.  That's because unmodified VIRTIO guest drivers typically
require access to all of guest memory.

If the guest software cooperates then it's possible to reduce the shared
memory region.  In other words, the vhost-user protocol doesn't demand
that guest RAM is shared, it just requires all addresses to be within
the shared memory region, whatever that may be.

> So aside from virtiofsd and crosvm's various pieces for crostini
> emulation what other out-of-VMM device emulation daemons are you aware
> of?

I haven't checked but my understanding was that crosvm uses its own
out-of-process device interface (not vhost-user).

The previously mentioned vhost-user ones were DPDK
(https://www.dpdk.org/) and SPDK (https://spdk.io/).

Sergio Lopez has implemented vhost-user-blk in Rust:
https://github.com/slp/qsd

I think Nutanix is also using vhost-user, but am not sure and I don't
know if that is open source.

There are probably more that I'm not aware of but a web search would
turn up.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Halil Pasic]

[-- Attachment #1: Type: text/plain, Size: 1958 bytes --]

On Wed, 11 Mar 2020 17:24:11 +0000
Stefan Hajnoczi <stefanha@redhat.com> wrote:

> > I thought so - but does any vhost-user implementation assume it has
> > access to the entire of the guests memory space? I can see why that
> > might be seen as undesirable from a security point of view.  
> 

On s390 we do assume we have access to the whole guest memory
space if VIRTIO_F_ACCESS_PLATFORM is not set.

> VMMs typically give the vhost-user device program access to the entirety
> of guest RAM.  That's because unmodified VIRTIO guest drivers typically
> require access to all of guest memory.
> 
> If the guest software cooperates then it's possible to reduce the shared
> memory region.  In other words, the vhost-user protocol doesn't demand
> that guest RAM is shared, it just requires all addresses to be within
> the shared memory region, whatever that may be.

In that case, I guess, we would have to negotiate
VIRTIO_F_ACCESS_PLATFORM. vhost-user already has support for
VIRTIO_F_ACCESS_PLATFORM, but in context of vhost-user it means: do
IOVA translation. Translation is basically boils down to IOMMU/IOTLB
support, and the specs (docs/interop/vhost-user.rst) says that the
translated stuff needs to be within a region set via
VHOST_USER_SET_MEM_TABLE.

I'm wondering how restricting the memory region shared vhost-user program
should work, unless we impose restrictions on what physical addresses
may be used for virtio buffers (regardless of whether what lands in the
virtqueue is an IOVA or a GPA), and make everybody in the stack aware of
this.

Of course sharing the whole guest memory space with the vhost-user
program does not necessarily mean the guest can actually access the
whole guest memory. For example memory encryption or other means of
protecting memory (e.g. s390 protected VMs) may ensure that the
vhost-user program can not mess with anything that in is not supposed to.

Regards,
Halil

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Srivatsa Vaddagiri]

On Mon, Mar 9, 2020 at 9:16 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> That already exists today.  Pick a vhost-user device implementation from
> QEMU, DPDK/SPDK, or rust-vmm and run it with your VMM of choice.
>
> > backend be emulated purely with some shared memory and some sockets for
> > passing messages/kicks from/to the VMM which then deals with the hypervisor
> > specifics of the virtio-transport?
>
> Yes, that is what vhost-user does.

Are there vhost-user based backends that emulates block and console devices?

From my quick read, vhost-user seems to depend on having backend complete access
to front-end's memory. Is that true? We are looking for a backend solution that
does not assume access to whole of front-end memory. Only the memory regions
hosting IO buffers need to be ideally shared between the two VMs. Also the
mechanism of memory sharing and signalling between VMs could be very
hypervisor specific.
Does vhost-user allow for such hyp-specific memory sharing and
signalling interfaces?

- vatsa

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Alex Bennée]

Srivatsa Vaddagiri <vatsa.ml@gmail.com> writes:

> On Mon, Mar 9, 2020 at 9:16 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>
>> That already exists today.  Pick a vhost-user device implementation from
>> QEMU, DPDK/SPDK, or rust-vmm and run it with your VMM of choice.
>>
>> > backend be emulated purely with some shared memory and some sockets for
>> > passing messages/kicks from/to the VMM which then deals with the hypervisor
>> > specifics of the virtio-transport?
>>
>> Yes, that is what vhost-user does.
>
> Are there vhost-user based backends that emulates block and console devices?
>
> From my quick read, vhost-user seems to depend on having backend complete access
> to front-end's memory. Is that true? We are looking for a backend solution that
> does not assume access to whole of front-end memory. Only the memory regions
> hosting IO buffers need to be ideally shared between the two VMs.

I think it currently is true because generally the pointers passed in
the virtqueues are directly into the various internal structures of the
kernel (e.g. pagecache, skb etc.).

Now for linux the are ways of declaring particular buffers to be in
specific locations - ostensibly for things like DMA-ability. But AFAICT
there is no mechanism to communicate the particular locations to be used
to the VMM and backend daemon.

> Also the
> mechanism of memory sharing and signalling between VMs could be very
> hypervisor specific.
> Does vhost-user allow for such hyp-specific memory sharing and
> signalling interfaces?

The daemons I've looked at so far are passed a shm fd and a control fd
from the VMM. From their point of view it's just posix shared memory and
a event queue via a socket. Setting that up is the VMMs problem.

>
> - vatsa


-- 
Alex Bennée

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Backend libraries for VirtIO device emulation

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2451 bytes --]

On Mon, Mar 09, 2020 at 05:42:39PM +0000, Alex Bennée wrote:
> 
> Srivatsa Vaddagiri <vatsa.ml@gmail.com> writes:
> 
> > On Mon, Mar 9, 2020 at 9:16 PM Stefan Hajnoczi <stefanha@redhat.com> wrote:
> >>
> >> That already exists today.  Pick a vhost-user device implementation from
> >> QEMU, DPDK/SPDK, or rust-vmm and run it with your VMM of choice.
> >>
> >> > backend be emulated purely with some shared memory and some sockets for
> >> > passing messages/kicks from/to the VMM which then deals with the hypervisor
> >> > specifics of the virtio-transport?
> >>
> >> Yes, that is what vhost-user does.
> >
> > Are there vhost-user based backends that emulates block and console devices?
> >
> > From my quick read, vhost-user seems to depend on having backend complete access
> > to front-end's memory. Is that true? We are looking for a backend solution that
> > does not assume access to whole of front-end memory. Only the memory regions
> > hosting IO buffers need to be ideally shared between the two VMs.
> 
> I think it currently is true because generally the pointers passed in
> the virtqueues are directly into the various internal structures of the
> kernel (e.g. pagecache, skb etc.).
> 
> Now for linux the are ways of declaring particular buffers to be in
> specific locations - ostensibly for things like DMA-ability. But AFAICT
> there is no mechanism to communicate the particular locations to be used
> to the VMM and backend daemon.

Yes, the problem is that guest userspace memory buffers could be
anywhere and the virtqueue DMA memory managed by the guest drivers is
also not allocated from a specific memory range.

When the guest cooperates (e.g. DPDK applications that use a fixed
memory range for virtio-net packet buffers), then it's possible to
reduce the shared memory region to not expose all of guest RAM.

> > Also the
> > mechanism of memory sharing and signalling between VMs could be very
> > hypervisor specific.
> > Does vhost-user allow for such hyp-specific memory sharing and
> > signalling interfaces?
> 
> The daemons I've looked at so far are passed a shm fd and a control fd
> from the VMM. From their point of view it's just posix shared memory and
> a event queue via a socket. Setting that up is the VMMs problem.

Yes, although I don't think there is any assumption that the fd is POSIX
shared memory.  The fd just needs to support mmap(2).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]