Why QEMU should move from C to Rust (clickbait alert ;))

Why QEMU should move from C to Rust (clickbait alert ;))

[Stefan Hajnoczi]

I just posted this on my blog, but my intention is to continue the
discussion around moving to Rust in the QEMU community. Thoughts about
how to increase safety in QEMU, both for and against moving to Rust
are welcome!

My KVM Forum 2018 presentation titled Security in QEMU: How Virtual
Machines provide Isolation reviewed security bugs in QEMU and found
the most common causes were C programming bugs. This includes buffer
overflows, use-after-free, uninitialized memory, and more. In this
post I will argue for using Rust as a safer language that prevents
these classes of bugs.

In 2018 the choice of a safer language was not clear. C++ offered safe
abstractions without an effective way to prohibit unsafe language
features. Go also offered safety but with concerns about runtime
costs. Rust looked promising but few people had deep experience with
it. In 2018 I was not able to argue confidently for moving away from C
in QEMU.

Now in 2020 the situation is clearer. C programming bugs are still the
main cause of CVEs in QEMU. Rust has matured, its ecosystem is growing
and healthy, and there are virtualization projects like Crosvm,
Firecracker, and cloud-hypervisor that prove Rust is an effective
language for writing Virtual Machine Monitors (VMM). In the QEMU
community Paolo Bonzini and Sergio Lopez's work on rust-vmm and
vhost-user code inspired me to look more closely at moving away from
C.

Do we need to change programming language?
---------------------------------------------------------------
Most security bugs in QEMU are C programming bugs. This is easy to
verify by looking through the CVE listings. Although I have only
reviewed CVEs it seems likely that non-security bugs are also mostly C
programming bugs.

Eliminating C programming bugs does not necessarily require switching
programming languages. Other approaches to reducing bug rates in
software include:

Coding style rules that forbid unsafe language features.
Building safe abstractions and prohibiting unsafe language features or
library APIs.
Static checkers that scan source code for bugs.
Dynamic sanitizers that run software with instrumentation to identify bugs.
Unit testing and fuzzing.

The problem is, the QEMU community has been doing these things for
years but new bugs are still introduced despite these efforts. It is
certainly possible to spend more energy on these efforts but the
evidence shows that bugs continue to slip through.

There are two issues with these approaches to reducing bugs. First,
although these approaches help find existing bugs, eliminating classes
of bugs so they cannot exist in the first place is a stronger
approach. This is hard to do with C since the language is unsafe,
placing the burden of safety on the programmer.

Second, much of the ability to write safe C code comes with
experience. Custom conventions, APIs, tooling, and processes to reduce
bugs is a hurdle for one-time contributors or newcomers. It makes the
codebase inaccessible unless we accept lower standards for some
contributors. Code quality should depend as little on experience as
possible but C is notorious for being a programming language that
requires a lot of practice before you can write production-quality
code.

Why Rust?
--------------
Safe languages eliminate memory safety bugs (and other classes like
concurrency bugs). Rust made this a priority in its design:

Use-after-free, double-free, memory leaks, and other lifetime bugs are
prevented at compile-time by the borrow checker where the compiler
checks ownership of data.
Buffer overflows and other memory corruptions are prevented by
compile-time and runtime bounds-checking.
Pointer deference bugs are prevented by the absense of NULL pointers
and strict ownership rules.
Uninitialized memory is prevented because all variables and fields
must be initialized.

Rust programs can still "panic" at runtime when safety cannot be
proven at compile time but this does not result in undefined behavior
as seen in C programs. The program simply aborts with a backtrace.
Bugs that could have resulted in arbitrary code execution in C become
at most denial-of-service bugs in Rust. This reduces the severity of
bugs.

As a result of this language design most C programming bugs that
plague QEMU today are either caught by the compiler or turn into a
safe program termination. It is reasonable to expect CVEs to reduce in
number and in severity when switching to Rust.

At the same time Rust eliminates the need for many of the measures
that the QEMU community added onto C because the Rust programming
language and its compiler already enforce safety. This means newcomers
and one-time contributors will not need QEMU-specific experience, can
write production-quality code more easily, and can get their code
merged more quickly. It also means reviewers will have to spend less
time pointing out C programming bugs or asking for changes that comply
with QEMU's way of doing things.

That said, Rust has a reputation for being a scary language due to the
borrow checker. Most programmers have not thought about object
lifetimes and ownership as systematically and explicitly as required
by Rust. This raises the bar to learning the language, but I look at
it this way: learning Rust is humanly possible, writing bug-free C
code is not.

How can we change programming language?
------------------------------------------------------------
When I checked in 2018 QEMU was 1.5 million lines of code. It has
grown since then. Moving a large codebase to a new programming
language is extremely difficult. If people want to convert QEMU to
Rust that would be great, but I personally don't have the appetite to
do it because I think the integration will be messy, result in a lot
of duplication, and there is too much un(der)maintained code that is
hard to convert.

The reason I am writing this post is because device emulation, the
main security attack surface for VMMs, can be done in a separate
program. That program can be written in any language and this is where
Rust comes in. For vhost devices it is possible to write Rust device
backends today and I hope this will become the default approach to
writing new devices.

For non-vhost devices the vfio-user project is working on an interface
out-of-process device emulation. It will be possible to implement
devices in Rust there too.

If you are implementing new device emulation code please consider
doing it in Rust!

Conclusion
---------------
Most security bugs in QEMU today are C programming bugs. Switching to
a safer programming language will significantly reduce security bugs
in QEMU. Rust is now mature and proven enough to use as the language
for device emulation code. Thanks to vhost-user and vfio-user using
Rust for device emulation does not require a big conversion of QEMU
code, it can simply be done in a separate program. This way attack
surfaces can be written in Rust to make them less susceptible to
security bugs going forward.

Re: Why QEMU should move from C to Rust (clickbait alert ;))

[Daniel P. Berrangé]

On Thu, Aug 06, 2020 at 11:24:13AM +0100, Stefan Hajnoczi wrote:

...snip...

I broadly agree with the points about why Rust is desirable as
a language for QEMU.

> How can we change programming language?
> ------------------------------------------------------------
> When I checked in 2018 QEMU was 1.5 million lines of code. It has
> grown since then. Moving a large codebase to a new programming
> language is extremely difficult. If people want to convert QEMU to
> Rust that would be great, but I personally don't have the appetite to
> do it because I think the integration will be messy, result in a lot
> of duplication, and there is too much un(der)maintained code that is
> hard to convert.
> 
> The reason I am writing this post is because device emulation, the
> main security attack surface for VMMs, can be done in a separate
> program. That program can be written in any language and this is where
> Rust comes in. For vhost devices it is possible to write Rust device
> backends today and I hope this will become the default approach to
> writing new devices.
> 
> For non-vhost devices the vfio-user project is working on an interface
> out-of-process device emulation. It will be possible to implement
> devices in Rust there too.

I guess this dovetails into our view of security bug classification.
Stuff that affects virtualization use cases is security critical,
while stuff that affects emulation is NOT security critical.

With this in mind, it is not a priority to convert the whole of
QEMU to Rust. The priority is the subset that affects virtualization
use cases. It is a much more tractable problem if we're looking at
creating rust based vhost-user / vfio-user, only for the devices
that are commonly used for KVM with modern OS, and ignore the many
100's of devices that are only there for emulation or foir legacy OS.


> If you are implementing new device emulation code please consider
> doing it in Rust!

Yes, but I think we'll need put in significant effort to guide / assist
people in taking this direction, and think about what it means for the
future of QEMU as a brand and GIT repo.

In many ways it is a good thing if the Rust vhost-user impls are
all in their own standalone git repos. They're likely to be independent
codebases, so there's little compelling reason to force them into the
QEMU git, where they'll have to use QEMU workflow, and QEMU release
cycle. They're better off free from QEMU where they can choose to adopt
modern development practices like GitLab merge requests if they
desire and release on a more frequent cycle than QEMU's 3-times a
year, etc. Would also make them more appealing for use by alternative
non-QEMU userspaces for KVM.

The downside is that QEMU git would only contain the "legacy" builtin
C impls of the devices, and all the "recommended" modern Rust impls
would be elsewhere. Essentially QEMU would no longer be a self-contained
provider of the complete solution. Many parts would be disaggregated,
and users now have the burden of finding all the right pieces to build
the best solution. We've already seen this to some extent with existing
vhost-user impls, but it feels like we'd be pushing towards that as a
more general model for the future which would amplify problems we've
largely been able to ignore upto now.

I'm not sure what a good answer here is. Perhaps QEMU could try to
become more of brand for an umbrella project that covers multiple
independant repos ? eg create new repos under gitlab.com/qemu-project/
but allow them to work fairly independantly from the main qemu.git ?
That way we can more easily promote a collection of QEMU repos as
showing the recommended architecture, without forcing everything
into qemu.git. We can leverage the QEMU website, wiki and documentation
in general to showcase the overall solution, while still letting the
pieces develop independently.

> Conclusion
> ---------------
> Most security bugs in QEMU today are C programming bugs. Switching to
> a safer programming language will significantly reduce security bugs
> in QEMU. Rust is now mature and proven enough to use as the language
> for device emulation code. Thanks to vhost-user and vfio-user using
> Rust for device emulation does not require a big conversion of QEMU
> code, it can simply be done in a separate program. This way attack
> surfaces can be written in Rust to make them less susceptible to
> security bugs going forward.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Why QEMU should move from C to Rust (clickbait alert ;))

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Thu, Aug 06, 2020 at 11:24:13AM +0100, Stefan Hajnoczi wrote:
>
> ...snip...
>
> I broadly agree with the points about why Rust is desirable as
> a language for QEMU.
>
>> How can we change programming language?
>> ------------------------------------------------------------
>> When I checked in 2018 QEMU was 1.5 million lines of code. It has
>> grown since then. Moving a large codebase to a new programming
>> language is extremely difficult. If people want to convert QEMU to
>> Rust that would be great, but I personally don't have the appetite to
>> do it because I think the integration will be messy, result in a lot
>> of duplication, and there is too much un(der)maintained code that is
>> hard to convert.
>> 
>> The reason I am writing this post is because device emulation, the
>> main security attack surface for VMMs, can be done in a separate
>> program. That program can be written in any language and this is where
>> Rust comes in. For vhost devices it is possible to write Rust device
>> backends today and I hope this will become the default approach to
>> writing new devices.
>> 
>> For non-vhost devices the vfio-user project is working on an interface
>> out-of-process device emulation. It will be possible to implement
>> devices in Rust there too.
>
> I guess this dovetails into our view of security bug classification.
> Stuff that affects virtualization use cases is security critical,
> while stuff that affects emulation is NOT security critical.
>
> With this in mind, it is not a priority to convert the whole of
> QEMU to Rust. The priority is the subset that affects virtualization
> use cases. It is a much more tractable problem if we're looking at
> creating rust based vhost-user / vfio-user, only for the devices
> that are commonly used for KVM with modern OS, and ignore the many
> 100's of devices that are only there for emulation or foir legacy OS.

I'm definitely interested in vhost-user daemons in Rust. The one I've
been writing so far has been tried to be a minimal POSIX base without
including QEMU utility functions but I'm sure it would be nicer to start
with a clean standalone Rust library.

>> If you are implementing new device emulation code please consider
>> doing it in Rust!
>
> Yes, but I think we'll need put in significant effort to guide / assist
> people in taking this direction, and think about what it means for the
> future of QEMU as a brand and GIT repo.
>
> In many ways it is a good thing if the Rust vhost-user impls are
> all in their own standalone git repos. They're likely to be independent
> codebases, so there's little compelling reason to force them into the
> QEMU git, where they'll have to use QEMU workflow, and QEMU release
> cycle. They're better off free from QEMU where they can choose to adopt
> modern development practices like GitLab merge requests if they
> desire and release on a more frequent cycle than QEMU's 3-times a
> year, etc. Would also make them more appealing for use by alternative
> non-QEMU userspaces for KVM.
>
> The downside is that QEMU git would only contain the "legacy" builtin
> C impls of the devices, and all the "recommended" modern Rust impls
> would be elsewhere. Essentially QEMU would no longer be a self-contained
> provider of the complete solution. Many parts would be disaggregated,
> and users now have the burden of finding all the right pieces to build
> the best solution. We've already seen this to some extent with existing
> vhost-user impls, but it feels like we'd be pushing towards that as a
> more general model for the future which would amplify problems we've
> largely been able to ignore upto now.
>
> I'm not sure what a good answer here is. Perhaps QEMU could try to
> become more of brand for an umbrella project that covers multiple
> independant repos ? eg create new repos under gitlab.com/qemu-project/
> but allow them to work fairly independantly from the main qemu.git ?
> That way we can more easily promote a collection of QEMU repos as
> showing the recommended architecture, without forcing everything
> into qemu.git. We can leverage the QEMU website, wiki and documentation
> in general to showcase the overall solution, while still letting the
> pieces develop independently.

I think there would be scope for a vhost-user-daemon repo with multiple
reference implementations.

So has any vhost-user stuff been prototyped in Rust yet?

>
>> Conclusion
>> ---------------
>> Most security bugs in QEMU today are C programming bugs. Switching to
>> a safer programming language will significantly reduce security bugs
>> in QEMU. Rust is now mature and proven enough to use as the language
>> for device emulation code. Thanks to vhost-user and vfio-user using
>> Rust for device emulation does not require a big conversion of QEMU
>> code, it can simply be done in a separate program. This way attack
>> surfaces can be written in Rust to make them less susceptible to
>> security bugs going forward.
>
>
> Regards,
> Daniel


-- 
Alex Bennée

Re: Why QEMU should move from C to Rust (clickbait alert ;))

[Stefan Hajnoczi]

On Thu, Aug 6, 2020 at 2:39 PM Alex Bennée <alex.bennee@linaro.org> wrote:
> So has any vhost-user stuff been prototyped in Rust yet?

Yes, Sergio implemented vhost-user-blk here:
https://github.com/slp/qsd

cloud-hypervisor also vhost-user device backends, like vhost-user-fs.

Stefan

Re: Why QEMU should move from C to Rust (clickbait alert ;))

[Stefan Hajnoczi]

On Thu, Aug 6, 2020 at 12:08 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> Yes, but I think we'll need put in significant effort to guide / assist
> people in taking this direction, and think about what it means for the
> future of QEMU as a brand and GIT repo.
>
> In many ways it is a good thing if the Rust vhost-user impls are
> all in their own standalone git repos. They're likely to be independent
> codebases, so there's little compelling reason to force them into the
> QEMU git, where they'll have to use QEMU workflow, and QEMU release
> cycle. They're better off free from QEMU where they can choose to adopt
> modern development practices like GitLab merge requests if they
> desire and release on a more frequent cycle than QEMU's 3-times a
> year, etc. Would also make them more appealing for use by alternative
> non-QEMU userspaces for KVM.
>
> The downside is that QEMU git would only contain the "legacy" builtin
> C impls of the devices, and all the "recommended" modern Rust impls
> would be elsewhere. Essentially QEMU would no longer be a self-contained
> provider of the complete solution. Many parts would be disaggregated,
> and users now have the burden of finding all the right pieces to build
> the best solution. We've already seen this to some extent with existing
> vhost-user impls, but it feels like we'd be pushing towards that as a
> more general model for the future which would amplify problems we've
> largely been able to ignore upto now.
>
> I'm not sure what a good answer here is. Perhaps QEMU could try to
> become more of brand for an umbrella project that covers multiple
> independant repos ? eg create new repos under gitlab.com/qemu-project/
> but allow them to work fairly independantly from the main qemu.git ?
> That way we can more easily promote a collection of QEMU repos as
> showing the recommended architecture, without forcing everything
> into qemu.git. We can leverage the QEMU website, wiki and documentation
> in general to showcase the overall solution, while still letting the
> pieces develop independently.

I agree.

Working independently and following Rust conventions ought to work
well for external programs like vhost-user device backends.

It's important that the QEMU source tree builds a complete system. Git
submodules can help with that and are already widely used today.

Stefan

Re: Why QEMU should move from C to Rust (clickbait alert ;))

[Sergio Lopez]

[-- Attachment #1: Type: text/plain, Size: 1836 bytes --]

On Thu, Aug 06, 2020 at 11:24:13AM +0100, Stefan Hajnoczi wrote:
<snip>
> Conclusion
> ---------------
> Most security bugs in QEMU today are C programming bugs. Switching to
> a safer programming language will significantly reduce security bugs
> in QEMU. Rust is now mature and proven enough to use as the language
> for device emulation code. Thanks to vhost-user and vfio-user using
> Rust for device emulation does not require a big conversion of QEMU
> code, it can simply be done in a separate program. This way attack
> surfaces can be written in Rust to make them less susceptible to
> security bugs going forward.
> 

Having worked on Rust implementations for vhost-user-fs and
vhost-user-blk, I'm 100% sold on this idea.

That said, there are a couple things that I think may help getting
more people into implementing vhost-user devices in Rust.

 1. Having a reference implementation for a simple device somewhere
 close or inside the QEMU source tree. I'd say vhost-user-blk is a
 clear candidate, given that a naive implementation for raw files
 without any I/O optimization is quite easy to read and understand.

 2. Integrating the ability to start-up vhost-user daemons from QEMU,
 in an easy and portable way. I know we can always rely on daemons
 like libvirt to do this for us, but I think it'd be nicer to be able
 to define a vhost-user device from the command line and have QEMU
 execute it with the proper parameters (BTW, Cloud-Hypervisor already
 does that). This would probably require some kind of configuration
 file, to be able to define which binary provides each vhost-user
 device personality, but could also be a way for "sanctioning"
 daemons (through the configuration defaults), and to have them adhere
 to a standardized command line format.

Thanks,
Sergio.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Why QEMU should move from C to Rust (clickbait alert ;))

[Daniel P. Berrangé]

On Thu, Aug 06, 2020 at 01:51:48PM +0200, Sergio Lopez wrote:
> On Thu, Aug 06, 2020 at 11:24:13AM +0100, Stefan Hajnoczi wrote:
> <snip>
> > Conclusion
> > ---------------
> > Most security bugs in QEMU today are C programming bugs. Switching to
> > a safer programming language will significantly reduce security bugs
> > in QEMU. Rust is now mature and proven enough to use as the language
> > for device emulation code. Thanks to vhost-user and vfio-user using
> > Rust for device emulation does not require a big conversion of QEMU
> > code, it can simply be done in a separate program. This way attack
> > surfaces can be written in Rust to make them less susceptible to
> > security bugs going forward.
> > 
> 
> Having worked on Rust implementations for vhost-user-fs and
> vhost-user-blk, I'm 100% sold on this idea.
> 
> That said, there are a couple things that I think may help getting
> more people into implementing vhost-user devices in Rust.
> 
>  1. Having a reference implementation for a simple device somewhere
>  close or inside the QEMU source tree. I'd say vhost-user-blk is a
>  clear candidate, given that a naive implementation for raw files
>  without any I/O optimization is quite easy to read and understand.
> 
>  2. Integrating the ability to start-up vhost-user daemons from QEMU,
>  in an easy and portable way. I know we can always rely on daemons
>  like libvirt to do this for us, but I think it'd be nicer to be able
>  to define a vhost-user device from the command line and have QEMU
>  execute it with the proper parameters (BTW, Cloud-Hypervisor already
>  does that). This would probably require some kind of configuration
>  file, to be able to define which binary provides each vhost-user
>  device personality, but could also be a way for "sanctioning"
>  daemons (through the configuration defaults), and to have them adhere
>  to a standardized command line format.

This second point is such a good idea that we already have defined
how todo this in QEMU - see the docs/interop/vhost-user.json file.
This specifies metadata files that should be installed into a
defined location such that QEMU/libvirt/other mgmt app can locate
vhost-user impls for each type of device, and priortize between
different impls.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Why QEMU should move from C to Rust (clickbait alert ;))

[Sergio Lopez]

[-- Attachment #1: Type: text/plain, Size: 2627 bytes --]

On Thu, Aug 06, 2020 at 01:01:30PM +0100, Daniel P. Berrangé wrote:
> On Thu, Aug 06, 2020 at 01:51:48PM +0200, Sergio Lopez wrote:
> > On Thu, Aug 06, 2020 at 11:24:13AM +0100, Stefan Hajnoczi wrote:
> > <snip>
> > > Conclusion
> > > ---------------
> > > Most security bugs in QEMU today are C programming bugs. Switching to
> > > a safer programming language will significantly reduce security bugs
> > > in QEMU. Rust is now mature and proven enough to use as the language
> > > for device emulation code. Thanks to vhost-user and vfio-user using
> > > Rust for device emulation does not require a big conversion of QEMU
> > > code, it can simply be done in a separate program. This way attack
> > > surfaces can be written in Rust to make them less susceptible to
> > > security bugs going forward.
> > > 
> > 
> > Having worked on Rust implementations for vhost-user-fs and
> > vhost-user-blk, I'm 100% sold on this idea.
> > 
> > That said, there are a couple things that I think may help getting
> > more people into implementing vhost-user devices in Rust.
> > 
> >  1. Having a reference implementation for a simple device somewhere
> >  close or inside the QEMU source tree. I'd say vhost-user-blk is a
> >  clear candidate, given that a naive implementation for raw files
> >  without any I/O optimization is quite easy to read and understand.
> > 
> >  2. Integrating the ability to start-up vhost-user daemons from QEMU,
> >  in an easy and portable way. I know we can always rely on daemons
> >  like libvirt to do this for us, but I think it'd be nicer to be able
> >  to define a vhost-user device from the command line and have QEMU
> >  execute it with the proper parameters (BTW, Cloud-Hypervisor already
> >  does that). This would probably require some kind of configuration
> >  file, to be able to define which binary provides each vhost-user
> >  device personality, but could also be a way for "sanctioning"
> >  daemons (through the configuration defaults), and to have them adhere
> >  to a standardized command line format.
> 
> This second point is such a good idea that we already have defined
> how todo this in QEMU - see the docs/interop/vhost-user.json file.
> This specifies metadata files that should be installed into a
> defined location such that QEMU/libvirt/other mgmt app can locate
> vhost-user impls for each type of device, and priortize between
> different impls.

Nice, but AFAIK QEMU still lacks the ability to process those files
and run the vhost-user device providers by itself. Or perhaps I just
can't find it (?).

Sergio.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Why QEMU should move from C to Rust (clickbait alert ;))

[Daniel P. Berrangé]

On Thu, Aug 06, 2020 at 03:38:45PM +0200, Sergio Lopez wrote:
> On Thu, Aug 06, 2020 at 01:01:30PM +0100, Daniel P. Berrangé wrote:
> > On Thu, Aug 06, 2020 at 01:51:48PM +0200, Sergio Lopez wrote:
> > > On Thu, Aug 06, 2020 at 11:24:13AM +0100, Stefan Hajnoczi wrote:
> > > <snip>
> > > > Conclusion
> > > > ---------------
> > > > Most security bugs in QEMU today are C programming bugs. Switching to
> > > > a safer programming language will significantly reduce security bugs
> > > > in QEMU. Rust is now mature and proven enough to use as the language
> > > > for device emulation code. Thanks to vhost-user and vfio-user using
> > > > Rust for device emulation does not require a big conversion of QEMU
> > > > code, it can simply be done in a separate program. This way attack
> > > > surfaces can be written in Rust to make them less susceptible to
> > > > security bugs going forward.
> > > > 
> > > 
> > > Having worked on Rust implementations for vhost-user-fs and
> > > vhost-user-blk, I'm 100% sold on this idea.
> > > 
> > > That said, there are a couple things that I think may help getting
> > > more people into implementing vhost-user devices in Rust.
> > > 
> > >  1. Having a reference implementation for a simple device somewhere
> > >  close or inside the QEMU source tree. I'd say vhost-user-blk is a
> > >  clear candidate, given that a naive implementation for raw files
> > >  without any I/O optimization is quite easy to read and understand.
> > > 
> > >  2. Integrating the ability to start-up vhost-user daemons from QEMU,
> > >  in an easy and portable way. I know we can always rely on daemons
> > >  like libvirt to do this for us, but I think it'd be nicer to be able
> > >  to define a vhost-user device from the command line and have QEMU
> > >  execute it with the proper parameters (BTW, Cloud-Hypervisor already
> > >  does that). This would probably require some kind of configuration
> > >  file, to be able to define which binary provides each vhost-user
> > >  device personality, but could also be a way for "sanctioning"
> > >  daemons (through the configuration defaults), and to have them adhere
> > >  to a standardized command line format.
> > 
> > This second point is such a good idea that we already have defined
> > how todo this in QEMU - see the docs/interop/vhost-user.json file.
> > This specifies metadata files that should be installed into a
> > defined location such that QEMU/libvirt/other mgmt app can locate
> > vhost-user impls for each type of device, and priortize between
> > different impls.
> 
> Nice, but AFAIK QEMU still lacks the ability to process those files
> and run the vhost-user device providers by itself. Or perhaps I just
> can't find it (?).

You're correct, thus far QEMU merely defined the standard, but thus
far libvirt is the only consumer I know of that implemented it. It
was anticipated that QEMU would implement support too but no one
has attempted it to my knowledge.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Why QEMU should move from C to Rust (clickbait alert ;))

[Stefan Hajnoczi]

On Thu, Aug 6, 2020 at 2:38 PM Sergio Lopez <slp@redhat.com> wrote:
> On Thu, Aug 06, 2020 at 01:01:30PM +0100, Daniel P. Berrangé wrote:
> > On Thu, Aug 06, 2020 at 01:51:48PM +0200, Sergio Lopez wrote:
> > > On Thu, Aug 06, 2020 at 11:24:13AM +0100, Stefan Hajnoczi wrote:
> > > <snip>
> > > > Conclusion
> > > > ---------------
> > > > Most security bugs in QEMU today are C programming bugs. Switching to
> > > > a safer programming language will significantly reduce security bugs
> > > > in QEMU. Rust is now mature and proven enough to use as the language
> > > > for device emulation code. Thanks to vhost-user and vfio-user using
> > > > Rust for device emulation does not require a big conversion of QEMU
> > > > code, it can simply be done in a separate program. This way attack
> > > > surfaces can be written in Rust to make them less susceptible to
> > > > security bugs going forward.
> > > >
> > >
> > > Having worked on Rust implementations for vhost-user-fs and
> > > vhost-user-blk, I'm 100% sold on this idea.
> > >
> > > That said, there are a couple things that I think may help getting
> > > more people into implementing vhost-user devices in Rust.
> > >
> > >  1. Having a reference implementation for a simple device somewhere
> > >  close or inside the QEMU source tree. I'd say vhost-user-blk is a
> > >  clear candidate, given that a naive implementation for raw files
> > >  without any I/O optimization is quite easy to read and understand.
> > >
> > >  2. Integrating the ability to start-up vhost-user daemons from QEMU,
> > >  in an easy and portable way. I know we can always rely on daemons
> > >  like libvirt to do this for us, but I think it'd be nicer to be able
> > >  to define a vhost-user device from the command line and have QEMU
> > >  execute it with the proper parameters (BTW, Cloud-Hypervisor already
> > >  does that). This would probably require some kind of configuration
> > >  file, to be able to define which binary provides each vhost-user
> > >  device personality, but could also be a way for "sanctioning"
> > >  daemons (through the configuration defaults), and to have them adhere
> > >  to a standardized command line format.
> >
> > This second point is such a good idea that we already have defined
> > how todo this in QEMU - see the docs/interop/vhost-user.json file.
> > This specifies metadata files that should be installed into a
> > defined location such that QEMU/libvirt/other mgmt app can locate
> > vhost-user impls for each type of device, and priortize between
> > different impls.
>
> Nice, but AFAIK QEMU still lacks the ability to process those files
> and run the vhost-user device providers by itself. Or perhaps I just
> can't find it (?).

How about a Python script? It can list available vhost-user programs
and their options:

  $ qemu-vhost-launcher list
  vhost-user-gpu Paravirtualized GPU with OpenGL support
  ...vhost-user-gpu --help output...
  vhost-user-blk Block device
  ...vhost-user-blk --help output...

You can write a configuration file:

  $ cat vhost-devices.conf # or yaml/toml/json?
  [blk1]
  backend=vhost-user-blk
  blk-file=/path/to/image.img

  [iso]
  backend=vhost-user-blk
  blk-file=/path/to/installer.iso
  read-only=true

And launch QEMU like this:

  $ qemu-vhost-launcher run -c vhost-devices.conf -- -M accel=kvm -m
1G -cpu host

The 'run' command builds QEMU command-line options for the devices
described in the configuration file. It launches the vhost-user device
backends and then QEMU. The QEMU vhost-user command-line options
(-chardev, -device) are appended to the command-line options.

Stefan

Re: Why QEMU should move from C to Rust (clickbait alert ;))

[Stefan Hajnoczi]

On Fri, Aug 7, 2020 at 10:27 AM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > Nice, but AFAIK QEMU still lacks the ability to process those files
> > and run the vhost-user device providers by itself. Or perhaps I just
> > can't find it (?).
>
> How about a Python script? It can list available vhost-user programs
> and their options:

Another option is systemd unit files. It's lower-level than a
dedicated tool but gives full control and it should be possible to
bring up an entire set of processes using a single systemctl command.

Stefan

Re: Why QEMU should move from C to Rust (clickbait alert ;))

[Stefan Hajnoczi]

On Thu, Aug 6, 2020 at 12:51 PM Sergio Lopez <slp@redhat.com> wrote:
>  1. Having a reference implementation for a simple device somewhere
>  close or inside the QEMU source tree. I'd say vhost-user-blk is a
>  clear candidate, given that a naive implementation for raw files
>  without any I/O optimization is quite easy to read and understand.

Yes, it's important to have crates that make it easy to build device backends.

The following common areas come to mind:
1. vhost-user protocol and vring APIs.
2. vhost-user conventions for command-line options.
3. Live migration support.
4. Sandboxing (seccomp, etc).
5. Management interface.

They can all be separate crates and you can choose which ones to use.

Rust-vmm and cloud-hypervisor have already started on some of these.
It would be nice to work together.

Stefan

Re: Why QEMU should move from C to Rust (clickbait alert ;))

[Alex Carter]

[-- Attachment #1: Type: text/plain, Size: 3320 bytes --]

Hi everyone,

My name is Alex, I’m a student at the University of Michigan and I just
completed an internship at IBM Research. There, I have been working on a
project very related to this topic. I tested using Cloud Hypervisor’s
Rust-based vhost-user virtiofs and block devices with QEMU. Bigger picture,
I wanted to explore the implications of using Rust for vhost-user devices
in QEMU.



I agree with the points from the original post, namely:

·      C programming bugs are responsible for a large number of CVEs, and
specifically CVEs coming from the implementations of virtual devices.

·      As a programming language, Rust has matured to a point where it is
worth considering it more seriously for production use. It has extensive
libraries and community support. Many big players in the industry are
already using Rust for production workloads.



Full Transparency: the Drawbacks:

It would be deceptive to only showcase Rust in an ideal light.

·      The benchmarks I ran show a noticeable performance hit from
switching to a RustVMM implementation of a virtiofsd device.

·      While Rust has matured greatly, it still is missing a bit. One
example of this that came up was that the rust compiler does not have
Control Flow Integrity (CFI) features. While these are not as important as
in “unsafe” languages such as C, the ability to express unsafe portions of
code does allow for some types of memory bugs – although to a much lesser
extent (an interesting case of this surfaced from Firecracker, and the
handling of mmio [1]). So further protections such as Control Flow
Integrity can still be desirable, even with rust code.

·      There have been years of optimization work put into the C
implementations of these devices, and it’s hard to evaluate how optimized
the relatively novel rust implementations are.

A piece of exciting news is that many of these drawbacks show a pathway for
future improvement. Improvements to rust infrastructure are very realistic.
Rust boils down to LLVM just like C, so porting over C’s CFI features
should be feasible. If more development resources are put into the RustVMM
project, there is no reason their implementations can’t be as optimized as
the C versions, and this could be greatly aided by expertise coming from
the QEMU communities familiarity with these topics.



I believe vhost-user devices are an excellent place to start since It
lowers the entry barrier for developing in Rust. The device only has to
interface with the C-based QEMU binary through a standardized protocol. It
removes many worries of moving entirely away from C, since adding a set of
Rust devices would simply be giving more options and room to explore.



I am putting together the scripts I used for all of the tests at this repo:

https://github.com/Alex-Carter01/Qemu-Rust-Testing

I am working to standardize everything to make it easier to replicate. I
would love any community involvement if people wanted to see how results
differ based on the hardware setup, build configuration of the devices etc.

The repo also has links to a recording of my original presentation and the
slides I was using if you would like to look at that format or see the
discussion which came out of it.

[-- Attachment #2: Type: text/html, Size: 6788 bytes --]

Re: Why QEMU should move from C to Rust (clickbait alert ;))

[Sergio Lopez]

[-- Attachment #1: Type: text/plain, Size: 4022 bytes --]

On Fri, Aug 21, 2020 at 04:18:03PM -0400, Alex Carter wrote:
> Hi everyone,
> 
> My name is Alex, I’m a student at the University of Michigan and I just
> completed an internship at IBM Research. There, I have been working on a
> project very related to this topic. I tested using Cloud Hypervisor’s
> Rust-based vhost-user virtiofs and block devices with QEMU. Bigger picture,
> I wanted to explore the implications of using Rust for vhost-user devices
> in QEMU.
> 
> 
> 
> I agree with the points from the original post, namely:
> 
> ·      C programming bugs are responsible for a large number of CVEs, and
> specifically CVEs coming from the implementations of virtual devices.
> 
> ·      As a programming language, Rust has matured to a point where it is
> worth considering it more seriously for production use. It has extensive
> libraries and community support. Many big players in the industry are
> already using Rust for production workloads.
> 
> 
> 
> Full Transparency: the Drawbacks:
> 
> It would be deceptive to only showcase Rust in an ideal light.
> 
> ·      The benchmarks I ran show a noticeable performance hit from
> switching to a RustVMM implementation of a virtiofsd device.

I think it'd be interesting to be able to repeat those tests in a
different environment. I ran multiple benchmarks in the past comparing
vhost-user-blk (Rust) vs. qemu/contrib/vhost-user-blk (C) and
vhost-user-fs (Rust) vs virtiofsd (C) and never found that performance
hit.

Much on contrary, I found Rust's zero-cost abstractions promise to
live up even with very idiomatic chunks of code (such as
vm-virtio::Queue).

> ·      While Rust has matured greatly, it still is missing a bit. One
> example of this that came up was that the rust compiler does not have
> Control Flow Integrity (CFI) features. While these are not as important as
> in “unsafe” languages such as C, the ability to express unsafe portions of
> code does allow for some types of memory bugs – although to a much lesser
> extent (an interesting case of this surfaced from Firecracker, and the
> handling of mmio [1]). So further protections such as Control Flow
> Integrity can still be desirable, even with rust code.
> 
> ·      There have been years of optimization work put into the C
> implementations of these devices, and it’s hard to evaluate how optimized
> the relatively novel rust implementations are.
> 
> A piece of exciting news is that many of these drawbacks show a pathway for
> future improvement. Improvements to rust infrastructure are very realistic.
> Rust boils down to LLVM just like C, so porting over C’s CFI features
> should be feasible. If more development resources are put into the RustVMM
> project, there is no reason their implementations can’t be as optimized as
> the C versions, and this could be greatly aided by expertise coming from
> the QEMU communities familiarity with these topics.
> 
> 
> 
> I believe vhost-user devices are an excellent place to start since It
> lowers the entry barrier for developing in Rust. The device only has to
> interface with the C-based QEMU binary through a standardized protocol. It
> removes many worries of moving entirely away from C, since adding a set of
> Rust devices would simply be giving more options and room to explore.
> 
> 
> 
> I am putting together the scripts I used for all of the tests at this repo:
> 
> https://github.com/Alex-Carter01/Qemu-Rust-Testing
> 
> I am working to standardize everything to make it easier to replicate. I
> would love any community involvement if people wanted to see how results
> differ based on the hardware setup, build configuration of the devices etc.

Sounds good. What kind of help would you need?

Thanks!
Sergio.

> The repo also has links to a recording of my original presentation and the
> slides I was using if you would like to look at that format or see the
> discussion which came out of it.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]