[RFC] virtio-villain: Guest fault injection for VMM robustness

[RFC] virtio-villain: Guest fault injection for VMM robustness

[Anatol Belski]

Hi,

I would like to share virtio-villain, a guest side fault injection
harness for virtio device models. The repository is at
https://github.com/weltling/virtio-villain

Comments on the approach, the coverage so far, and test cases worth
adding are all welcome. While the tool targets multiple hypervisors,
QEMU support is active and has already surfaced a few initial bugs.

Threat model. The harness operates under the hostile guest model. A
malicious or buggy guest driver violates the virtio protocol on
purpose. It targets the modern 1.2+ interface only. The test binary
runs as init inside a minimal initramfs, bypasses the kernel virtio
drivers with initcall_blacklist, walks the PCI capability list
itself, programs the queues by hand, and asserts on the exact spec
mandated values. Each test violates one driver MUST rule and checks
that the host VMM process handles it gracefully without crashing,
wedging, or corrupting state.

Scope and status. The primary target so far is Cloud Hypervisor,
which accounts for most of the findings to date. Testing focuses on
the amd64 and aarch64 architectures. QEMU is supported as a PCI host
and as the microvm MMIO host. Differential testing across VMMs,
running the same suite and comparing outcomes, is the roadmap. This
is a spec violation fault injection tool that surfaces unvalidated
input handling rather than an exploit framework, so no CVEs have
been issued.

What is already covered. There are two transports, modern virtio PCI
and virtio MMIO via the QEMU microvm machine type. A dozen device
categories are present. They include net, block with zoned support,
console, entropy, balloon, vsock, virtio-fs, IOMMU, virtio-mem,
virtio-pmem, RTC, and watchdog, plus the admin queue and the split
and packed virtqueue layers. Test cases derive from the virtio spec
normative text, each tied to a specific section.

Beyond the static suite there are two extra layers.

  - Host coordinated sidecars. A test can ship a small Python file
    that drives the host, for example pause, snapshot, hot plug, or
    config change, while the guest makes a precise assertion the
    kernel driver cannot make. As an example, snapshot the VM while
    the guest sits between two block reads, then assert the used
    ring drains deterministically on resume.

  - A coverage guided mutation fuzzer. It encodes a descriptor chain
    into a 4096 byte blob, patches it into a fuzz guest, boots the
    VMM, and classifies the result. With a coverage instrumented
    build it feeds new edges back into the corpus. Crashes are
    grouped by error class for triage.

Impact. Many findings have led to substantial improvements in
virtio spec compliance and stability in Cloud Hypervisor, from
input validation gaps to a config read that could take down a VM
on aarch64. In QEMU the harness has surfaced a few guest triggered
hangs and a double completion. The repository README lists the
details with PR links and the test ID for each.

Thanks for reading.

Anatol Belski

Re: [RFC] virtio-villain: Guest fault injection for VMM robustness

[Daniel P. Berrangé]

On Sat, Jul 04, 2026 at 03:37:21PM +0200, Anatol Belski wrote:
> Hi,
> 
> I would like to share virtio-villain, a guest side fault injection
> harness for virtio device models. The repository is at
> https://github.com/weltling/virtio-villain
> 
> Comments on the approach, the coverage so far, and test cases worth
> adding are all welcome. While the tool targets multiple hypervisors,
> QEMU support is active and has already surfaced a few initial bugs.

snip

> Impact. Many findings have led to substantial improvements in
> virtio spec compliance and stability in Cloud Hypervisor, from
> input validation gaps to a config read that could take down a VM
> on aarch64. In QEMU the harness has surfaced a few guest triggered
> hangs and a double completion. The repository README lists the
> details with PR links and the test ID for each.

The README only lists  cloud hypervisor bugs.  Did you file any for
QEMU yet ?


If this is finding bugs in QEMU this test harness sounds like the kind
of thing we ought to have integrated in QEMU's meson test suite such
that it runs in CI to prevent regressions.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [RFC] virtio-villain: Guest fault injection for VMM robustness

[Anatol Belski]

Hi Daniel,

On Mon, 2026-07-06 at 09:29 +0100, Daniel P. Berrangé wrote:
> On Sat, Jul 04, 2026 at 03:37:21PM +0200, Anatol Belski wrote:
> 
> 
> The README only lists  cloud hypervisor bugs.  Did you file any for
> QEMU yet ?

Thanks for checking. Yeah, the README only has the Cloud Hypervisor
fixes so far, those being the ones already merged. The QEMU ones are
not filed yet, they are still just reproducer tests in the harness.

There are several findings, though. As one example, a block read whose
data descriptor points at the device own MMIO BAR deadlocks the VM
under the BQL, so a single descriptor freezes the whole thing.
https://github.com/weltling/virtio-villain/blob/main/tests/vring/t0063_desc_addr_pci_bar.c

I have been reaching out to a few QEMU maintainers individually first,
to confirm these are ordinary robustness bugs and not security
sensitive before anything goes public. I am being careful because
automated and AI generated reports are flooding projects right now,
and I did not want to add noise or mishandle something sensitive.

> 
> If this is finding bugs in QEMU this test harness sounds like the
> kind
> of thing we ought to have integrated in QEMU's meson test suite such
> that it runs in CI to prevent regressions.
> 

Makes sense. I have been exercising QEMU regularly anyway, since it is
the de facto standard to compare against, so wiring the harness into
the QEMU test setup to catch regressions is a natural fit. One note on
shape. qtest drives a device from outside the guest, poking its
registers over a socket with no guest code running. virtio-villain
works the other way around, it boots a tiny init into an initramfs,
with no full OS, and that guest drives the device as real guest code.
So it stays lightweight, just a booted guest rather than an external
driver. I think that is worth exploring. The harness is public, so it
is there to build on, and I am glad to help work out how it could fit.

Thanks

Anatol

Re: [RFC] virtio-villain: Guest fault injection for VMM robustness

[Daniel P. Berrangé]

On Mon, Jul 06, 2026 at 09:22:12PM +0200, Anatol Belski wrote:
> Hi Daniel,
> 
> On Mon, 2026-07-06 at 09:29 +0100, Daniel P. Berrangé wrote:
> > On Sat, Jul 04, 2026 at 03:37:21PM +0200, Anatol Belski wrote:
> > 
> > 
> > The README only lists  cloud hypervisor bugs.  Did you file any for
> > QEMU yet ?

snip

> I have been reaching out to a few QEMU maintainers individually first,
> to confirm these are ordinary robustness bugs and not security
> sensitive before anything goes public. I am being careful because
> automated and AI generated reports are flooding projects right now,
> and I did not want to add noise or mishandle something sensitive.

We appreciate the intent not to flood projects ! Feel free to
drip-feed issues to us, a handful at a time though over days
or weeks if there will be alot to process.

We recently switched our security disclosure process over to using
our regular GitLab issue tracker, so would prefer any bugs to be
filed there, rather than emailing people directly:

  https://www.qemu.org/contribute/security-process/

This gives us resilience if individual manitainers are offline or
over-burdened by other work.

If it has security implications just select the "confidential"
tick box when filing. We can easily make it public if we decide
it is not a security issue during triage.

> > If this is finding bugs in QEMU this test harness sounds like the
> > kind
> > of thing we ought to have integrated in QEMU's meson test suite such
> > that it runs in CI to prevent regressions.
> > 
> 
> Makes sense. I have been exercising QEMU regularly anyway, since it is
> the de facto standard to compare against, so wiring the harness into
> the QEMU test setup to catch regressions is a natural fit. One note on
> shape. qtest drives a device from outside the guest, poking its
> registers over a socket with no guest code running. virtio-villain
> works the other way around, it boots a tiny init into an initramfs,
> with no full OS, and that guest drives the device as real guest code.
> So it stays lightweight, just a booted guest rather than an external
> driver. I think that is worth exploring. The harness is public, so it
> is there to build on, and I am glad to help work out how it could fit.

We also have tests/functional/ in QEMU where we boot real guest OS
disk images and/or kernel/initrd pairs. The minimal initramfs
approach would probably fit in nicely with that, and indeed I have
long wanted us to replace some of our full-fat disk images with
minimal self contained initramfs images to speed up smoke testing.

With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [RFC] virtio-villain: Guest fault injection for VMM robustness

[Anatol Belski]

On Tue, 2026-07-07 at 11:39 +0100, Daniel P. Berrangé wrote:

> We appreciate the intent not to flood projects ! Feel free to
> drip-feed issues to us, a handful at a time though over days
> or weeks if there will be alot to process.
> 
> We recently switched our security disclosure process over to using
> our regular GitLab issue tracker, so would prefer any bugs to be
> filed there, rather than emailing people directly:
> 
>   https://www.qemu.org/contribute/security-process/

Perfect, that makes it easy. I will file on the GitLab tracker and
send them in small batches over the next couple of weeks rather than
all at once. For the ones with a host side impact, a guest triggered
hang or use after free, I will tick the confidential box and let
triage decide whether it stays that way.


> We also have tests/functional/ in QEMU where we boot real guest OS
> disk images and/or kernel/initrd pairs. The minimal initramfs
> approach would probably fit in nicely with that, and indeed I have
> long wanted us to replace some of our full-fat disk images with
> minimal self contained initramfs images to speed up smoke testing.

That sounds like a clean fit. The harness already boots a kernel plus
a tiny initramfs with a single static init and no full OS, so it
should slot into tests/functional/ with little friction. Each test
finishes in under a second when passing, since there is no full VM
boot involved, which makes it suitable for smoke testing too. Would
it be useful to have these wired in there directly, or is there a
different shape that would work better for you?

Thanks

Anatol

Re: [RFC] virtio-villain: Guest fault injection for VMM robustness

[Daniel P. Berrangé]

On Tue, Jul 07, 2026 at 11:09:12PM +0200, Anatol Belski wrote:
> On Tue, 2026-07-07 at 11:39 +0100, Daniel P. Berrangé wrote:
> 
> > We appreciate the intent not to flood projects ! Feel free to
> > drip-feed issues to us, a handful at a time though over days
> > or weeks if there will be alot to process.
> > 
> > We recently switched our security disclosure process over to using
> > our regular GitLab issue tracker, so would prefer any bugs to be
> > filed there, rather than emailing people directly:
> > 
> >   https://www.qemu.org/contribute/security-process/
> 
> Perfect, that makes it easy. I will file on the GitLab tracker and
> send them in small batches over the next couple of weeks rather than
> all at once. For the ones with a host side impact, a guest triggered
> hang or use after free, I will tick the confidential box and let
> triage decide whether it stays that way.
> 
> 
> > We also have tests/functional/ in QEMU where we boot real guest OS
> > disk images and/or kernel/initrd pairs. The minimal initramfs
> > approach would probably fit in nicely with that, and indeed I have
> > long wanted us to replace some of our full-fat disk images with
> > minimal self contained initramfs images to speed up smoke testing.
> 
> That sounds like a clean fit. The harness already boots a kernel plus
> a tiny initramfs with a single static init and no full OS, so it
> should slot into tests/functional/ with little friction. Each test
> finishes in under a second when passing, since there is no full VM
> boot involved, which makes it suitable for smoke testing too. Would
> it be useful to have these wired in there directly, or is there a
> different shape that would work better for you?

Yes, I think it'd be interesting to try to fit them directly into a
functional test script. I guess having the virtio-villain repo as a
git submodule would probably be the way to go to access the individual
tests to build the initramfs.


With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [RFC] virtio-villain: Guest fault injection for VMM robustness

[Anatol Belski]

On Wed, 2026-07-08 at 09:09 +0100, Daniel P. Berrangé wrote:


> Yes, I think it'd be interesting to try to fit them directly into a
> functional test script. I guess having the virtio-villain repo as a
> git submodule would probably be the way to go to access the
> individual
> tests to build the initramfs.

A submodule would work. The build is a plain musl-gcc static link
plus a cpio wrap, so it should integrate without pulling in extra
dependencies.

One question on the kernel side. The harness currently uses the
Cloud Hypervisor prebuilt kernel

https://github.com/cloud-hypervisor/linux/releases/tag/ch-release-v6.16.9-20260508

which seems fully sufficient for the goals here. Is there a QEMU
preferred prebuilt kernel for tests/functional/ that I should use
instead?


Regards

Anatol

Re: [RFC] virtio-villain: Guest fault injection for VMM robustness

[Daniel P. Berrangé]

On Wed, Jul 08, 2026 at 08:25:19PM +0200, Anatol Belski wrote:
> On Wed, 2026-07-08 at 09:09 +0100, Daniel P. Berrangé wrote:
> 
> 
> > Yes, I think it'd be interesting to try to fit them directly into a
> > functional test script. I guess having the virtio-villain repo as a
> > git submodule would probably be the way to go to access the
> > individual
> > tests to build the initramfs.
> 
> A submodule would work. The build is a plain musl-gcc static link
> plus a cpio wrap, so it should integrate without pulling in extra
> dependencies.
> 
> One question on the kernel side. The harness currently uses the
> Cloud Hypervisor prebuilt kernel
> 
> https://github.com/cloud-hypervisor/linux/releases/tag/ch-release-v6.16.9-20260508
> 
> which seems fully sufficient for the goals here. Is there a QEMU
> preferred prebuilt kernel for tests/functional/ that I should use
> instead?

We don't have any stock kernel. Tests refer to any kenrel/initrd
image that they require and download it via the "Asset" class
functions which have caching built-in.


With regards,
Daniel
-- 
|: https://berrange.com       ~~        https://hachyderm.io/@berrange :|
|: https://libvirt.org          ~~          https://entangle-photo.org :|
|: https://pixelfed.art/berrange   ~~    https://fstop138.berrange.com :|

Re: [RFC] virtio-villain: Guest fault injection for VMM robustness

[Anatol Belski]

On Thu, 2026-07-09 at 08:17 +0100, Daniel P. Berrangé wrote:


> We don't have any stock kernel. Tests refer to any kenrel/initrd
> image that they require and download it via the "Asset" class
> functions which have caching built-in.

Thanks. With this I have all the info to try, I think. The CH kernel
will do fine then, I will just point the asset reference at that
release.

Meanwhile, I have filed the first finding on the tracker
https://gitlab.com/qemu-project/qemu/-/work_items/3968

Will follow up with the rest on a slower pace, as discussed.

Regards

Anatol

Re: [RFC] virtio-villain: Guest fault injection for VMM robustness

[Matias Ezequiel Vara Larsen]

On Sat, Jul 04, 2026 at 03:37:21PM +0200, Anatol Belski wrote:
> Hi,
> 
> I would like to share virtio-villain, a guest side fault injection
> harness for virtio device models. The repository is at
> https://github.com/weltling/virtio-villain
> 
> Comments on the approach, the coverage so far, and test cases worth
> adding are all welcome. While the tool targets multiple hypervisors,
> QEMU support is active and has already surfaced a few initial bugs.
> 
> Threat model. The harness operates under the hostile guest model. A
> malicious or buggy guest driver violates the virtio protocol on
> purpose. It targets the modern 1.2+ interface only. The test binary
> runs as init inside a minimal initramfs, bypasses the kernel virtio
> drivers with initcall_blacklist, walks the PCI capability list
> itself, programs the queues by hand, and asserts on the exact spec
> mandated values. Each test violates one driver MUST rule and checks
> that the host VMM process handles it gracefully without crashing,
> wedging, or corrupting state.
> 
> Scope and status. The primary target so far is Cloud Hypervisor,
> which accounts for most of the findings to date. Testing focuses on
> the amd64 and aarch64 architectures. QEMU is supported as a PCI host
> and as the microvm MMIO host. Differential testing across VMMs,
> running the same suite and comparing outcomes, is the roadmap. This
> is a spec violation fault injection tool that surfaces unvalidated
> input handling rather than an exploit framework, so no CVEs have
> been issued.
> 
Thanks for sharing it. I am also interesting on finding violations of
the virtio specification. I presented something two years ago in LPC
(see https://lpc.events/event/18/contributions/1897/). As a result of
that work, we worked adding kani proofs in rust-vmm devices. The
validation of the spec is coded in RUST and by using kani we verify that
the implemented devices follows the specification. For some
requirements, we were able to prove that the device is conform. The
proofs are written together with the implementation of the device like
tests.  

> What is already covered. There are two transports, modern virtio PCI
> and virtio MMIO via the QEMU microvm machine type. A dozen device
> categories are present. They include net, block with zoned support,
> console, entropy, balloon, vsock, virtio-fs, IOMMU, virtio-mem,
> virtio-pmem, RTC, and watchdog, plus the admin queue and the split
> and packed virtqueue layers. Test cases derive from the virtio spec
> normative text, each tied to a specific section.
> 
> Beyond the static suite there are two extra layers.
> 
>   - Host coordinated sidecars. A test can ship a small Python file
>     that drives the host, for example pause, snapshot, hot plug, or
>     config change, while the guest makes a precise assertion the
>     kernel driver cannot make. As an example, snapshot the VM while
>     the guest sits between two block reads, then assert the used
>     ring drains deterministically on resume.
> 
>   - A coverage guided mutation fuzzer. It encodes a descriptor chain
>     into a 4096 byte blob, patches it into a fuzz guest, boots the
>     VMM, and classifies the result. With a coverage instrumented
>     build it feeds new edges back into the corpus. Crashes are
>     grouped by error class for triage.
> 
> Impact. Many findings have led to substantial improvements in
> virtio spec compliance and stability in Cloud Hypervisor, from

Feel free to file patches to the virtio specification in cases you think
the spec is too permissive. 

> input validation gaps to a config read that could take down a VM
> on aarch64. In QEMU the harness has surfaced a few guest triggered

At some point, I wanted to add some checks in QEMU to inform user when
guest violates the spec. That would help to catch some issues with less
testing. For example, two years ago the virtio-sound driver was silently
violating the spec by writing the content of the buffers that were
already in the available ring. 

> hangs and a double completion. The repository README lists the
> details with PR links and the test ID for each.
> 
> Thanks for reading.
> 
> Anatol Belski
> 

Re: [RFC] virtio-villain: Guest fault injection for VMM robustness

[Anatol Belski]

Hi Matias,

On Fri, 2026-07-10 at 14:52 +0200, Matias Ezequiel Vara Larsen wrote:
> 


> Thanks for sharing it. I am also interesting on finding violations of
> the virtio specification. I presented something two years ago in LPC
> (see https://lpc.events/event/18/contributions/1897/). As a result of
> that work, we worked adding kani proofs in rust-vmm devices. The
> validation of the spec is coded in RUST and by using kani we verify
> that
> the implemented devices follows the specification. For some
> requirements, we were able to prove that the device is conform. The
> proofs are written together with the implementation of the device
> like
> tests.  

Thanks for the pointer. The virtio-sound case is a good example of why
this matters. The driver was exposing buffers back into the avail ring
while still owned by the device, and nothing caught it until it was
traced by hand.

The work you describe verifies that the rust-vmm device implementations
follow the spec both through formal proofs and runtime observation.
virtio-villain comes at it from the other end by acting as a
deliberately broken guest driver against any VMM and checking whether
the device handles the violation gracefully. Different sides of the
same medal.

> 
> Feel free to file patches to the virtio specification in cases you
> think
> the spec is too permissive. 

That would be worth doing. Several of the divergences I see between
VMMs come down to the spec being silent on what the device should do,
so tightening the normative text would help everyone.

One example is that the spec defines the descriptor addr and len fields
in virtio 1.4 section 2.7.5 but never states what the device must do
when addr plus len falls outside guest physical memory. Every VMM needs
to handle this, and Cloud Hypervisor already fixed it after the harness
surfaced the issue. A normative statement there would save each
implementation from rediscovering the same gap on its own.

> 
> At some point, I wanted to add some checks in QEMU to inform user
> when
> guest violates the spec. That would help to catch some issues with
> less
> testing. For example, two years ago the virtio-sound driver was
> silently
> violating the spec by writing the content of the buffers that were
> already in the available ring. 

That lines up with the discussion in your talk about placing the
observer on the device side. virtio-villain already classifies each
case as pass, reject, or wedge, so the cases where QEMU silently
tolerates a violation instead of flagging it are straightforward to
spot.

Regards

Anatol
>