[Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Stefan Hajnoczi]

Hi folks,
I'd like to start fuzzing emulated devices in QEMU.  Here is an
internship project idea I'm proposing to do this.

Any thoughts?  Want to co-mentor this in Google Summer of Code or Outreachy?

Stefan

'''Summary:''' Integrate oss-fuzz into QEMU so that the virtio-blk
device can be fuzz tested.

oss-fuzz offers a fuzz testing service to open source projects.  This
means random inputs are continuously tested against the program in
order to find crashes and other bugs.  Fuzz testing complements
hand-written test suites by exploring the input space of a program and
therefore the code paths that may be taken.

The goal of this project is to integrate oss-fuzz into QEMU so that
the virtio-blk-pci device can be fuzzed at both the VIRTIO and PCI bus
level.  virtio-blk-pci is a PCI device, which means it is connected to
the virtual machine's PCI bus and has a certain set of registers that
can be programmed by the guest.  Furthermore, it is a VIRTIO device -
this is the specification the describes most of the functionality of
virtio-blk.  Bugs exist at both the PCI and VIRTIO levels, so it's
important to fuzz both of them.

Fuzzing emulated devices involves accessing their hardware registers
randomly to make the device respond.  QEMU has a device testing
interface called "qtest" that accepts read/write and other commands
over a socket and is ideal for writing device-level tests.  You may
find that oss-fuzz works better integrated directly into the QEMU
program instead of as a separate qtest program, so you can consider
adding a new command-line option to QEMU for running in oss-fuzz mode.

This project involves learning about VIRTIO and PCI devices, as well
as figuring out how to integrate oss-fuzz into QEMU so that it can
effective explore the code paths in virtio-blk device emulation code.
You will enjoy this project if you want to learn how device emulation
works and are interested in fuzzers.

'''Links:'''
* [https://github.com/google/oss-fuzz/blob/master/docs/ideal_integration.md
oss-fuzz integration overview]
* [https://github.com/google/fuzzer-test-suite/blob/master/tutorial/libFuzzerTutorial.md
libfuzzer tutorial]
* [http://docs.oasis-open.org/virtio/virtio/v1.0/cs04/virtio-v1.0-cs04.html
VIRTIO specification]
* [https://wiki.osdev.org/PCI PCI bus overview]

'''Details:'''
* Skill level: intermediate
* Language: C
* Mentor: Stefan Hajnoczi <stefanha@redhat.com>

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Dmitry Vyukov]

On Wed, Jan 9, 2019 at 6:34 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> Hi folks,
> I'd like to start fuzzing emulated devices in QEMU.  Here is an
> internship project idea I'm proposing to do this.
>
> Any thoughts?  Want to co-mentor this in Google Summer of Code or Outreachy?
>
> Stefan
>
> '''Summary:''' Integrate oss-fuzz into QEMU so that the virtio-blk
> device can be fuzz tested.

Sounds great!
+some people from OSS-Fuzz as FYI

Once a first fuzzer paves the road, it should be much simpler to add
other fuzzers as well. E.g. image parsing, or instruction emulation (I
know it's not security critical, but given a cost effective way to
find bugs it may be worth it). I understand that a GSOC project should
have a limit scope, but I think paving the road is important part of
this.

> oss-fuzz offers a fuzz testing service to open source projects.  This
> means random inputs are continuously tested against the program in
> order to find crashes and other bugs.  Fuzz testing complements
> hand-written test suites by exploring the input space of a program and
> therefore the code paths that may be taken.
>
> The goal of this project is to integrate oss-fuzz into QEMU so that
> the virtio-blk-pci device can be fuzzed at both the VIRTIO and PCI bus
> level.  virtio-blk-pci is a PCI device, which means it is connected to
> the virtual machine's PCI bus and has a certain set of registers that
> can be programmed by the guest.  Furthermore, it is a VIRTIO device -
> this is the specification the describes most of the functionality of
> virtio-blk.  Bugs exist at both the PCI and VIRTIO levels, so it's
> important to fuzz both of them.
>
> Fuzzing emulated devices involves accessing their hardware registers
> randomly to make the device respond.  QEMU has a device testing
> interface called "qtest" that accepts read/write and other commands
> over a socket and is ideal for writing device-level tests.  You may
> find that oss-fuzz works better integrated directly into the QEMU
> program instead of as a separate qtest program, so you can consider
> adding a new command-line option to QEMU for running in oss-fuzz mode.
>
> This project involves learning about VIRTIO and PCI devices, as well
> as figuring out how to integrate oss-fuzz into QEMU so that it can
> effective explore the code paths in virtio-blk device emulation code.
> You will enjoy this project if you want to learn how device emulation
> works and are interested in fuzzers.
>
> '''Links:'''
> * [https://github.com/google/oss-fuzz/blob/master/docs/ideal_integration.md
> oss-fuzz integration overview]
> * [https://github.com/google/fuzzer-test-suite/blob/master/tutorial/libFuzzerTutorial.md
> libfuzzer tutorial]
> * [http://docs.oasis-open.org/virtio/virtio/v1.0/cs04/virtio-v1.0-cs04.html
> VIRTIO specification]
> * [https://wiki.osdev.org/PCI PCI bus overview]
>
> '''Details:'''
> * Skill level: intermediate
> * Language: C
> * Mentor: Stefan Hajnoczi <stefanha@redhat.com>

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Bandan Das]

Hi Stefan,

Stefan Hajnoczi <stefanha@gmail.com> writes:

> Hi folks,
> I'd like to start fuzzing emulated devices in QEMU.  Here is an
> internship project idea I'm proposing to do this.
>
> Any thoughts?  Want to co-mentor this in Google Summer of Code or Outreachy?
>
> Stefan
>
> '''Summary:''' Integrate oss-fuzz into QEMU so that the virtio-blk
> device can be fuzz tested.
>
> oss-fuzz offers a fuzz testing service to open source projects.  This
> means random inputs are continuously tested against the program in
> order to find crashes and other bugs.  Fuzz testing complements
> hand-written test suites by exploring the input space of a program and
> therefore the code paths that may be taken.
>
> The goal of this project is to integrate oss-fuzz into QEMU so that
> the virtio-blk-pci device can be fuzzed at both the VIRTIO and PCI bus
> level.  virtio-blk-pci is a PCI device, which means it is connected to
> the virtual machine's PCI bus and has a certain set of registers that
> can be programmed by the guest.  Furthermore, it is a VIRTIO device -
> this is the specification the describes most of the functionality of
> virtio-blk.  Bugs exist at both the PCI and VIRTIO levels, so it's
> important to fuzz both of them.
>
> Fuzzing emulated devices involves accessing their hardware registers
> randomly to make the device respond.  QEMU has a device testing
> interface called "qtest" that accepts read/write and other commands
> over a socket and is ideal for writing device-level tests.  You may
> find that oss-fuzz works better integrated directly into the QEMU
> program instead of as a separate qtest program, so you can consider
> adding a new command-line option to QEMU for running in oss-fuzz mode.

This sounds very interesting and if successful can easily be a stepping stone
to other sections. I would be interested in co-mentoring specifically focusing
on the PCI code.

One of the things I remember getting into trouble with when I was trying to hack
on this, especially fuzzing Qemu as a whole is what would the run environment
be like ? Would Qemu attempt to run a regular guest in oss-fuzz mode or only
a certain part of Qemu (emulated devices for example) be somehow run without
interacting with other dependent components ?

Bandan

>
> This project involves learning about VIRTIO and PCI devices, as well
> as figuring out how to integrate oss-fuzz into QEMU so that it can
> effective explore the code paths in virtio-blk device emulation code.
> You will enjoy this project if you want to learn how device emulation
> works and are interested in fuzzers.
>
> '''Links:'''
> * [https://github.com/google/oss-fuzz/blob/master/docs/ideal_integration.md
> oss-fuzz integration overview]
> * [https://github.com/google/fuzzer-test-suite/blob/master/tutorial/libFuzzerTutorial.md
> libfuzzer tutorial]
> * [http://docs.oasis-open.org/virtio/virtio/v1.0/cs04/virtio-v1.0-cs04.html
> VIRTIO specification]
> * [https://wiki.osdev.org/PCI PCI bus overview]
>
> '''Details:'''
> * Skill level: intermediate
> * Language: C
> * Mentor: Stefan Hajnoczi <stefanha@redhat.com>

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Dmitry Vyukov]

On Thu, Jan 10, 2019 at 2:40 PM Bandan Das <bsd@redhat.com> wrote:
>
> Hi Stefan,
>
> Stefan Hajnoczi <stefanha@gmail.com> writes:
>
> > Hi folks,
> > I'd like to start fuzzing emulated devices in QEMU.  Here is an
> > internship project idea I'm proposing to do this.
> >
> > Any thoughts?  Want to co-mentor this in Google Summer of Code or Outreachy?
> >
> > Stefan
> >
> > '''Summary:''' Integrate oss-fuzz into QEMU so that the virtio-blk
> > device can be fuzz tested.
> >
> > oss-fuzz offers a fuzz testing service to open source projects.  This
> > means random inputs are continuously tested against the program in
> > order to find crashes and other bugs.  Fuzz testing complements
> > hand-written test suites by exploring the input space of a program and
> > therefore the code paths that may be taken.
> >
> > The goal of this project is to integrate oss-fuzz into QEMU so that
> > the virtio-blk-pci device can be fuzzed at both the VIRTIO and PCI bus
> > level.  virtio-blk-pci is a PCI device, which means it is connected to
> > the virtual machine's PCI bus and has a certain set of registers that
> > can be programmed by the guest.  Furthermore, it is a VIRTIO device -
> > this is the specification the describes most of the functionality of
> > virtio-blk.  Bugs exist at both the PCI and VIRTIO levels, so it's
> > important to fuzz both of them.
> >
> > Fuzzing emulated devices involves accessing their hardware registers
> > randomly to make the device respond.  QEMU has a device testing
> > interface called "qtest" that accepts read/write and other commands
> > over a socket and is ideal for writing device-level tests.  You may
> > find that oss-fuzz works better integrated directly into the QEMU
> > program instead of as a separate qtest program, so you can consider
> > adding a new command-line option to QEMU for running in oss-fuzz mode.
>
> This sounds very interesting and if successful can easily be a stepping stone
> to other sections. I would be interested in co-mentoring specifically focusing
> on the PCI code.
>
> One of the things I remember getting into trouble with when I was trying to hack
> on this, especially fuzzing Qemu as a whole is what would the run environment
> be like ? Would Qemu attempt to run a regular guest in oss-fuzz mode or only
> a certain part of Qemu (emulated devices for example) be somehow run without
> interacting with other dependent components ?

Ideally (from fuzzing perspective) we test just a single component and
everything else is stubbed out. But I don't know how feasible this is.
AFL can fork the whole process and then feed the input in some way.
This would allow to test the whole qemu without stubbing anything.
Should be possible with libFuzzer with some effort too.
I would say: start with whatever is simpler to do and then we will see
how good/bad it works.

One thing I forgot to mention (though, not the top priority) is
libprotobuf-mutator:
https://github.com/google/libprotobuf-mutator#libprotobuf-mutator
http://llvm.org/devmtg/2017-10/slides/Serebryany-Structure-aware%20fuzzing%20for%20Clang%20and%20LLVM%20with%20libprotobuf-mutator.pdf
https://chromium.googlesource.com/chromium/src/testing/libfuzzer/+/HEAD/libprotobuf-mutator.md

It allows fuzzer to work with structured inputs. Say, if you want as
input some sequence of actions (function calls) with some particular
arguments. This can be encoded as a protobuf message, and then fuzzer
function will interpret this protobuf message as actual actions. For
complex inputs this may be much more efficient than blob-mutations and
then some ad-hoc interpretation of that blob.




> > This project involves learning about VIRTIO and PCI devices, as well
> > as figuring out how to integrate oss-fuzz into QEMU so that it can
> > effective explore the code paths in virtio-blk device emulation code.
> > You will enjoy this project if you want to learn how device emulation
> > works and are interested in fuzzers.
> >
> > '''Links:'''
> > * [https://github.com/google/oss-fuzz/blob/master/docs/ideal_integration.md
> > oss-fuzz integration overview]
> > * [https://github.com/google/fuzzer-test-suite/blob/master/tutorial/libFuzzerTutorial.md
> > libfuzzer tutorial]
> > * [http://docs.oasis-open.org/virtio/virtio/v1.0/cs04/virtio-v1.0-cs04.html
> > VIRTIO specification]
> > * [https://wiki.osdev.org/PCI PCI bus overview]
> >
> > '''Details:'''
> > * Skill level: intermediate
> > * Language: C
> > * Mentor: Stefan Hajnoczi <stefanha@redhat.com>

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Max Moroz]

+Oliver and Jonathan

I'm a little confused. Do you want to fuzz QEMU or to fuzz something else
using QEMU? In case of the latter, there was some discussion on OSS-Fuzz
and (I think) even a build support was (sort of) added:
https://github.com/google/oss-fuzz/issues/1754

In any case, it would probably be "integrate QEMU into OSS-Fuzz" and not
vice versa, since OSS-Fuzz is an infrastructure and you can't integrate it
into QEMU.

On Thu, Jan 10, 2019 at 6:02 AM Dmitry Vyukov <dvyukov@google.com> wrote:

> On Thu, Jan 10, 2019 at 2:40 PM Bandan Das <bsd@redhat.com> wrote:
> >
> > Hi Stefan,
> >
> > Stefan Hajnoczi <stefanha@gmail.com> writes:
> >
> > > Hi folks,
> > > I'd like to start fuzzing emulated devices in QEMU.  Here is an
> > > internship project idea I'm proposing to do this.
> > >
> > > Any thoughts?  Want to co-mentor this in Google Summer of Code or
> Outreachy?
> > >
> > > Stefan
> > >
> > > '''Summary:''' Integrate oss-fuzz into QEMU so that the virtio-blk
> > > device can be fuzz tested.
> > >
> > > oss-fuzz offers a fuzz testing service to open source projects.  This
> > > means random inputs are continuously tested against the program in
> > > order to find crashes and other bugs.  Fuzz testing complements
> > > hand-written test suites by exploring the input space of a program and
> > > therefore the code paths that may be taken.
> > >
> > > The goal of this project is to integrate oss-fuzz into QEMU so that
> > > the virtio-blk-pci device can be fuzzed at both the VIRTIO and PCI bus
> > > level.  virtio-blk-pci is a PCI device, which means it is connected to
> > > the virtual machine's PCI bus and has a certain set of registers that
> > > can be programmed by the guest.  Furthermore, it is a VIRTIO device -
> > > this is the specification the describes most of the functionality of
> > > virtio-blk.  Bugs exist at both the PCI and VIRTIO levels, so it's
> > > important to fuzz both of them.
> > >
> > > Fuzzing emulated devices involves accessing their hardware registers
> > > randomly to make the device respond.  QEMU has a device testing
> > > interface called "qtest" that accepts read/write and other commands
> > > over a socket and is ideal for writing device-level tests.  You may
> > > find that oss-fuzz works better integrated directly into the QEMU
> > > program instead of as a separate qtest program, so you can consider
> > > adding a new command-line option to QEMU for running in oss-fuzz mode.
> >
> > This sounds very interesting and if successful can easily be a stepping
> stone
> > to other sections. I would be interested in co-mentoring specifically
> focusing
> > on the PCI code.
> >
> > One of the things I remember getting into trouble with when I was trying
> to hack
> > on this, especially fuzzing Qemu as a whole is what would the run
> environment
> > be like ? Would Qemu attempt to run a regular guest in oss-fuzz mode or
> only
> > a certain part of Qemu (emulated devices for example) be somehow run
> without
> > interacting with other dependent components ?
>
> Ideally (from fuzzing perspective) we test just a single component and
> everything else is stubbed out. But I don't know how feasible this is.
> AFL can fork the whole process and then feed the input in some way.
> This would allow to test the whole qemu without stubbing anything.
> Should be possible with libFuzzer with some effort too.
> I would say: start with whatever is simpler to do and then we will see
> how good/bad it works.
>
> One thing I forgot to mention (though, not the top priority) is
> libprotobuf-mutator:
> https://github.com/google/libprotobuf-mutator#libprotobuf-mutator
>
> http://llvm.org/devmtg/2017-10/slides/Serebryany-Structure-aware%20fuzzing%20for%20Clang%20and%20LLVM%20with%20libprotobuf-mutator.pdf
>
> https://chromium.googlesource.com/chromium/src/testing/libfuzzer/+/HEAD/libprotobuf-mutator.md
>
> It allows fuzzer to work with structured inputs. Say, if you want as
> input some sequence of actions (function calls) with some particular
> arguments. This can be encoded as a protobuf message, and then fuzzer
> function will interpret this protobuf message as actual actions. For
> complex inputs this may be much more efficient than blob-mutations and
> then some ad-hoc interpretation of that blob.
>
>
>
>
> > > This project involves learning about VIRTIO and PCI devices, as well
> > > as figuring out how to integrate oss-fuzz into QEMU so that it can
> > > effective explore the code paths in virtio-blk device emulation code.
> > > You will enjoy this project if you want to learn how device emulation
> > > works and are interested in fuzzers.
> > >
> > > '''Links:'''
> > > * [
> https://github.com/google/oss-fuzz/blob/master/docs/ideal_integration.md
> > > oss-fuzz integration overview]
> > > * [
> https://github.com/google/fuzzer-test-suite/blob/master/tutorial/libFuzzerTutorial.md
> > > libfuzzer tutorial]
> > > * [
> http://docs.oasis-open.org/virtio/virtio/v1.0/cs04/virtio-v1.0-cs04.html
> > > VIRTIO specification]
> > > * [https://wiki.osdev.org/PCI PCI bus overview]
> > >
> > > '''Details:'''
> > > * Skill level: intermediate
> > > * Language: C
> > > * Mentor: Stefan Hajnoczi <stefanha@redhat.com>
>

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Paolo Bonzini]

On 10/01/19 17:07, Max Moroz via Qemu-devel wrote:
> +Oliver and Jonathan
> 
> I'm a little confused. Do you want to fuzz QEMU or to fuzz something else
> using QEMU? In case of the latter, there was some discussion on OSS-Fuzz
> and (I think) even a build support was (sort of) added:
> https://github.com/google/oss-fuzz/issues/1754

We want to fuzz QEMU.

The input is a sequence of operations on a PCI device or on guest
memory, and you try and crash QEMU by passing it crazy operations.

The qtest mode that Stefan mentioned provides something like a "JTAG
over ASCII" where a testcase can inject elementary I/O operations into
QEMU.  However, that is certainly too much for AFL to fuzz effectively;
it would have to "learn" the qtest protocol, how to drive the PCI host
bridge, how to setup a PCI device, and only then it would be able to
find bugs.  So one idea would be to build another layer on top of qtest,
that accepts higher-level operations and builds the qtest ASCII input
from those.

However, compared to LLVM/Clang that were the motivation for
libprotobuf-mutator, in our case the underlying operands to the
operations are still binary.  So it may also be possible to use the
usual AFL binary mutator, and encode the operations in a way that has
very high entropy.  This way AFL can just throw crap at the qtest input
generator, and the resulting input qtest will always represent some kind
of interaction with a PCI device.

Thanks,

Paolo

> In any case, it would probably be "integrate QEMU into OSS-Fuzz" and not
> vice versa, since OSS-Fuzz is an infrastructure and you can't integrate it
> into QEMU.

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Stefan Hajnoczi]

On Thu, Jan 10, 2019 at 11:25 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 10/01/19 17:07, Max Moroz via Qemu-devel wrote:
> > +Oliver and Jonathan
> >
> > I'm a little confused. Do you want to fuzz QEMU or to fuzz something else
> > using QEMU? In case of the latter, there was some discussion on OSS-Fuzz
> > and (I think) even a build support was (sort of) added:
> > https://github.com/google/oss-fuzz/issues/1754
>
> We want to fuzz QEMU.
>
> The input is a sequence of operations on a PCI device or on guest
> memory, and you try and crash QEMU by passing it crazy operations.
>
> The qtest mode that Stefan mentioned provides something like a "JTAG
> over ASCII" where a testcase can inject elementary I/O operations into
> QEMU.  However, that is certainly too much for AFL to fuzz effectively;
> it would have to "learn" the qtest protocol, how to drive the PCI host
> bridge, how to setup a PCI device, and only then it would be able to
> find bugs.  So one idea would be to build another layer on top of qtest,
> that accepts higher-level operations and builds the qtest ASCII input
> from those.

Question for the oss-fuzz folks: QEMU's qtest protocol involves a
2-process architecture with a qtest process that connects to a QEMU
process.  We don't care about the code coverage/sanitizers/etc on the
test process, only the QEMU process.  The LLVMFuzzerTestOneInput()
entry point would nevertheless be in the qtest process.  Does this fit
into the oss-fuzz architecture?

Thanks,
Stefan

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Max Moroz]

Paolo, thanks for clarifying, that makes total sense! I'd suggest not
focusing on AFL, but go for a libFuzzer-based fuzz target (i.e.
write LLVMFuzzerTestOneInput function), in that case you'll get both
libFuzzer and AFL engines pluggable to that fuzz target. OSS-Fuzz runs both.

On Thu, Jan 10, 2019 at 10:49 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:

> On Thu, Jan 10, 2019 at 11:25 PM Paolo Bonzini <pbonzini@redhat.com>
> wrote:
> > On 10/01/19 17:07, Max Moroz via Qemu-devel wrote:
> > > +Oliver and Jonathan
> > >
> > > I'm a little confused. Do you want to fuzz QEMU or to fuzz something
> else
> > > using QEMU? In case of the latter, there was some discussion on
> OSS-Fuzz
> > > and (I think) even a build support was (sort of) added:
> > > https://github.com/google/oss-fuzz/issues/1754
> >
> > We want to fuzz QEMU.
> >
> > The input is a sequence of operations on a PCI device or on guest
> > memory, and you try and crash QEMU by passing it crazy operations.
> >
> > The qtest mode that Stefan mentioned provides something like a "JTAG
> > over ASCII" where a testcase can inject elementary I/O operations into
> > QEMU.  However, that is certainly too much for AFL to fuzz effectively;
> > it would have to "learn" the qtest protocol, how to drive the PCI host
> > bridge, how to setup a PCI device, and only then it would be able to
> > find bugs.  So one idea would be to build another layer on top of qtest,
> > that accepts higher-level operations and builds the qtest ASCII input
> > from those.
>
> Question for the oss-fuzz folks: QEMU's qtest protocol involves a
> 2-process architecture with a qtest process that connects to a QEMU
> process.  We don't care about the code coverage/sanitizers/etc on the
> test process, only the QEMU process.  The LLVMFuzzerTestOneInput()
> entry point would nevertheless be in the qtest process.  Does this fit
> into the oss-fuzz architecture?
>

We usually have a single fuzzing process, it starts with a fuzzing engine's
main function and is calling LLVMFuzzerTestOneInput with various inputs and
keep mutating them based on the coverage feedback. Running a second process
which you don't care too much about might be fine, but the fuzzing process
should be "replacing" or should I say "imitating" the process whose
coverage you're interested in. Does it make any sense?



>
> Thanks,
> Stefan
>

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Paolo Bonzini]

On 11/01/19 16:04, Max Moroz wrote:
> We usually have a single fuzzing process, it starts with a fuzzing
> engine's main function and is calling LLVMFuzzerTestOneInput with
> various inputs and keep mutating them based on the coverage feedback.
> Running a second process which you don't care too much about might be
> fine, but the fuzzing process should be "replacing" or should I say
> "imitating" the process whose coverage you're interested in.

What do you mean by replacing or imitating?

Avoiding fork would probably be hard.  I'm mostly afraid that some state
guest state is not resetted properly across runs, and this would result
in non-reproducible crashes.

It seems to me that the task can be approached with AFL and a test case
postprocessor to generate the qtest input; however, my knowledge of
libFuzzer is very very limited.

Paolo

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Max Moroz]

On Fri, Jan 11, 2019 at 7:34 AM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 11/01/19 16:04, Max Moroz wrote:
> > We usually have a single fuzzing process, it starts with a fuzzing
> > engine's main function and is calling LLVMFuzzerTestOneInput with
> > various inputs and keep mutating them based on the coverage feedback.
> > Running a second process which you don't care too much about might be
> > fine, but the fuzzing process should be "replacing" or should I say
> > "imitating" the process whose coverage you're interested in.
>
> What do you mean by replacing or imitating?
>

To give you an example, when we fuzz ffmpeg, we do not run ffmpeg's main
function. We write LLVMFuzzerTestOneInput that would do the necessary
initialization, reset the state, etc, and then would pass (data, size)
provided by a fuzzing engine to the API(s) we're trying to fuzz. So, in
your case, there should not be a regular QEMU process, and instead the fuzz
target (i.e. LLVMFuzzerTestOneInput) should be doing certain initialization
(which is usually done by the QEMU process) and then call the API you want
to fuzz.


>
> Avoiding fork would probably be hard.  I'm mostly afraid that some state
> guest state is not resetted properly across runs, and this would result
> in non-reproducible crashes.
>
> It seems to me that the task can be approached with AFL and a test case
> postprocessor to generate the qtest input; however, my knowledge of
> libFuzzer is very very limited.
>
> Paolo
>

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Paolo Bonzini]

On 11/01/19 16:41, Max Moroz wrote:
> 
> 
> On Fri, Jan 11, 2019 at 7:34 AM Paolo Bonzini <pbonzini@redhat.com
> <mailto:pbonzini@redhat.com>> wrote:
> 
>     On 11/01/19 16:04, Max Moroz wrote:
>     > We usually have a single fuzzing process, it starts with a fuzzing
>     > engine's main function and is calling LLVMFuzzerTestOneInput with
>     > various inputs and keep mutating them based on the coverage feedback.
>     > Running a second process which you don't care too much about might be
>     > fine, but the fuzzing process should be "replacing" or should I say
>     > "imitating" the process whose coverage you're interested in.
> 
>     What do you mean by replacing or imitating?
> 
> To give you an example, when we fuzz ffmpeg, we do not run ffmpeg's main
> function. We write LLVMFuzzerTestOneInput that would do the necessary
> initialization, reset the state, etc, and then would pass (data, size)
> provided by a fuzzing engine to the API(s) we're trying to fuzz. So, in
> your case, there should not be a regular QEMU process, and instead the
> fuzz target (i.e. LLVMFuzzerTestOneInput) should be doing certain
> initialization (which is usually done by the QEMU process) and then call
> the API you want to fuzz.

The main issue is that we are not really testing an API and QEMU has a
lot of global state.  Of course there are C functions to do the
elementary I/O operations, but there are two problems.  The first and
smaller is that one input would correspond to a sequence of invocations
of the functions, not just one function invocation; the larger, is that
the knowledge of the guest (memory map, placement of devices, etc.) is
not easy to consume from within QEMU.  We do have a library that
includes that knowledge, and it would be easy to use it from a mutator
or input postprocessor, but I'm afraid that it would be very very
intrusive to embed that library into QEMU, replacing all of main() etc.

The simplest way would be to test the input by fork()-ing QEMU, followed
by waitpid() in the parent and invoking the original main() in the
child.  But that wouldn't buy much and would probably even be slower
than an efficient deferred fork server.

Paolo

>  
> 
> 
>     Avoiding fork would probably be hard.  I'm mostly afraid that some state
>     guest state is not resetted properly across runs, and this would result
>     in non-reproducible crashes.
> 
>     It seems to me that the task can be approached with AFL and a test case
>     postprocessor to generate the qtest input; however, my knowledge of
>     libFuzzer is very very limited.
> 
>     Paolo
> 

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Jonathan Metzman]

Could you clarify what you think the relationship between the qtest
process, QEMU, and afl-fuzz will look like when fuzzing?

Is it something like this:
1. afl-fuzz mutates a buffer, starts a qtest process, and gives the qtest
process the mutated buffer.
2. The qtest process starts a QEMU process and interacts with QEMU process
based on the buffer AFL gave it (qtest).
3. goto 1

I don't think this works (under normal circumstances). AFL will think it is
fuzzing qtest and will not learn about coverage or crashes from qsym.
There probably are ways to get this working, but I just want to make sure I
understand.

Thanks,
Jonathan

On Fri, Jan 11, 2019 at 8:16 AM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 11/01/19 16:41, Max Moroz wrote:
> >
> >
> > On Fri, Jan 11, 2019 at 7:34 AM Paolo Bonzini <pbonzini@redhat.com
> > <mailto:pbonzini@redhat.com>> wrote:
> >
> >     On 11/01/19 16:04, Max Moroz wrote:
> >     > We usually have a single fuzzing process, it starts with a fuzzing
> >     > engine's main function and is calling LLVMFuzzerTestOneInput with
> >     > various inputs and keep mutating them based on the coverage
> feedback.
> >     > Running a second process which you don't care too much about might
> be
> >     > fine, but the fuzzing process should be "replacing" or should I say
> >     > "imitating" the process whose coverage you're interested in.
> >
> >     What do you mean by replacing or imitating?
> >
> > To give you an example, when we fuzz ffmpeg, we do not run ffmpeg's main
> > function. We write LLVMFuzzerTestOneInput that would do the necessary
> > initialization, reset the state, etc, and then would pass (data, size)
> > provided by a fuzzing engine to the API(s) we're trying to fuzz. So, in
> > your case, there should not be a regular QEMU process, and instead the
> > fuzz target (i.e. LLVMFuzzerTestOneInput) should be doing certain
> > initialization (which is usually done by the QEMU process) and then call
> > the API you want to fuzz.
>
> The main issue is that we are not really testing an API and QEMU has a
> lot of global state.

Of course there are C functions to do the
> elementary I/O operations, but there are two problems.  The first and
> smaller is that one input would correspond to a sequence of invocations
> of the functions, not just one function invocation; the larger, is that
> the knowledge of the guest (memory map, placement of devices, etc.) is
> not easy to consume from within QEMU.  We do have a library that
> includes that knowledge, and it would be easy to use it from a mutator
> or input postprocessor, but I'm afraid that it would be very very
> intrusive to embed that library into QEMU, replacing all of main() etc.
>
> The simplest way would be to test the input by fork()-ing QEMU, followed
> by waitpid() in the parent and invoking the original main() in the
> child.  But that wouldn't buy much and would probably even be slower
> than an efficient deferred fork server.
>
> Paolo
>
> >
> >
> >
> >     Avoiding fork would probably be hard.  I'm mostly afraid that some
> state
> >     guest state is not resetted properly across runs, and this would
> result
> >     in non-reproducible crashes.
> >
> >     It seems to me that the task can be approached with AFL and a test
> case
> >     postprocessor to generate the qtest input; however, my knowledge of
> >     libFuzzer is very very limited.
> >
> >     Paolo
> >
>
>

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Paolo Bonzini]

On 11/01/19 20:09, Jonathan Metzman wrote:
> Could you clarify what you think the relationship between the qtest
> process, QEMU, and afl-fuzz will look like when fuzzing?
> 
> Is it something like this:
> 1. afl-fuzz mutates a buffer, starts a qtest process, and gives the
> qtest process the mutated buffer.
> 2. The qtest process starts a QEMU process and interacts with QEMU
> process based on the buffer AFL gave it (qtest).
> 3. goto 1
> 
> I don't think this works (under normal circumstances). AFL will think it
> is fuzzing qtest and will not learn about coverage or crashes from qsym.
> There probably are ways to get this working, but I just want to make
> sure I understand.

It should be possible to turn the qtest process into a test
postprocessor, and remove the second process.  It's much harder to
remove the QEMU process as well and turn it into a TestOneInput function.

Paolo

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Jonathan Metzman]

>It should be possible to turn the qtest process into a test
postprocessor,

OSS-Fuzz doesn't support AFL's preprocessors, but adding support shouldn't
be hard.

>It's much harder to
remove the QEMU process as well and turn it into a TestOneInput function.

Got it. I am not familiar with postprocessors but I guess this should work
as long as the QEMU process can interpret the buffer output by the
postprocessor.

There may be some other complications with having an AFL-only fuzzer on
OSS-Fuzz.
The main one is no pruning. On OSS-Fuzz pruning is only done by libFuzzer,
since AFL and libFuzzer share a corpus.
I'm not sure how we would handle this.

On Fri, Jan 11, 2019 at 12:27 PM Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 11/01/19 20:09, Jonathan Metzman wrote:
> > Could you clarify what you think the relationship between the qtest
> > process, QEMU, and afl-fuzz will look like when fuzzing?
> >
> > Is it something like this:
> > 1. afl-fuzz mutates a buffer, starts a qtest process, and gives the
> > qtest process the mutated buffer.
> > 2. The qtest process starts a QEMU process and interacts with QEMU
> > process based on the buffer AFL gave it (qtest).
> > 3. goto 1
> >
> > I don't think this works (under normal circumstances). AFL will think it
> > is fuzzing qtest and will not learn about coverage or crashes from qsym.
> > There probably are ways to get this working, but I just want to make
> > sure I understand.
>
> It should be possible to turn the qtest process into a test
> postprocessor, and remove the second process.  It's much harder to
> remove the QEMU process as well and turn it into a TestOneInput function.
>
> Paolo
>

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1963 bytes --]

On Fri, Jan 11, 2019 at 05:16:40PM +0100, Paolo Bonzini wrote:
> On 11/01/19 16:41, Max Moroz wrote:
> > On Fri, Jan 11, 2019 at 7:34 AM Paolo Bonzini <pbonzini@redhat.com
> > <mailto:pbonzini@redhat.com>> wrote:
> > 
> >     On 11/01/19 16:04, Max Moroz wrote:
> >     > We usually have a single fuzzing process, it starts with a fuzzing
> >     > engine's main function and is calling LLVMFuzzerTestOneInput with
> >     > various inputs and keep mutating them based on the coverage feedback.
> >     > Running a second process which you don't care too much about might be
> >     > fine, but the fuzzing process should be "replacing" or should I say
> >     > "imitating" the process whose coverage you're interested in.
> > 
> >     What do you mean by replacing or imitating?
> > 
> > To give you an example, when we fuzz ffmpeg, we do not run ffmpeg's main
> > function. We write LLVMFuzzerTestOneInput that would do the necessary
> > initialization, reset the state, etc, and then would pass (data, size)
> > provided by a fuzzing engine to the API(s) we're trying to fuzz. So, in
> > your case, there should not be a regular QEMU process, and instead the
> > fuzz target (i.e. LLVMFuzzerTestOneInput) should be doing certain
> > initialization (which is usually done by the QEMU process) and then call
> > the API you want to fuzz.
> 
> The main issue is that we are not really testing an API and QEMU has a
> lot of global state.

With regards to the GSoC/Outreachy project, I think the mentors (me?)
need to figure this out beforehand by experimentation.  The QEMU folks
don't know the details of oss-fuzz and vice versa.  But with a weekend
or two's worth of playing around we could figure out a reasonable way of
integrating qtest/oss-fuzz.

Then the intern has a clear direction to follow this summer and won't be
demotivated by failed attempts at working with two codebases they are
unfamiliar with :).

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Internship idea: virtio-blk oss-fuzz support

[Bandan Das]

[Ccing Nitesh]
Stefan Hajnoczi <stefanha@gmail.com> writes:

> On Fri, Jan 11, 2019 at 05:16:40PM +0100, Paolo Bonzini wrote:
>> On 11/01/19 16:41, Max Moroz wrote:
>> > On Fri, Jan 11, 2019 at 7:34 AM Paolo Bonzini <pbonzini@redhat.com
>> > <mailto:pbonzini@redhat.com>> wrote:
>> > 
>> >     On 11/01/19 16:04, Max Moroz wrote:
>> >     > We usually have a single fuzzing process, it starts with a fuzzing
>> >     > engine's main function and is calling LLVMFuzzerTestOneInput with
>> >     > various inputs and keep mutating them based on the coverage feedback.
>> >     > Running a second process which you don't care too much about might be
>> >     > fine, but the fuzzing process should be "replacing" or should I say
>> >     > "imitating" the process whose coverage you're interested in.
>> > 
>> >     What do you mean by replacing or imitating?
>> > 
>> > To give you an example, when we fuzz ffmpeg, we do not run ffmpeg's main
>> > function. We write LLVMFuzzerTestOneInput that would do the necessary
>> > initialization, reset the state, etc, and then would pass (data, size)
>> > provided by a fuzzing engine to the API(s) we're trying to fuzz. So, in
>> > your case, there should not be a regular QEMU process, and instead the
>> > fuzz target (i.e. LLVMFuzzerTestOneInput) should be doing certain
>> > initialization (which is usually done by the QEMU process) and then call
>> > the API you want to fuzz.
>> 
>> The main issue is that we are not really testing an API and QEMU has a
>> lot of global state.
>
> With regards to the GSoC/Outreachy project, I think the mentors (me?)
> need to figure this out beforehand by experimentation.  The QEMU folks
> don't know the details of oss-fuzz and vice versa.  But with a weekend
> or two's worth of playing around we could figure out a reasonable way of
> integrating qtest/oss-fuzz.
>

If you recall, Nitesh and myself did experiment with the plumbing although
our entry point in qtest for calling the fuzzing function was simpler. Figuring out
the right entry point with a subset of absolutely necessary initialization
is what's next.

Bandan

> Then the intern has a clear direction to follow this summer and won't be
> demotivated by failed attempts at working with two codebases they are
> unfamiliar with :).
>
> Stefan