Extending the 0-day system with syzkaller?

Extending the 0-day system with syzkaller?

[David Drysdale]

[-- Attachment #1: Type: text/plain, Size: 739 bytes --]

Hi Fengguang / LKP-folk,

Quick question -- how easy is it to add extra builds/tests/checks to
your marvellous 0-day kbuild system?

The reason I ask is that I've recently been exploring syzkaller [1],
which is a system call fuzzer written by some of my colleagues here at
Google (cc'ed).  Although it's fairly new, it has uncovered a bunch of
kernel bugs already [2] so I wondered if it might be a good candidate
for inclusion in the 0-day checks at some point.

(As an aside, I'm in the process of writing an article about syzkaller
for LWN, which might also expose it to more folk.)

What do you think?

Thanks,
David

[1] https://github.com/google/syzkaller
[2] https://github.com/google/syzkaller/wiki/Found-Bugs

Re: Extending the 0-day system with syzkaller?

[Kostya Serebryany]

[-- Attachment #1: Type: text/plain, Size: 1346 bytes --]

CC-ing more people after today's conversation at the Intel Tech Days.

We'd like to add kasan and syzkaller [1,2,3,4] to the 0-day kbuild system.
We believe this has a large potential to find old bugs and prevent
regressions in the Kernel.
How do we achieve this?

Thanks,

--kcc

[1] https://github.com/google/syzkaller
[2] https://github.com/google/syzkaller/wiki/Found-Bugs
[3] https://lwn.net/Articles/677764/
[4] https://www.kernel.org/doc/Documentation/kasan.txt


On Tue, Dec 15, 2015 at 3:49 AM, David Drysdale <drysdale@google.com> wrote:

> Hi Fengguang / LKP-folk,
>
> Quick question -- how easy is it to add extra builds/tests/checks to
> your marvellous 0-day kbuild system?
>
> The reason I ask is that I've recently been exploring syzkaller [1],
> which is a system call fuzzer written by some of my colleagues here at
> Google (cc'ed).  Although it's fairly new, it has uncovered a bunch of
> kernel bugs already [2] so I wondered if it might be a good candidate
> for inclusion in the 0-day checks at some point.
>
> (As an aside, I'm in the process of writing an article about syzkaller
> for LWN, which might also expose it to more folk.)
>
> What do you think?
>
> Thanks,
> David
>
> [1] https://github.com/google/syzkaller
> [2] https://github.com/google/syzkaller/wiki/Found-Bugs
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 2784 bytes --]

Re: Extending the 0-day system with syzkaller?

[Hart, Darren]

[-- Attachment #1: Type: text/plain, Size: 2307 bytes --]

Hi Fengguang,

I met with Kostya at Intel Tech Days and he had some compelling arguments for including some of these tests.

Dave H: I took a quick look at Kasan, which appears to require an existing config option (CONFIG_KASAN) for dynamic memory access checking using shadow memory. Is this something you would like to see added to 0-day? Do we have anything today which provides comparable coverage?

Combining Kasan and other existing kernel integrity checking, the syzkaller fuzz tester is showing promising results and the reports come in the form of kernel oops and similar things which we already check for in 0-day.

For the others on Cc, do you have additional context for or against including syzkaller and kasan in 0day?

Fengguang, what are your thoughts on including these in 0-day?

Thanks,

--
Darren Hart
Intel Open Source Technology Center

On 4/12/16, 10:24 PM, "Kostya Serebryany" <kcc(a)google.com<mailto:kcc@google.com>> wrote:

CC-ing more people after today's conversation at the Intel Tech Days.

We'd like to add kasan and syzkaller [1,2,3,4] to the 0-day kbuild system.
We believe this has a large potential to find old bugs and prevent regressions in the Kernel.
How do we achieve this?

Thanks,

--kcc

[1] https://github.com/google/syzkaller
[2] https://github.com/google/syzkaller/wiki/Found-Bugs
[3] https://lwn.net/Articles/677764/
[4] https://www.kernel.org/doc/Documentation/kasan.txt


On Tue, Dec 15, 2015 at 3:49 AM, David Drysdale <drysdale(a)google.com<mailto:drysdale@google.com>> wrote:
Hi Fengguang / LKP-folk,

Quick question -- how easy is it to add extra builds/tests/checks to
your marvellous 0-day kbuild system?

The reason I ask is that I've recently been exploring syzkaller [1],
which is a system call fuzzer written by some of my colleagues here at
Google (cc'ed).  Although it's fairly new, it has uncovered a bunch of
kernel bugs already [2] so I wondered if it might be a good candidate
for inclusion in the 0-day checks at some point.

(As an aside, I'm in the process of writing an article about syzkaller
for LWN, which might also expose it to more folk.)

What do you think?

Thanks,
David

[1] https://github.com/google/syzkaller
[2] https://github.com/google/syzkaller/wiki/Found-Bugs

Re: Extending the 0-day system with syzkaller?

[Kostya Serebryany]

[-- Attachment #1: Type: text/plain, Size: 2614 bytes --]

Ping...
(I know that the recent news at Intel might be quite distracting for some
you...)

On Fri, Apr 15, 2016 at 1:05 PM, Hart, Darren <darren.hart@intel.com> wrote:

> Hi Fengguang,
>
> I met with Kostya at Intel Tech Days and he had some compelling arguments
> for including some of these tests.
>
> Dave H: I took a quick look at Kasan, which appears to require an existing
> config option (CONFIG_KASAN) for dynamic memory access checking using
> shadow memory. Is this something you would like to see added to 0-day? Do
> we have anything today which provides comparable coverage?
>
> Combining Kasan and other existing kernel integrity checking, the
> syzkaller fuzz tester is showing promising results and the reports come in
> the form of kernel oops and similar things which we already check for in
> 0-day.
>
> For the others on Cc, do you have additional context for or against
> including syzkaller and kasan in 0day?
>
> Fengguang, what are your thoughts on including these in 0-day?
>
> Thanks,
>
> --
> Darren Hart
> Intel Open Source Technology Center
>
> On 4/12/16, 10:24 PM, "Kostya Serebryany" <kcc@google.com<mailto:
> kcc(a)google.com>> wrote:
>
> CC-ing more people after today's conversation at the Intel Tech Days.
>
> We'd like to add kasan and syzkaller [1,2,3,4] to the 0-day kbuild system.
> We believe this has a large potential to find old bugs and prevent
> regressions in the Kernel.
> How do we achieve this?
>
> Thanks,
>
> --kcc
>
> [1] https://github.com/google/syzkaller
> [2] https://github.com/google/syzkaller/wiki/Found-Bugs
> [3] https://lwn.net/Articles/677764/
> [4] https://www.kernel.org/doc/Documentation/kasan.txt
>
>
> On Tue, Dec 15, 2015 at 3:49 AM, David Drysdale <drysdale@google.com
> <mailto:drysdale@google.com>> wrote:
> Hi Fengguang / LKP-folk,
>
> Quick question -- how easy is it to add extra builds/tests/checks to
> your marvellous 0-day kbuild system?
>
> The reason I ask is that I've recently been exploring syzkaller [1],
> which is a system call fuzzer written by some of my colleagues here at
> Google (cc'ed).  Although it's fairly new, it has uncovered a bunch of
> kernel bugs already [2] so I wondered if it might be a good candidate
> for inclusion in the 0-day checks at some point.
>
> (As an aside, I'm in the process of writing an article about syzkaller
> for LWN, which might also expose it to more folk.)
>
> What do you think?
>
> Thanks,
> David
>
> [1] https://github.com/google/syzkaller
> [2] https://github.com/google/syzkaller/wiki/Found-Bugs
>
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 3790 bytes --]

Re: Extending the 0-day system with syzkaller?

[Philip Li]

[-- Attachment #1: Type: text/plain, Size: 3488 bytes --]

On Wed, Apr 27, 2016 at 11:03:42AM -0700, Kostya Serebryany wrote:
> Ping...
> (I know that the recent news at Intel might be quite distracting for some
> you...)
hi all, this is Philip from 0day team, sorry for late reply. We are glad to add
syzkaller test suite to 0day, and we will do detail plan of it within this quarter
and share with by end of June including what the first step can be and any support
if needed.

Is this time frame ok? because right now, all resources are booked, and
formal action may need start next quarter.

One quick question is i recall an early discussion is that this enabling requires new gcc,
is this still be true?

Thanks

> 
> On Fri, Apr 15, 2016 at 1:05 PM, Hart, Darren <darren.hart@intel.com> wrote:
> 
> > Hi Fengguang,
> >
> > I met with Kostya at Intel Tech Days and he had some compelling arguments
> > for including some of these tests.
> >
> > Dave H: I took a quick look at Kasan, which appears to require an existing
> > config option (CONFIG_KASAN) for dynamic memory access checking using
> > shadow memory. Is this something you would like to see added to 0-day? Do
> > we have anything today which provides comparable coverage?
> >
> > Combining Kasan and other existing kernel integrity checking, the
> > syzkaller fuzz tester is showing promising results and the reports come in
> > the form of kernel oops and similar things which we already check for in
> > 0-day.
> >
> > For the others on Cc, do you have additional context for or against
> > including syzkaller and kasan in 0day?
> >
> > Fengguang, what are your thoughts on including these in 0-day?
> >
> > Thanks,
> >
> > --
> > Darren Hart
> > Intel Open Source Technology Center
> >
> > On 4/12/16, 10:24 PM, "Kostya Serebryany" <kcc@google.com<mailto:
> > kcc(a)google.com>> wrote:
> >
> > CC-ing more people after today's conversation at the Intel Tech Days.
> >
> > We'd like to add kasan and syzkaller [1,2,3,4] to the 0-day kbuild system.
> > We believe this has a large potential to find old bugs and prevent
> > regressions in the Kernel.
> > How do we achieve this?
> >
> > Thanks,
> >
> > --kcc
> >
> > [1] https://github.com/google/syzkaller
> > [2] https://github.com/google/syzkaller/wiki/Found-Bugs
> > [3] https://lwn.net/Articles/677764/
> > [4] https://www.kernel.org/doc/Documentation/kasan.txt
> >
> >
> > On Tue, Dec 15, 2015 at 3:49 AM, David Drysdale <drysdale@google.com
> > <mailto:drysdale@google.com>> wrote:
> > Hi Fengguang / LKP-folk,
> >
> > Quick question -- how easy is it to add extra builds/tests/checks to
> > your marvellous 0-day kbuild system?
> >
> > The reason I ask is that I've recently been exploring syzkaller [1],
> > which is a system call fuzzer written by some of my colleagues here at
> > Google (cc'ed).  Although it's fairly new, it has uncovered a bunch of
> > kernel bugs already [2] so I wondered if it might be a good candidate
> > for inclusion in the 0-day checks at some point.
> >
> > (As an aside, I'm in the process of writing an article about syzkaller
> > for LWN, which might also expose it to more folk.)
> >
> > What do you think?
> >
> > Thanks,
> > David
> >
> > [1] https://github.com/google/syzkaller
> > [2] https://github.com/google/syzkaller/wiki/Found-Bugs
> >
> >

> _______________________________________________
> LKP mailing list
> LKP(a)lists.01.org
> https://lists.01.org/mailman/listinfo/lkp

Re: Extending the 0-day system with syzkaller?

[Dmitry Vyukov]

[-- Attachment #1: Type: text/plain, Size: 3991 bytes --]

On Thu, Apr 28, 2016 at 3:41 AM, Philip Li <philip.li@intel.com> wrote:
> On Wed, Apr 27, 2016 at 11:03:42AM -0700, Kostya Serebryany wrote:
>> Ping...
>> (I know that the recent news at Intel might be quite distracting for some
>> you...)
> hi all, this is Philip from 0day team, sorry for late reply. We are glad to add
> syzkaller test suite to 0day, and we will do detail plan of it within this quarter
> and share with by end of June including what the first step can be and any support
> if needed.
>
> Is this time frame ok? because right now, all resources are booked, and
> formal action may need start next quarter.

Hi Philip,

The time frame is OK

> One quick question is i recall an early discussion is that this enabling requires new gcc,
> is this still be true?

Syzkaller requires compiler support. I use new gcc. However, there are
currently patches in flight that add support for gcc plugins to kernel
build system. As far as I understand, with the plugin support no
messing with gcc is required (KCOV instrumentation plugin will also be
in kernel tree).



>> On Fri, Apr 15, 2016 at 1:05 PM, Hart, Darren <darren.hart@intel.com> wrote:
>>
>> > Hi Fengguang,
>> >
>> > I met with Kostya at Intel Tech Days and he had some compelling arguments
>> > for including some of these tests.
>> >
>> > Dave H: I took a quick look at Kasan, which appears to require an existing
>> > config option (CONFIG_KASAN) for dynamic memory access checking using
>> > shadow memory. Is this something you would like to see added to 0-day? Do
>> > we have anything today which provides comparable coverage?
>> >
>> > Combining Kasan and other existing kernel integrity checking, the
>> > syzkaller fuzz tester is showing promising results and the reports come in
>> > the form of kernel oops and similar things which we already check for in
>> > 0-day.
>> >
>> > For the others on Cc, do you have additional context for or against
>> > including syzkaller and kasan in 0day?
>> >
>> > Fengguang, what are your thoughts on including these in 0-day?
>> >
>> > Thanks,
>> >
>> > --
>> > Darren Hart
>> > Intel Open Source Technology Center
>> >
>> > On 4/12/16, 10:24 PM, "Kostya Serebryany" <kcc@google.com<mailto:
>> > kcc(a)google.com>> wrote:
>> >
>> > CC-ing more people after today's conversation at the Intel Tech Days.
>> >
>> > We'd like to add kasan and syzkaller [1,2,3,4] to the 0-day kbuild system.
>> > We believe this has a large potential to find old bugs and prevent
>> > regressions in the Kernel.
>> > How do we achieve this?
>> >
>> > Thanks,
>> >
>> > --kcc
>> >
>> > [1] https://github.com/google/syzkaller
>> > [2] https://github.com/google/syzkaller/wiki/Found-Bugs
>> > [3] https://lwn.net/Articles/677764/
>> > [4] https://www.kernel.org/doc/Documentation/kasan.txt
>> >
>> >
>> > On Tue, Dec 15, 2015 at 3:49 AM, David Drysdale <drysdale@google.com
>> > <mailto:drysdale@google.com>> wrote:
>> > Hi Fengguang / LKP-folk,
>> >
>> > Quick question -- how easy is it to add extra builds/tests/checks to
>> > your marvellous 0-day kbuild system?
>> >
>> > The reason I ask is that I've recently been exploring syzkaller [1],
>> > which is a system call fuzzer written by some of my colleagues here at
>> > Google (cc'ed).  Although it's fairly new, it has uncovered a bunch of
>> > kernel bugs already [2] so I wondered if it might be a good candidate
>> > for inclusion in the 0-day checks at some point.
>> >
>> > (As an aside, I'm in the process of writing an article about syzkaller
>> > for LWN, which might also expose it to more folk.)
>> >
>> > What do you think?
>> >
>> > Thanks,
>> > David
>> >
>> > [1] https://github.com/google/syzkaller
>> > [2] https://github.com/google/syzkaller/wiki/Found-Bugs
>> >
>> >
>
>> _______________________________________________
>> LKP mailing list
>> LKP(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/lkp
>

Re: Extending the 0-day system with syzkaller?

[Kees Cook]

[-- Attachment #1: Type: text/plain, Size: 1497 bytes --]

On Thu, Apr 28, 2016 at 4:14 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Thu, Apr 28, 2016 at 3:41 AM, Philip Li <philip.li@intel.com> wrote:
>> On Wed, Apr 27, 2016 at 11:03:42AM -0700, Kostya Serebryany wrote:
>>> Ping...
>>> (I know that the recent news at Intel might be quite distracting for some
>>> you...)
>> hi all, this is Philip from 0day team, sorry for late reply. We are glad to add
>> syzkaller test suite to 0day, and we will do detail plan of it within this quarter
>> and share with by end of June including what the first step can be and any support
>> if needed.
>>
>> Is this time frame ok? because right now, all resources are booked, and
>> formal action may need start next quarter.
>
> Hi Philip,
>
> The time frame is OK
>
>> One quick question is i recall an early discussion is that this enabling requires new gcc,
>> is this still be true?
>
> Syzkaller requires compiler support. I use new gcc. However, there are
> currently patches in flight that add support for gcc plugins to kernel
> build system. As far as I understand, with the plugin support no
> messing with gcc is required (KCOV instrumentation plugin will also be
> in kernel tree).

I'm still waiting on the kbuild maintainer to speak up on this series,
so if other folks could chime in on the thread saying they would put
it to good use, I would greatly appreciate that:

https://lkml.org/lkml/2016/4/7/750

-Kees

-- 
Kees Cook
Chrome OS & Brillo Security

Re: Extending the 0-day system with syzkaller?

[Fengguang Wu]

[-- Attachment #1: Type: text/plain, Size: 945 bytes --]

> One quick question is i recall an early discussion is that this enabling requires new gcc,
> is this still be true?

I see Debian just upgraded its gcc packages to 6.1, which makes it
more convenient for us to integrate with syzkaller.

Since syzkaller is more about a client-server system than a hands on
tool like trinity, it may need non-trivial modifications to 0-day and
possibly syzkaller to achieve the integration. For example, syzkaller
and 0-day each has its own way to run QEMU, which should be resolved
somehow.

Another question is, the 0-day model relies heavily on bisecting to
cover the large number of kernel trees. If a bug cannot be easily
reproduced and bisected, we don't know to whom it should be reported
to. Trinity offers an option "-sN: use N as random seed." to help
reproduce bugs it found. I wonder if syzkaller has similar mechanism
to help with reproducing bugs it found?

Thanks,
Fengguang

Re: Extending the 0-day system with syzkaller?

[Dmitry Vyukov]

[-- Attachment #1: Type: text/plain, Size: 2830 bytes --]

On Tue, May 3, 2016 at 6:01 AM, Fengguang Wu <fengguang.wu@intel.com> wrote:
>> One quick question is i recall an early discussion is that this enabling requires new gcc,
>> is this still be true?
>
> I see Debian just upgraded its gcc packages to 6.1, which makes it
> more convenient for us to integrate with syzkaller.
>
> Since syzkaller is more about a client-server system than a hands on
> tool like trinity, it may need non-trivial modifications to 0-day and
> possibly syzkaller to achieve the integration. For example, syzkaller
> and 0-day each has its own way to run QEMU, which should be resolved
> somehow.
>
> Another question is, the 0-day model relies heavily on bisecting to
> cover the large number of kernel trees. If a bug cannot be easily
> reproduced and bisected, we don't know to whom it should be reported
> to. Trinity offers an option "-sN: use N as random seed." to help
> reproduce bugs it found. I wonder if syzkaller has similar mechanism
> to help with reproducing bugs it found?


Hi Fengguang,

Yes, syzkaller setup is more elaborate, but architecture is pretty
modular so I believe we can resolve these issues.

What does your infrastructure do? I guess it winds up qemu instances,
copies necessary binaries and then runs some command? How does it
monitor for crashes? By parsing console output? How do you control
duration of testing? By instructing trinity to do fixed number of
iterations, or by killing qemu?

Syzkaller already has a number of simpler tools. For example,
syz-stress works more or less like trinity -- you can copy it onto
test machine and instruct to run random programs. I can cook another
tool that will fit into your setup. One inherent complication, though,
is that syzkaller is stateful, it needs to accumulate corpus of
interesting programs (based on code coverage) to work more
efficiently. So we will need some way to save corpus from qemu
instances externally and then pass the corpus to new qemu instances.
But on the other hand, that was proven to be super efficient in
finding regressions -- namely, in 15 minutes it can test almost
everything that it discovered previously.

Re reproducers: syzkaller can do even better than trinity. Syzkaller
logs all executed programs to stdout (in default mode, or it can also
log to local files or kernel console). If you have the log with
programs that were executed before the crash, then it's possible (1)
to re-execute the log on other kernels, or (2) extract the guilty
program from the log, minimize it and convert to an executable C
program to present to developers
(https://github.com/google/syzkaller/wiki/Crash-reproducer-programs).
Based on my experience (2) is real game changer, developers much more
willing to fix bugs when they have a small reproducer.

Re: Extending the 0-day system with syzkaller?

[Fengguang Wu]

[-- Attachment #1: Type: text/plain, Size: 3976 bytes --]

Hi Dmitry,

On Tue, May 03, 2016 at 10:00:04AM +0200, Dmitry Vyukov wrote:
> On Tue, May 3, 2016 at 6:01 AM, Fengguang Wu <fengguang.wu@intel.com> wrote:
> >> One quick question is i recall an early discussion is that this enabling requires new gcc,
> >> is this still be true?
> >
> > I see Debian just upgraded its gcc packages to 6.1, which makes it
> > more convenient for us to integrate with syzkaller.
> >
> > Since syzkaller is more about a client-server system than a hands on
> > tool like trinity, it may need non-trivial modifications to 0-day and
> > possibly syzkaller to achieve the integration. For example, syzkaller
> > and 0-day each has its own way to run QEMU, which should be resolved
> > somehow.
> >
> > Another question is, the 0-day model relies heavily on bisecting to
> > cover the large number of kernel trees. If a bug cannot be easily
> > reproduced and bisected, we don't know to whom it should be reported
> > to. Trinity offers an option "-sN: use N as random seed." to help
> > reproduce bugs it found. I wonder if syzkaller has similar mechanism
> > to help with reproducing bugs it found?
> 
> 
> Hi Fengguang,
> 
> Yes, syzkaller setup is more elaborate, but architecture is pretty
> modular so I believe we can resolve these issues.

Yes syzkaller looks very interesting. I'll try it out in more details
and try enabling it in 0day.

> What does your infrastructure do? I guess it winds up qemu instances,
> copies necessary binaries and then runs some command? How does it
> monitor for crashes? By parsing console output? How do you control
> duration of testing? By instructing trinity to do fixed number of
> iterations, or by killing qemu?

Basically each 0day boot server runs a number of QEMU instances
with various locally cached QEMU images. Trinity binaries are built
into these images. A boot script in the images will run trinity in
background, sleep for some time and then kill trinity. If tests run
smoothly, the scripts inside the QEMU image will be able to shutdown
VM in time. Otherwise if things get stuck inside VM, the host side
script which started the QEMU VM and monitors its serial output may
decide to kill the QEMU process when either timeout or find kernel
panic patterns.

> Syzkaller already has a number of simpler tools. For example,
> syz-stress works more or less like trinity -- you can copy it onto
> test machine and instruct to run random programs. I can cook another
> tool that will fit into your setup.

OK, we may sort details out in private emails. Preferably the resulted
system should get the most out of what syzkaller offers and excels.

> One inherent complication, though,
> is that syzkaller is stateful, it needs to accumulate corpus of
> interesting programs (based on code coverage) to work more
> efficiently. So we will need some way to save corpus from qemu
> instances externally and then pass the corpus to new qemu instances.
> But on the other hand, that was proven to be super efficient in
> finding regressions -- namely, in 15 minutes it can test almost
> everything that it discovered previously.

That looks very impressive!

> Re reproducers: syzkaller can do even better than trinity. Syzkaller
> logs all executed programs to stdout (in default mode, or it can also
> log to local files or kernel console). If you have the log with
> programs that were executed before the crash, then it's possible (1)
> to re-execute the log on other kernels, or (2) extract the guilty
> program from the log, minimize it and convert to an executable C
> program to present to developers
> (https://github.com/google/syzkaller/wiki/Crash-reproducer-programs).
> Based on my experience (2) is real game changer, developers much more
> willing to fix bugs when they have a small reproducer.

Cannot agree any more -- it looks the right way to go.
Hope we can make it run in 0day soon!

Thanks,
Fengguang

Re: Extending the 0-day system with syzkaller?

[Dmitry Vyukov]

[-- Attachment #1: Type: text/plain, Size: 4750 bytes --]

On Thu, May 5, 2016 at 4:48 PM, Fengguang Wu <fengguang.wu@intel.com> wrote:
> Hi Dmitry,
>
> On Tue, May 03, 2016 at 10:00:04AM +0200, Dmitry Vyukov wrote:
>> On Tue, May 3, 2016 at 6:01 AM, Fengguang Wu <fengguang.wu@intel.com> wrote:
>> >> One quick question is i recall an early discussion is that this enabling requires new gcc,
>> >> is this still be true?
>> >
>> > I see Debian just upgraded its gcc packages to 6.1, which makes it
>> > more convenient for us to integrate with syzkaller.
>> >
>> > Since syzkaller is more about a client-server system than a hands on
>> > tool like trinity, it may need non-trivial modifications to 0-day and
>> > possibly syzkaller to achieve the integration. For example, syzkaller
>> > and 0-day each has its own way to run QEMU, which should be resolved
>> > somehow.
>> >
>> > Another question is, the 0-day model relies heavily on bisecting to
>> > cover the large number of kernel trees. If a bug cannot be easily
>> > reproduced and bisected, we don't know to whom it should be reported
>> > to. Trinity offers an option "-sN: use N as random seed." to help
>> > reproduce bugs it found. I wonder if syzkaller has similar mechanism
>> > to help with reproducing bugs it found?
>>
>>
>> Hi Fengguang,
>>
>> Yes, syzkaller setup is more elaborate, but architecture is pretty
>> modular so I believe we can resolve these issues.
>
> Yes syzkaller looks very interesting. I'll try it out in more details
> and try enabling it in 0day.
>
>> What does your infrastructure do? I guess it winds up qemu instances,
>> copies necessary binaries and then runs some command? How does it
>> monitor for crashes? By parsing console output? How do you control
>> duration of testing? By instructing trinity to do fixed number of
>> iterations, or by killing qemu?
>
> Basically each 0day boot server runs a number of QEMU instances
> with various locally cached QEMU images. Trinity binaries are built
> into these images. A boot script in the images will run trinity in
> background, sleep for some time and then kill trinity. If tests run
> smoothly, the scripts inside the QEMU image will be able to shutdown
> VM in time. Otherwise if things get stuck inside VM, the host side
> script which started the QEMU VM and monitors its serial output may
> decide to kill the QEMU process when either timeout or find kernel
> panic patterns.


On second thought, what do you think about running syzkaller as is?
Syzkaller does roughly the same thing. Syzkaller is configurable with
kernel, image, qemu-binary, etc. I can provide any missing
functionality for automation (e.g. run for an hour and exit). That
would be the simplest option.

If it does not work for you for some reason, we would need a way to
copy files to/from a VM with scp. Then we could copy the corpus.zip
into a VM, start fuzzing for some time, then it will finish and save
an updated corpus.zip, then we copy out the corpus to host for
subsequent runs.


>> Syzkaller already has a number of simpler tools. For example,
>> syz-stress works more or less like trinity -- you can copy it onto
>> test machine and instruct to run random programs. I can cook another
>> tool that will fit into your setup.
>
> OK, we may sort details out in private emails. Preferably the resulted
> system should get the most out of what syzkaller offers and excels.
>
>> One inherent complication, though,
>> is that syzkaller is stateful, it needs to accumulate corpus of
>> interesting programs (based on code coverage) to work more
>> efficiently. So we will need some way to save corpus from qemu
>> instances externally and then pass the corpus to new qemu instances.
>> But on the other hand, that was proven to be super efficient in
>> finding regressions -- namely, in 15 minutes it can test almost
>> everything that it discovered previously.
>
> That looks very impressive!
>
>> Re reproducers: syzkaller can do even better than trinity. Syzkaller
>> logs all executed programs to stdout (in default mode, or it can also
>> log to local files or kernel console). If you have the log with
>> programs that were executed before the crash, then it's possible (1)
>> to re-execute the log on other kernels, or (2) extract the guilty
>> program from the log, minimize it and convert to an executable C
>> program to present to developers
>> (https://github.com/google/syzkaller/wiki/Crash-reproducer-programs).
>> Based on my experience (2) is real game changer, developers much more
>> willing to fix bugs when they have a small reproducer.
>
> Cannot agree any more -- it looks the right way to go.
> Hope we can make it run in 0day soon!
>
> Thanks,
> Fengguang

Re: Extending the 0-day system with syzkaller?

[Dmitry Vyukov]

[-- Attachment #1: Type: text/plain, Size: 5014 bytes --]

Hi Fengguang,

Any updates on this? How do you see the way forward?




On Fri, May 6, 2016 at 10:08 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Thu, May 5, 2016 at 4:48 PM, Fengguang Wu <fengguang.wu@intel.com> wrote:
>> Hi Dmitry,
>>
>> On Tue, May 03, 2016 at 10:00:04AM +0200, Dmitry Vyukov wrote:
>>> On Tue, May 3, 2016 at 6:01 AM, Fengguang Wu <fengguang.wu@intel.com> wrote:
>>> >> One quick question is i recall an early discussion is that this enabling requires new gcc,
>>> >> is this still be true?
>>> >
>>> > I see Debian just upgraded its gcc packages to 6.1, which makes it
>>> > more convenient for us to integrate with syzkaller.
>>> >
>>> > Since syzkaller is more about a client-server system than a hands on
>>> > tool like trinity, it may need non-trivial modifications to 0-day and
>>> > possibly syzkaller to achieve the integration. For example, syzkaller
>>> > and 0-day each has its own way to run QEMU, which should be resolved
>>> > somehow.
>>> >
>>> > Another question is, the 0-day model relies heavily on bisecting to
>>> > cover the large number of kernel trees. If a bug cannot be easily
>>> > reproduced and bisected, we don't know to whom it should be reported
>>> > to. Trinity offers an option "-sN: use N as random seed." to help
>>> > reproduce bugs it found. I wonder if syzkaller has similar mechanism
>>> > to help with reproducing bugs it found?
>>>
>>>
>>> Hi Fengguang,
>>>
>>> Yes, syzkaller setup is more elaborate, but architecture is pretty
>>> modular so I believe we can resolve these issues.
>>
>> Yes syzkaller looks very interesting. I'll try it out in more details
>> and try enabling it in 0day.
>>
>>> What does your infrastructure do? I guess it winds up qemu instances,
>>> copies necessary binaries and then runs some command? How does it
>>> monitor for crashes? By parsing console output? How do you control
>>> duration of testing? By instructing trinity to do fixed number of
>>> iterations, or by killing qemu?
>>
>> Basically each 0day boot server runs a number of QEMU instances
>> with various locally cached QEMU images. Trinity binaries are built
>> into these images. A boot script in the images will run trinity in
>> background, sleep for some time and then kill trinity. If tests run
>> smoothly, the scripts inside the QEMU image will be able to shutdown
>> VM in time. Otherwise if things get stuck inside VM, the host side
>> script which started the QEMU VM and monitors its serial output may
>> decide to kill the QEMU process when either timeout or find kernel
>> panic patterns.
>
>
> On second thought, what do you think about running syzkaller as is?
> Syzkaller does roughly the same thing. Syzkaller is configurable with
> kernel, image, qemu-binary, etc. I can provide any missing
> functionality for automation (e.g. run for an hour and exit). That
> would be the simplest option.
>
> If it does not work for you for some reason, we would need a way to
> copy files to/from a VM with scp. Then we could copy the corpus.zip
> into a VM, start fuzzing for some time, then it will finish and save
> an updated corpus.zip, then we copy out the corpus to host for
> subsequent runs.
>
>
>>> Syzkaller already has a number of simpler tools. For example,
>>> syz-stress works more or less like trinity -- you can copy it onto
>>> test machine and instruct to run random programs. I can cook another
>>> tool that will fit into your setup.
>>
>> OK, we may sort details out in private emails. Preferably the resulted
>> system should get the most out of what syzkaller offers and excels.
>>
>>> One inherent complication, though,
>>> is that syzkaller is stateful, it needs to accumulate corpus of
>>> interesting programs (based on code coverage) to work more
>>> efficiently. So we will need some way to save corpus from qemu
>>> instances externally and then pass the corpus to new qemu instances.
>>> But on the other hand, that was proven to be super efficient in
>>> finding regressions -- namely, in 15 minutes it can test almost
>>> everything that it discovered previously.
>>
>> That looks very impressive!
>>
>>> Re reproducers: syzkaller can do even better than trinity. Syzkaller
>>> logs all executed programs to stdout (in default mode, or it can also
>>> log to local files or kernel console). If you have the log with
>>> programs that were executed before the crash, then it's possible (1)
>>> to re-execute the log on other kernels, or (2) extract the guilty
>>> program from the log, minimize it and convert to an executable C
>>> program to present to developers
>>> (https://github.com/google/syzkaller/wiki/Crash-reproducer-programs).
>>> Based on my experience (2) is real game changer, developers much more
>>> willing to fix bugs when they have a small reproducer.
>>
>> Cannot agree any more -- it looks the right way to go.
>> Hope we can make it run in 0day soon!
>>
>> Thanks,
>> Fengguang

Re: Extending the 0-day system with syzkaller?

[Fengguang Wu]

[-- Attachment #1: Type: text/plain, Size: 5431 bytes --]

Hi Dmitry,

On Mon, Jun 06, 2016 at 07:11:02PM +0200, Dmitry Vyukov wrote:
> Hi Fengguang,
> 
> Any updates on this? How do you see the way forward?

Sorry no progress yet, however syzkaller is in our high priority list
and will be my next major topic to focus on.

Thanks,
Fengguang

> On Fri, May 6, 2016 at 10:08 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> > On Thu, May 5, 2016 at 4:48 PM, Fengguang Wu <fengguang.wu@intel.com> wrote:
> >> Hi Dmitry,
> >>
> >> On Tue, May 03, 2016 at 10:00:04AM +0200, Dmitry Vyukov wrote:
> >>> On Tue, May 3, 2016 at 6:01 AM, Fengguang Wu <fengguang.wu@intel.com> wrote:
> >>> >> One quick question is i recall an early discussion is that this enabling requires new gcc,
> >>> >> is this still be true?
> >>> >
> >>> > I see Debian just upgraded its gcc packages to 6.1, which makes it
> >>> > more convenient for us to integrate with syzkaller.
> >>> >
> >>> > Since syzkaller is more about a client-server system than a hands on
> >>> > tool like trinity, it may need non-trivial modifications to 0-day and
> >>> > possibly syzkaller to achieve the integration. For example, syzkaller
> >>> > and 0-day each has its own way to run QEMU, which should be resolved
> >>> > somehow.
> >>> >
> >>> > Another question is, the 0-day model relies heavily on bisecting to
> >>> > cover the large number of kernel trees. If a bug cannot be easily
> >>> > reproduced and bisected, we don't know to whom it should be reported
> >>> > to. Trinity offers an option "-sN: use N as random seed." to help
> >>> > reproduce bugs it found. I wonder if syzkaller has similar mechanism
> >>> > to help with reproducing bugs it found?
> >>>
> >>>
> >>> Hi Fengguang,
> >>>
> >>> Yes, syzkaller setup is more elaborate, but architecture is pretty
> >>> modular so I believe we can resolve these issues.
> >>
> >> Yes syzkaller looks very interesting. I'll try it out in more details
> >> and try enabling it in 0day.
> >>
> >>> What does your infrastructure do? I guess it winds up qemu instances,
> >>> copies necessary binaries and then runs some command? How does it
> >>> monitor for crashes? By parsing console output? How do you control
> >>> duration of testing? By instructing trinity to do fixed number of
> >>> iterations, or by killing qemu?
> >>
> >> Basically each 0day boot server runs a number of QEMU instances
> >> with various locally cached QEMU images. Trinity binaries are built
> >> into these images. A boot script in the images will run trinity in
> >> background, sleep for some time and then kill trinity. If tests run
> >> smoothly, the scripts inside the QEMU image will be able to shutdown
> >> VM in time. Otherwise if things get stuck inside VM, the host side
> >> script which started the QEMU VM and monitors its serial output may
> >> decide to kill the QEMU process when either timeout or find kernel
> >> panic patterns.
> >
> >
> > On second thought, what do you think about running syzkaller as is?
> > Syzkaller does roughly the same thing. Syzkaller is configurable with
> > kernel, image, qemu-binary, etc. I can provide any missing
> > functionality for automation (e.g. run for an hour and exit). That
> > would be the simplest option.
> >
> > If it does not work for you for some reason, we would need a way to
> > copy files to/from a VM with scp. Then we could copy the corpus.zip
> > into a VM, start fuzzing for some time, then it will finish and save
> > an updated corpus.zip, then we copy out the corpus to host for
> > subsequent runs.
> >
> >
> >>> Syzkaller already has a number of simpler tools. For example,
> >>> syz-stress works more or less like trinity -- you can copy it onto
> >>> test machine and instruct to run random programs. I can cook another
> >>> tool that will fit into your setup.
> >>
> >> OK, we may sort details out in private emails. Preferably the resulted
> >> system should get the most out of what syzkaller offers and excels.
> >>
> >>> One inherent complication, though,
> >>> is that syzkaller is stateful, it needs to accumulate corpus of
> >>> interesting programs (based on code coverage) to work more
> >>> efficiently. So we will need some way to save corpus from qemu
> >>> instances externally and then pass the corpus to new qemu instances.
> >>> But on the other hand, that was proven to be super efficient in
> >>> finding regressions -- namely, in 15 minutes it can test almost
> >>> everything that it discovered previously.
> >>
> >> That looks very impressive!
> >>
> >>> Re reproducers: syzkaller can do even better than trinity. Syzkaller
> >>> logs all executed programs to stdout (in default mode, or it can also
> >>> log to local files or kernel console). If you have the log with
> >>> programs that were executed before the crash, then it's possible (1)
> >>> to re-execute the log on other kernels, or (2) extract the guilty
> >>> program from the log, minimize it and convert to an executable C
> >>> program to present to developers
> >>> (https://github.com/google/syzkaller/wiki/Crash-reproducer-programs).
> >>> Based on my experience (2) is real game changer, developers much more
> >>> willing to fix bugs when they have a small reproducer.
> >>
> >> Cannot agree any more -- it looks the right way to go.
> >> Hope we can make it run in 0day soon!
> >>
> >> Thanks,
> >> Fengguang

Re: Extending the 0-day system with syzkaller?

[Dmitry Vyukov]

[-- Attachment #1: Type: text/plain, Size: 467 bytes --]

On Tue, Jun 7, 2016 at 3:04 PM, Fengguang Wu <fengguang.wu@intel.com> wrote:
> Hi Dmitry,
>
> On Mon, Jun 06, 2016 at 07:11:02PM +0200, Dmitry Vyukov wrote:
>> Hi Fengguang,
>>
>> Any updates on this? How do you see the way forward?
>
> Sorry no progress yet, however syzkaller is in our high priority list
> and will be my next major topic to focus on.


Great!
Ping this thread when you start. We are also interested and ready to
do supporting work.

Re: Extending the 0-day system with syzkaller?

[Fengguang Wu]

[-- Attachment #1: Type: text/plain, Size: 602 bytes --]

On Tue, Jun 07, 2016 at 03:19:02PM +0200, Dmitry Vyukov wrote:
> On Tue, Jun 7, 2016 at 3:04 PM, Fengguang Wu <fengguang.wu@intel.com> wrote:
> > Hi Dmitry,
> >
> > On Mon, Jun 06, 2016 at 07:11:02PM +0200, Dmitry Vyukov wrote:
> >> Hi Fengguang,
> >>
> >> Any updates on this? How do you see the way forward?
> >
> > Sorry no progress yet, however syzkaller is in our high priority list
> > and will be my next major topic to focus on.
> 
> 
> Great!
> Ping this thread when you start. We are also interested and ready to
> do supporting work.

OK, thank you!

Regards,
Fengguang