Re: About libfuzzer in qemu

Re: About libfuzzer in qemu

[Alexander Bulekov]

On 210304 1843, Yan Zhiqiang wrote:
> Hello Alex,
> I'm learning the fuzz in QEMU recently, I review the fuzz code under
> /tests/qtest/fuzz which is written by you.
> I learn a lot from it, but I stuck when I want to debug the fuzz code.
> I use the gdb with command as follows:
> 
> >  gdb -q --args ./qemu-fuzz-i386 --fuzz-target=generic-fuzz-virtio-vga
> > ./fuzz-output
> 
> and set breakpoint at generic_fuzz.c:generic_fuzz.
> It acctually stop when hit the breakpoint. But the function argument Size
> is zero and then goto _Exit(0). (try many times but always the same)

Hi Zhiqiang,
Happy to have more people look at the fuzzing code.
We run each input in a forked process. Maybe you need to run 
"set follow-fork-mode child" in gdb?

> Then input `c` to continue.
> However, it never hit the breakpoint after that. Just as the picture showed.
> [image: 1614854239086.jpg]
> I tried the qtest debug method, but failed.
> I want to know the real process state to learn the QEMU fuzz and add a new
> fuzzer for QEMU.
> Could you tell me what's the right method to debug the fuzz code? Thank you!

For debugging crash, I usually build QEMU with --enable-sanitizers
(ASAN), and I convert the crash to a "QTest" reproducer, so it can be
debugged in a normal build of qemu. There's an RFC that has instructions
for how to do this:
https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg06035.html

Let me know if I can provide any more info.
-Alex

> 
> Regards,
> Zhiqiang Yan

Re: About libfuzzer in qemu

[Qiuhao Li]

On Thu, 2021-03-04 at 10:23 -0500, Alexander Bulekov wrote:
> On 210304 1843, Yan Zhiqiang wrote:
> > Hello Alex,
> > I'm learning the fuzz in QEMU recently, I review the fuzz code
> > under
> > /tests/qtest/fuzz which is written by you.
> > I learn a lot from it, but I stuck when I want to debug the fuzz
> > code.
> > I use the gdb with command as follows:
> > 
> > >  gdb -q --args ./qemu-fuzz-i386 --fuzz-target=generic-fuzz-
> > > virtio-vga
> > > ./fuzz-output
> > 
> > and set breakpoint at generic_fuzz.c:generic_fuzz.
> > It acctually stop when hit the breakpoint. But the function
> > argument Size
> > is zero and then goto _Exit(0). (try many times but always the
> > same)
> 
> Hi Zhiqiang,
> Happy to have more people look at the fuzzing code.
> We run each input in a forked process. Maybe you need to run 
> "set follow-fork-mode child" in gdb?

Hi Alex,

Just curious why you choose to use the libfuzzer at first instead of
AFL and its descendants like AFL++ since they use a forkserver by
design, and the performance also seems better [1].

[1] https://www.fuzzbench.com/reports/2021-02-13-paper/index.html

Thank you.
  Qiuhao Li

Re: About libfuzzer in qemu

[Alexander Bulekov]

On 210305 1551, Qiuhao Li wrote:
> On Thu, 2021-03-04 at 10:23 -0500, Alexander Bulekov wrote:
> > On 210304 1843, Yan Zhiqiang wrote:
> > > Hello Alex,
> > > I'm learning the fuzz in QEMU recently, I review the fuzz code
> > > under
> > > /tests/qtest/fuzz which is written by you.
> > > I learn a lot from it, but I stuck when I want to debug the fuzz
> > > code.
> > > I use the gdb with command as follows:
> > > 
> > > >  gdb -q --args ./qemu-fuzz-i386 --fuzz-target=generic-fuzz-
> > > > virtio-vga
> > > > ./fuzz-output
> > > 
> > > and set breakpoint at generic_fuzz.c:generic_fuzz.
> > > It acctually stop when hit the breakpoint. But the function
> > > argument Size
> > > is zero and then goto _Exit(0). (try many times but always the
> > > same)
> > 
> > Hi Zhiqiang,
> > Happy to have more people look at the fuzzing code.
> > We run each input in a forked process. Maybe you need to run 
> > "set follow-fork-mode child" in gdb?
> 
> Hi Alex,
> 
> Just curious why you choose to use the libfuzzer at first instead of
> AFL and its descendants like AFL++ since they use a forkserver by
> design, and the performance also seems better [1].
> 
> [1] https://www.fuzzbench.com/reports/2021-02-13-paper/index.html

Hi Qiuhao,
It was a primary goal to run the fuzzers on OSS-Fuzz, and at the time,
AFL++ was not an option on OSS-Fuzz. Because I didn't find any great
way to reset QEMU between inputs, we hacked-in a forkserver.
It still seems that OSS-Fuzz is primarily designed around libfuzzer, but
now that there seems to be AFL++ support, it would be nice to start
using AFL++ as well. I just haven't had the time to look into it ;)

I've seen the fuzz-bench results - It will be interesting to see the
performance for an enormous target like QEMU. There are also some things
that AFL++ doesn't do as well as libfuzzer - some: examples here: 
https://github.com/AFLplusplus/fuzzer-challenges/tree/main/libfuzzer

-Alex

> 
> Thank you.
>   Qiuhao Li
> 
> 
> 

Re: About libfuzzer in qemu

[Alexander Bulekov]

I see the problem. You only hit the breakpoint in the first forked
process? I was able to sort-of work around this, using detach-on-fork.


gdb -q --args ./qemu-fuzz-i386
(gef) set follow-fork-mode child
(gef) set detach-on-fork off
(gef) b generic_fuzz.c:667
(gef) r --fuzz-target=generic-fuzz-virtio-vga -runs=1000
...
Thread 3.1 "qemu-fuzz-i386" hit Breakpoint 1, generic_fuzz
(gef) c
Continuing.
[Thread 0x7ffff32defc0 (LWP 806) exited]
(gef) info inferiors
  Num  Description       Executable
  1    process 31866     /home/alxndr/Downloads/qemu/build-fuzz/qemu-fuzz-i386
  2    process 802       /home/alxndr/Downloads/qemu/build-fuzz/qemu-fuzz-i386
* 3    <null>            /home/alxndr/Downloads/qemu/build-fuzz/qemu-fuzz-i386

# I make a guess that the parent is 802.

(gef) inferior 2 
[Switching to inferior 2 [process 802] (/home/alxndr/Downloads/qemu/build-fuzz/qemu-fuzz-i386)]
(gef) c
...
[New Thread 0x7fffca9fe700 (LWP 4538)]
[Switching to Thread 0x7ffff32defc0 (LWP 4537)]
Thread 3.1 "qemu-fuzz-i386" hit Breakpoint 1, generic_fuzz
(gef) c
Continuing.
[Thread 0x7ffff32defc0 (LWP 4537) exited]
(gef) inferior 2
[Switching to inferior 2 [process 802] (/home/alxndr/Downloads/qemu/build-fuzz/qemu-fuzz-i386)]
(gef) c
...

There's probably ways to automate most of this with gdb breakpoint
commands.


I usually don't need to debug more than one child, so this is a new
problem for me :) . Are you simply trying to reproduce a bug/crash?
My basic workflow for that is:
QEMU_FUZZ_TIMEOUT=0 QTEST_LOG=1 FUZZ_SERIALIZE_QTEST=1 \
./qemu-fuzz-target \
       --fuzz-target=generic-fuzz-virtio-vga ./crash-... > /tmp/out
./scripts/oss-fuzz/reorder_fuzzer_qtest_trace.py /tmp/out > /tmp/repro

# In /tmp/out find the line "Starting qemu with Arguments:" and copy the
# args ( without -qtest /dev/null)
less /tmp/out 

export QEMU_ARGS="-display none -machine accel=qtest, -m 512M -machine q35 -nodefaults -device virtio-vga"

# Reproduce the crash on a non-fuzz binary
./qemu-system-i386 $QEMU_ARGS -qtest stdio < /tmp/repro

For most cases, this should work. Let me know if there's any problem
-Alex



On 210308 1058, Yan Zhiqiang wrote:
> It hasn't changed.
> Can you show me the specific details of the entire debugging process?
> This is my debugging process as follows.(It will not hit the breakpoint as
> you can see)
> 
> ☁  build_qemu  printenv
> > XDG_RUNTIME_DIR=/run/user/500
> > XDG_SESSION_ID=2
> > ZSH=/home/ubuntu/.oh-my-zsh
> > _=/usr/bin/printenv
> > *QEMU_FUZZ_TIMEOUT=0*
> 
> 
> 
> ☁  build_qemu  gdb -q --args ./qemu-fuzz-i386
> > --fuzz-target=generic-fuzz-virtio-vga fuzz-output/
> > pwndbg: loaded 189 commands. Type pwndbg [filter] for a list.
> > pwndbg: created $rebase, $ida gdb functions (can be used with print/break)
> > Reading symbols from ./qemu-fuzz-i386...done.
> > warning: File "/home/ubuntu/qemu-lastest-fuzz-test/qemu/.gdbinit"
> > auto-loading has been declined by your `auto-load safe-path' set to
> > "$debugdir:$datadir/auto-load".
> > To enable execution of this file add
> >         add-auto-load-safe-path
> > /home/ubuntu/qemu-lastest-fuzz-test/qemu/.gdbinit
> > line to your configuration file "/home/ubuntu/.gdbinit".
> > To completely disable this security protection add
> >         set auto-load safe-path /
> > line to your configuration file "/home/ubuntu/.gdbinit".
> > For more information about this security protection see the
> > "Auto-loading safe path" section in the GDB manual.  E.g., run from the
> > shell:
> >         info "(gdb)Auto-loading safe path"
> > pwndbg> b generic_fuzz.c:655
> > Breakpoint 1 at 0x2ca5938: file ../qemu/tests/qtest/fuzz/generic_fuzz.c,
> > line 655.
> > pwndbg> set follow-fork-mode child
> > pwndbg> r
> > Starting program:
> > /home/ubuntu/qemu-lastest-fuzz-test/build_qemu/qemu-fuzz-i386
> > --fuzz-target=generic-fuzz-virtio-vga fuzz-output/
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > [New Thread 0x7ffff0fff700 (LWP 36468)]
> > ==36456==WARNING: ASan doesn't fully support makecontext/swapcontext
> > functions and may produce false positives in some cases!
> > [New Thread 0x7fffd05ff700 (LWP 36469)]
> > INFO: libFuzzer ignores flags that start with '--'
> > INFO: Seed: 1537774383
> > INFO: Loaded 1 modules   (791403 inline 8-bit counters): 791403
> > [0x55555f076000, 0x55555f13736b),
> > INFO: Loaded 1 PC tables (791403 PCs): 791403
> > [0x55555e461b00,0x55555f0751b0),
> > [New Thread 0x7fffcd179700 (LWP 36471)]
> > INFO:     7892 files found in fuzz-output/
> > INFO: -max_len is not provided; libFuzzer will not generate inputs larger
> > than 4096 bytes
> > Matching objects by name virtio*
> > This process will try to fuzz the following MemoryRegions:
> >   * vga ioports remapped[0] (size 20)
> >   * qemu extended regs[0] (size 8)
> >   * virtio-pci-device[0] (size 1000)
> >   * vga[4] (size 1)
> >   * msix-pba[0] (size 8)
> >   * vga[2] (size 10)
> >   * bochs dispi interface[0] (size 16)
> >   * virtio-pci-notify[0] (size 1000)
> >   * vga[0] (size 2)
> >   * bus master[0] (size 0)
> >   * msix-table[0] (size 30)
> >   * vga-lowmem[0] (size 20000)
> >   * virtio-pci-common[0] (size 800)
> >   * virtio-pci-notify-pio[0] (size 4)
> >   * vbe[0] (size 4)
> >   * bus master container[0] (size 0)
> >   * virtio-vga-msix[0] (size 1000)
> >   * vga[3] (size 2)
> >   * virtio-pci-isr[0] (size 800)
> >   * virtio-pci[0] (size 4000)
> >   * vga[1] (size 1)
> > [New process 36472]
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> > [New Thread 0x7fffcd179700 (LWP 36473)]
> > [Switching to Thread 0x7ffff7fdfcc0 (LWP 36472)]
> > ERROR: Could not find ELF base!
> > Thread 2.1 "qemu-fuzz-i386" hit Breakpoint 1, generic_fuzz
> > (s=0x6120000001c0, Data=0x60200008ba30 "\276", Size=0) at
> > ../qemu/tests/qtest/fuzz/generic_fuzz.c:655
> > 655             if (timeout) {
> > ERROR: Could not find ELF base!
> > LEGEND: STACK | HEAP | CODE | DATA | RWX | RODATA
> > ────────────────────────────────────────────────────────────────[
> > REGISTERS ]────────────────────────────────────────────────────────────────
> >  RAX  0x0
> >  RBX  0x7fffffffc680 —▸ 0x55555f076000 (__start___sancov_cntrs) ◂— 0x0
> >  RCX  0x2aaaac0fcc93c0
> >  RDX  0x2aaaac0fcc9380
> >  RDI  0x0
> >  RSI  0x1
> >  R8   0x555560337400 (fuzzer::TPC) ◂— 0x0
> >  R9   0x0
> >  R10  0x1
> >  R11  0x201
> >  R12  0x60200008ba30 ◂— 0xbe
> >  R13  0x555560337400 (fuzzer::TPC) ◂— 0x0
> >  R14  0x0
> >  R15  0x7fffffffca67 ◂— 0x80000
> >  RBP  0x7fffffffc8d0 —▸ 0x7fffffffc9d0 —▸ 0x7fffffffca10 —▸ 0x7fffffffca90
> > —▸ 0x7fffffffcad0 ◂— ...
> >  RSP  0x7fffffffc4c0 ◂— 0x41b58ab3
> >  RIP  0x5555581f9938 (generic_fuzz+616) ◂— mov    eax, dword ptr [rip +
> > 0x5300b82]
> > ─────────────────────────────────────────────────────────────────[ DISASM
> > ]──────────────────────────────────────────────────────────────────
> >  ► 0x5555581f9938 <generic_fuzz+616>    mov    eax, dword ptr [rip +
> > 0x5300b82] <0x55555d4fa4c0>
> >    0x5555581f993e <generic_fuzz+622>    mov    esi, eax
> >    0x5555581f9940 <generic_fuzz+624>    mov    dword ptr [rbx + 0x170], eax
> >    0x5555581f9946 <generic_fuzz+630>    call
> > __sanitizer_cov_trace_const_cmp4 <__sanitizer_cov_trace_const_cmp4>
> >    0x5555581f994b <generic_fuzz+635>    mov    eax, dword ptr [rbx + 0x170]
> >    0x5555581f9951 <generic_fuzz+641>    cmp    eax, 0
> >    0x5555581f9954 <generic_fuzz+644>    jne    generic_fuzz+669
> > <generic_fuzz+669>
> >    0x5555581f995a <generic_fuzz+650>    mov    al, byte ptr [rip +
> > 0x6e7c911] <0x55555f076271>
> >    0x5555581f9960 <generic_fuzz+656>    add    al, 1
> >    0x5555581f9962 <generic_fuzz+658>    mov    byte ptr [rip + 0x6e7c909],
> > al <0x55555f076271>
> >    0x5555581f9968 <generic_fuzz+664>    jmp    generic_fuzz+2134
> > <generic_fuzz+2134>
> > ──────────────────────────────────────────────────────────────[ SOURCE
> > (CODE) ]──────────────────────────────────────────────────────────────
> > In file:
> > /home/ubuntu/qemu-lastest-fuzz-test/qemu/tests/qtest/fuzz/generic_fuzz.c
> >    650          * Sometimes the fuzzer will find inputs that take quite a
> > long time to
> >    651          * process. Often times, these inputs do not result in new
> > coverage.
> >    652          * Even if these inputs might be interesting, they can slow
> > down the
> >    653          * fuzzer, overall. Set a timeout to avoid hurting
> > performance, too much
> >    654          */
> >  ► 655         if (timeout) {
> >    656             struct sigaction sact;
> >    657             struct itimerval timer;
> >    658
> >    659             sigemptyset(&sact.sa_mask);
> >    660             sact.sa_flags   = SA_NODEFER;
> > ──────────────────────────────────────────────────────────────────[ STACK
> > ]──────────────────────────────────────────────────────────────────
> > 00:0000│ rsp  0x7fffffffc4c0 ◂— 0x41b58ab3
> > 01:0008│      0x7fffffffc4c8 —▸ 0x55555c611c33 ◂— xor    esp, dword ptr
> > [rax] /* '3 32 80 7 ops:631 144 152 8 sact:656 368 32 9 timer:657' */
> > 02:0010│      0x7fffffffc4d0 —▸ 0x5555581f96d0 (generic_fuzz) ◂— push   rbp
> > 03:0018│      0x7fffffffc4d8 —▸ 0x5555581fbeb6 (pci_enum+1654) ◂— mov
> >  rdi, qword ptr [rbx + 0xb8]
> > 04:0020│      0x7fffffffc4e0 —▸ 0x5555581fc510 (op_in) ◂— push   rbp
> > 05:0028│      0x7fffffffc4e8 —▸ 0x5555581fce80 (op_out) ◂— push   rbp
> > 06:0030│      0x7fffffffc4f0 —▸ 0x5555581fda30 (op_read) ◂— push   rbp
> > 07:0038│      0x7fffffffc4f8 —▸ 0x5555581fe4d0 (op_write) ◂— push   rbp
> > ────────────────────────────────────────────────────────────────[
> > BACKTRACE ]────────────────────────────────────────────────────────────────
> >  ► f 0     5555581f9938 generic_fuzz+616
> >    f 1     55555820802f LLVMFuzzerTestOneInput+1247
> >    f 2     5555580ecb5b
> >    f 3     5555580eecdb
> >    f 4     5555580ef3b6
> >    f 5     5555580e1e71
> >    f 6     555558108d93 main+35
> >    f 7     7ffff5668bf7 __libc_start_main+231
> >
> > ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
> > pwndbg> c
> > Continuing.
> > [Thread 0x7fffcd179700 (LWP 36473) exited]
> > [Inferior 2 (process 36472) exited normally]
> > INFO: seed corpus: files: 7892 min: 1b max: 4096b total: 3532937b rss:
> > 225Mb
> > pwndbg> #2      pulse  ft: 216 lim: 4 exec/s: 1 rss: 226Mb
> > #4      pulse  cov: 218 ft: 220 corp: 2/2b lim: 4 exec/s: 2 rss: 226Mb
> > #8      pulse  cov: 225 ft: 1268 corp: 6/6b lim: 4 exec/s: 4 rss: 226Mb
> > #16     pulse  cov: 1571 ft: 1803 corp: 12/22b lim: 4 exec/s: 8 rss: 226Mb
> > #32     pulse  cov: 1759 ft: 2257 corp: 26/90b lim: 4 exec/s: 16 rss: 226Mb
> > #64     pulse  cov: 1798 ft: 3062 corp: 50/220b lim: 4 exec/s: 32 rss:
> > 227Mb
> > #128    pulse  cov: 3678 ft: 8040 corp: 105/703b lim: 4 exec/s: 42 rss:
> > 229Mb
> > #256    pulse  cov: 3921 ft: 9791 corp: 204/1716b lim: 4 exec/s: 64 rss:
> > 232Mb
> > i386: wrong value for queue_enable 1818
> > i386: wrong value for queue_enable ffa17a18
> > i386: wrong value for queue_enable 185a55be
> > i386: wrong value for queue_enable 35b5a01
> > i386: wrong value for queue_enable 5a55be00
> > #512    pulse  cov: 5089 ft: 13954 corp: 386/5363b lim: 4 exec/s: 73 rss:
> > 241Mb
> > i386: wrong value for queue_enable 67666369
> > i386: wrong value for queue_enable 20ffff13
> > #1024   pulse  cov: 5236 ft: 18351 corp: 679/16Kb lim: 4 exec/s: 85 rss:
> > 264Mb
> > i386: wrong value for queue_enable 13400200
> > i386: wrong value for queue_enable 943b
> > i386: wrong value for queue_enable 555a
> > i386: wrong value for queue_enable 13400200
> > .........
> 
> 
> 
> Regards,
> Zhiqiang Yan
> 
> Alexander Bulekov <alxndr@bu.edu> 于2021年3月6日周六 下午1:11写道：
> 
> >
> >
> > On 210305 1049, Yan Zhiqiang wrote:
> > >  * Yes, I tried "set follow-fork-mode child" as well, but it's also the
> > > same situation. It will never hit the breakpoint twice(it's strange).
> >
> > Strange - I just tried this and I could set and hit a breakpoint in
> > virtio-gpu code. Maybe the problem is the timeout. Can you try setting
> > the environment variable QEMU_FUZZ_TIMEOUT=0 ?
> >
> > -Alex
> >
> > >
> > >  * Thank you for the method to debug crash, I learned a lot from it.
> > >
> > > Regards,
> > > Zhiqiang Yan
> > >
> > > Alexander Bulekov <alxndr@bu.edu> 于2021年3月4日周四 下午11:24写道：
> > >
> > > > On 210304 1843, Yan Zhiqiang wrote:
> > > > > Hello Alex,
> > > > > I'm learning the fuzz in QEMU recently, I review the fuzz code under
> > > > > /tests/qtest/fuzz which is written by you.
> > > > > I learn a lot from it, but I stuck when I want to debug the fuzz
> > code.
> > > > > I use the gdb with command as follows:
> > > > >
> > > > > >  gdb -q --args ./qemu-fuzz-i386
> > --fuzz-target=generic-fuzz-virtio-vga
> > > > > > ./fuzz-output
> > > > >
> > > > > and set breakpoint at generic_fuzz.c:generic_fuzz.
> > > > > It acctually stop when hit the breakpoint. But the function argument
> > Size
> > > > > is zero and then goto _Exit(0). (try many times but always the same)
> > > >
> > > > Hi Zhiqiang,
> > > > Happy to have more people look at the fuzzing code.
> > > > We run each input in a forked process. Maybe you need to run
> > > > "set follow-fork-mode child" in gdb?
> > > >
> > > > > Then input `c` to continue.
> > > > > However, it never hit the breakpoint after that. Just as the picture
> > > > showed.
> > > > > [image: 1614854239086.jpg]
> > > > > I tried the qtest debug method, but failed.
> > > > > I want to know the real process state to learn the QEMU fuzz and add
> > a
> > > > new
> > > > > fuzzer for QEMU.
> > > > > Could you tell me what's the right method to debug the fuzz code?
> > Thank
> > > > you!
> > > >
> > > > For debugging crash, I usually build QEMU with --enable-sanitizers
> > > > (ASAN), and I convert the crash to a "QTest" reproducer, so it can be
> > > > debugged in a normal build of qemu. There's an RFC that has
> > instructions
> > > > for how to do this:
> > > > https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg06035.html
> > > >
> > > > Let me know if I can provide any more info.
> > > > -Alex
> > > >
> > > > >
> > > > > Regards,
> > > > > Zhiqiang Yan
> > > >
> > > >
> > > >
> >

Re: About libfuzzer in qemu

[Alexander Bulekov]

On 210307 2242, Alexander Bulekov wrote:
> My basic workflow for that is:
> QEMU_FUZZ_TIMEOUT=0 QTEST_LOG=1 FUZZ_SERIALIZE_QTEST=1 \
> ./qemu-fuzz-target \
>        --fuzz-target=generic-fuzz-virtio-vga ./crash-... > /tmp/out
                                                           ^
                                        Oops that should be 2> or &>

> ./scripts/oss-fuzz/reorder_fuzzer_qtest_trace.py /tmp/out > /tmp/repro
> 
> # In /tmp/out find the line "Starting qemu with Arguments:" and copy the
> # args ( without -qtest /dev/null)
> less /tmp/out 
> 
> export QEMU_ARGS="-display none -machine accel=qtest, -m 512M -machine q35 -nodefaults -device virtio-vga"
> 
> # Reproduce the crash on a non-fuzz binary
> ./qemu-system-i386 $QEMU_ARGS -qtest stdio < /tmp/repro