[PATCH 0/4] fuzz: misc patches

[PATCH 0/4] fuzz: misc patches

[Alexander Bulekov]

Hi,
This removes ASan from the default build of the device fuzzer (it can be
enabled using --enable-sanitizers) and adds some content to the
documentation.

This set also contains a respin of this patch:
https://patchew.org/QEMU/20200524143738.23218-1-alxndr@bu.edu/

Thanks
-Alex

Alexander Bulekov (4):
  fuzz: build without AddressSanitizer, by default
  docs/fuzz: describe building fuzzers with enable-sanitizers
  docs/fuzz: add information about useful libFuzzer flags
  docs/fuzz: add instructions for generating a coverage report

 configure              | 10 +++----
 docs/devel/fuzzing.txt | 63 ++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 66 insertions(+), 7 deletions(-)

-- 
2.26.2

[PATCH 1/4] fuzz: build without AddressSanitizer, by default

[Alexander Bulekov]

We already have a nice --enable-sanitizers option to enable
AddressSanitizer. There is no reason to duplicate and force this
functionality in --enable-fuzzing. In the future, if more sanitizers are
added to --enable-sanitizers, it might be impossible to build with both
--enable-sanitizers and --enable-fuzzing, since not all sanitizers are
compatible with libFuzzer. In that case, we could enable ASAN with
--extra-cflags="-fsanitize=address"

Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
---
 configure | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/configure b/configure
index 8a65240d4a..010c0ca479 100755
--- a/configure
+++ b/configure
@@ -6319,7 +6319,7 @@ fi
 # checks for fuzzer
 if test "$fuzzing" = "yes" ; then
   write_c_fuzzer_skeleton
-  if compile_prog "$CPU_CFLAGS -Werror -fsanitize=address,fuzzer" ""; then
+  if compile_prog "$CPU_CFLAGS -Werror -fsanitize=fuzzer" ""; then
       have_fuzzer=yes
   fi
 fi
@@ -7858,11 +7858,11 @@ if test "$sheepdog" = "yes" ; then
 fi
 if test "$fuzzing" = "yes" ; then
   if test "$have_fuzzer" = "yes"; then
-    FUZZ_LDFLAGS=" -fsanitize=address,fuzzer"
-    FUZZ_CFLAGS=" -fsanitize=address,fuzzer"
-    CFLAGS=" -fsanitize=address,fuzzer-no-link"
+    FUZZ_LDFLAGS=" -fsanitize=fuzzer"
+    FUZZ_CFLAGS=" -fsanitize=fuzzer"
+    CFLAGS=" -fsanitize=fuzzer-no-link"
   else
-    error_exit "Your compiler doesn't support -fsanitize=address,fuzzer"
+    error_exit "Your compiler doesn't support -fsanitize=fuzzer"
     exit 1
   fi
 fi
-- 
2.26.2

[PATCH 2/4] docs/fuzz: describe building fuzzers with enable-sanitizers

[Alexander Bulekov]

Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
---
 docs/devel/fuzzing.txt | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/docs/devel/fuzzing.txt b/docs/devel/fuzzing.txt
index 324d2cd92b..382ade974e 100644
--- a/docs/devel/fuzzing.txt
+++ b/docs/devel/fuzzing.txt
@@ -23,9 +23,12 @@ AddressSanitizer mmaps ~20TB of memory, as part of its detection. This results
 in a large page-map, and a much slower fork().
 
 To build the fuzzers, install a recent version of clang:
-Configure with (substitute the clang binaries with the version you installed):
+Configure with (substitute the clang binaries with the version you installed).
+Here, enable-sanitizers, is optional but it allows us to reliably detect bugs
+such as out-of-bounds accesses, use-after-frees, double-frees etc.
 
-    CC=clang-8 CXX=clang++-8 /path/to/configure --enable-fuzzing
+    CC=clang-8 CXX=clang++-8 /path/to/configure --enable-fuzzing \
+                                                --enable-sanitizers
 
 Fuzz targets are built similarly to system/softmmu:
 
-- 
2.26.2

[PATCH 3/4] docs/fuzz: add information about useful libFuzzer flags

[Alexander Bulekov]

Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
---
 docs/devel/fuzzing.txt | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/docs/devel/fuzzing.txt b/docs/devel/fuzzing.txt
index 382ade974e..284d57f8fd 100644
--- a/docs/devel/fuzzing.txt
+++ b/docs/devel/fuzzing.txt
@@ -48,6 +48,43 @@ Information about these is available by passing -help=1
 Now the only thing left to do is wait for the fuzzer to trigger potential
 crashes.
 
+== Useful libFuzzer flags ==
+
+As mentioned above, libFuzzer accepts some arguments. Passing -help=1 will list
+the available arguments. In particular, these arguments might be helpful:
+
+$CORPUS_DIR/ : Specify a directory as the last argument to libFuzzer. libFuzzer
+stores each "interesting" input in this corpus directory. The next time you run
+libFuzzer, it will read all of the inputs from the corpus, and continue fuzzing
+from there. You can also specify multiple directories. libFuzzer loads existing
+inputs from all specified directories, but will only write new ones to the
+first one specified.
+
+-max_len=4096 : specify the maximum byte-length of the inputs libFuzzer will
+generate.
+
+-close_fd_mask={1,2,3} : close, stderr, or both. Useful for targets that
+trigger many debug/error messages, or create output on the serial console.
+
+-jobs=4 -workers=4 : These arguments configure libFuzzer to run 4 fuzzers in
+parallel (4 fuzzing jobs in 4 worker processes). Alternatively, with only
+-jobs=N, libFuzzer automatically spawns a number of workers less than or equal
+to half the available CPU cores. Replace 4 with a number appropriate for your
+machine. Make sure to specify a $CORPUS_DIR, which will allow the parallel
+fuzzers to share information about the interesting inputs they find.
+
+-use_value_profile=1 : For each comparison operation, libFuzzer computes 
+(caller_pc&4095) | (popcnt(Arg1 ^ Arg2) << 12) and places this in the coverage
+table. Useful for targets with "magic" constants. If Arg1 came from the fuzzer's
+input and Arg2 is a magic constant, then each time the Hamming distance
+between Arg1 and Arg2 decreases, libFuzzer adds the input to the corpus.
+
+-shrink=1 : Tries to make elements of the corpus "smaller". Might lead to
+better coverage performance, depending on the target.
+
+Note that libFuzzer's exact behavior will depend on the version of
+clang and libFuzzer used to build the device fuzzers.
+
 == Adding a new fuzzer ==
 Coverage over virtual devices can be improved by adding additional fuzzers.
 Fuzzers are kept in tests/qtest/fuzz/ and should be added to
-- 
2.26.2

[PATCH 4/4] docs/fuzz: add instructions for generating a coverage report

[Alexander Bulekov]

Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
---
 docs/devel/fuzzing.txt | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/docs/devel/fuzzing.txt b/docs/devel/fuzzing.txt
index 284d57f8fd..a9816ffce9 100644
--- a/docs/devel/fuzzing.txt
+++ b/docs/devel/fuzzing.txt
@@ -85,6 +85,25 @@ better coverage performance, depending on the target.
 Note that libFuzzer's exact behavior will depend on the version of
 clang and libFuzzer used to build the device fuzzers.
 
+== Generating Coverage Reports ==
+Code coverage is a crucial metric for evaluating a fuzzer's performance.
+libFuzzer's output provides a "cov: " column that provides a total number of
+unique blocks/edges covered. To examine coverage on a line-by-line basis we
+can use Clang coverage:
+
+ 1. Configure libFuzzer to store a corpus of all interesting inputs (see
+    CORPUS_DIR above)
+ 2. ./configure the QEMU build with:
+    --enable-sanitizers \
+    --extra-cflags="-fprofile-instr-generate -fcoverage-mapping"
+ 3. Re-run the fuzzer. Specify $CORPUS_DIR/* as an argument, telling libfuzzer
+    to execute all of the inputs in $CORPUS_DIR and exit. Once the process
+    exits, you should find a file, "default.profraw" in the working directory.
+ 4. Execute these commands to generate a detailed HTML coverage-report:
+ llvm-profdata merge -output=default.profdata default.profraw
+ llvm-cov show ./path/to/qemu-fuzz-i386 -instr-profile=default.profdata \
+ --format html -output-dir=/path/to/output/report
+
 == Adding a new fuzzer ==
 Coverage over virtual devices can be improved by adding additional fuzzers.
 Fuzzers are kept in tests/qtest/fuzz/ and should be added to
-- 
2.26.2

Re: [PATCH 1/4] fuzz: build without AddressSanitizer, by default

[Philippe Mathieu-Daudé]

On 7/6/20 9:55 PM, Alexander Bulekov wrote:
> We already have a nice --enable-sanitizers option to enable
> AddressSanitizer. There is no reason to duplicate and force this
> functionality in --enable-fuzzing. In the future, if more sanitizers are
> added to --enable-sanitizers, it might be impossible to build with both
> --enable-sanitizers and --enable-fuzzing, since not all sanitizers are
> compatible with libFuzzer. In that case, we could enable ASAN with
> --extra-cflags="-fsanitize=address"
> 
> Signed-off-by: Alexander Bulekov <alxndr@bu.edu>

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> ---
>  configure | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/configure b/configure
> index 8a65240d4a..010c0ca479 100755
> --- a/configure
> +++ b/configure
> @@ -6319,7 +6319,7 @@ fi
>  # checks for fuzzer
>  if test "$fuzzing" = "yes" ; then
>    write_c_fuzzer_skeleton
> -  if compile_prog "$CPU_CFLAGS -Werror -fsanitize=address,fuzzer" ""; then
> +  if compile_prog "$CPU_CFLAGS -Werror -fsanitize=fuzzer" ""; then
>        have_fuzzer=yes
>    fi
>  fi
> @@ -7858,11 +7858,11 @@ if test "$sheepdog" = "yes" ; then
>  fi
>  if test "$fuzzing" = "yes" ; then
>    if test "$have_fuzzer" = "yes"; then
> -    FUZZ_LDFLAGS=" -fsanitize=address,fuzzer"
> -    FUZZ_CFLAGS=" -fsanitize=address,fuzzer"
> -    CFLAGS=" -fsanitize=address,fuzzer-no-link"
> +    FUZZ_LDFLAGS=" -fsanitize=fuzzer"
> +    FUZZ_CFLAGS=" -fsanitize=fuzzer"
> +    CFLAGS=" -fsanitize=fuzzer-no-link"
>    else
> -    error_exit "Your compiler doesn't support -fsanitize=address,fuzzer"
> +    error_exit "Your compiler doesn't support -fsanitize=fuzzer"
>      exit 1
>    fi
>  fi
> 

Re: [PATCH 2/4] docs/fuzz: describe building fuzzers with enable-sanitizers

[Philippe Mathieu-Daudé]

On 7/6/20 9:55 PM, Alexander Bulekov wrote:
> Signed-off-by: Alexander Bulekov <alxndr@bu.edu>

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> ---
>  docs/devel/fuzzing.txt | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/docs/devel/fuzzing.txt b/docs/devel/fuzzing.txt
> index 324d2cd92b..382ade974e 100644
> --- a/docs/devel/fuzzing.txt
> +++ b/docs/devel/fuzzing.txt
> @@ -23,9 +23,12 @@ AddressSanitizer mmaps ~20TB of memory, as part of its detection. This results
>  in a large page-map, and a much slower fork().
>  
>  To build the fuzzers, install a recent version of clang:
> -Configure with (substitute the clang binaries with the version you installed):
> +Configure with (substitute the clang binaries with the version you installed).
> +Here, enable-sanitizers, is optional but it allows us to reliably detect bugs
> +such as out-of-bounds accesses, use-after-frees, double-frees etc.
>  
> -    CC=clang-8 CXX=clang++-8 /path/to/configure --enable-fuzzing
> +    CC=clang-8 CXX=clang++-8 /path/to/configure --enable-fuzzing \
> +                                                --enable-sanitizers
>  
>  Fuzz targets are built similarly to system/softmmu:
>  
> 

Re: [PATCH 4/4] docs/fuzz: add instructions for generating a coverage report

[Alexander Bulekov]

On 200706 1555, Alexander Bulekov wrote:
> Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
> ---
>  docs/devel/fuzzing.txt | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/docs/devel/fuzzing.txt b/docs/devel/fuzzing.txt
> index 284d57f8fd..a9816ffce9 100644
> --- a/docs/devel/fuzzing.txt
> +++ b/docs/devel/fuzzing.txt
> @@ -85,6 +85,25 @@ better coverage performance, depending on the target.
>  Note that libFuzzer's exact behavior will depend on the version of
>  clang and libFuzzer used to build the device fuzzers.
>  
> +== Generating Coverage Reports ==
> +Code coverage is a crucial metric for evaluating a fuzzer's performance.
> +libFuzzer's output provides a "cov: " column that provides a total number of
> +unique blocks/edges covered. To examine coverage on a line-by-line basis we
> +can use Clang coverage:
> +
> + 1. Configure libFuzzer to store a corpus of all interesting inputs (see
> +    CORPUS_DIR above)
> + 2. ./configure the QEMU build with:
> +    --enable-sanitizers \
Oops... that should be --enable-fuzzing \

> +    --extra-cflags="-fprofile-instr-generate -fcoverage-mapping"
> + 3. Re-run the fuzzer. Specify $CORPUS_DIR/* as an argument, telling libfuzzer
> +    to execute all of the inputs in $CORPUS_DIR and exit. Once the process
> +    exits, you should find a file, "default.profraw" in the working directory.
> + 4. Execute these commands to generate a detailed HTML coverage-report:
> + llvm-profdata merge -output=default.profdata default.profraw
> + llvm-cov show ./path/to/qemu-fuzz-i386 -instr-profile=default.profdata \
> + --format html -output-dir=/path/to/output/report
> +
>  == Adding a new fuzzer ==
>  Coverage over virtual devices can be improved by adding additional fuzzers.
>  Fuzzers are kept in tests/qtest/fuzz/ and should be added to
> -- 
> 2.26.2
> 

Re: [PATCH 1/4] fuzz: build without AddressSanitizer, by default

[Thomas Huth]

On 06/07/2020 21.55, Alexander Bulekov wrote:
> We already have a nice --enable-sanitizers option to enable
> AddressSanitizer. There is no reason to duplicate and force this
> functionality in --enable-fuzzing. In the future, if more sanitizers are
> added to --enable-sanitizers, it might be impossible to build with both
> --enable-sanitizers and --enable-fuzzing, since not all sanitizers are
> compatible with libFuzzer. In that case, we could enable ASAN with
> --extra-cflags="-fsanitize=address"
> 
> Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
> ---
>  configure | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/configure b/configure
> index 8a65240d4a..010c0ca479 100755
> --- a/configure
> +++ b/configure
> @@ -6319,7 +6319,7 @@ fi
>  # checks for fuzzer
>  if test "$fuzzing" = "yes" ; then
>    write_c_fuzzer_skeleton
> -  if compile_prog "$CPU_CFLAGS -Werror -fsanitize=address,fuzzer" ""; then
> +  if compile_prog "$CPU_CFLAGS -Werror -fsanitize=fuzzer" ""; then
>        have_fuzzer=yes
>    fi
>  fi
> @@ -7858,11 +7858,11 @@ if test "$sheepdog" = "yes" ; then
>  fi
>  if test "$fuzzing" = "yes" ; then
>    if test "$have_fuzzer" = "yes"; then
> -    FUZZ_LDFLAGS=" -fsanitize=address,fuzzer"
> -    FUZZ_CFLAGS=" -fsanitize=address,fuzzer"
> -    CFLAGS=" -fsanitize=address,fuzzer-no-link"
> +    FUZZ_LDFLAGS=" -fsanitize=fuzzer"
> +    FUZZ_CFLAGS=" -fsanitize=fuzzer"
> +    CFLAGS=" -fsanitize=fuzzer-no-link"
>    else
> -    error_exit "Your compiler doesn't support -fsanitize=address,fuzzer"
> +    error_exit "Your compiler doesn't support -fsanitize=fuzzer"
>      exit 1
>    fi
>  fi
> 

Reviewed-by: Thomas Huth <thuth@redhat.com>

I can add the missing $CFLAGS from commit 0ab6c2384ccae89 when picking
up the patch.

 Thomas

Re: [PATCH 2/4] docs/fuzz: describe building fuzzers with enable-sanitizers

[Thomas Huth]

On 06/07/2020 21.55, Alexander Bulekov wrote:
> Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
> ---
>  docs/devel/fuzzing.txt | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/docs/devel/fuzzing.txt b/docs/devel/fuzzing.txt
> index 324d2cd92b..382ade974e 100644
> --- a/docs/devel/fuzzing.txt
> +++ b/docs/devel/fuzzing.txt
> @@ -23,9 +23,12 @@ AddressSanitizer mmaps ~20TB of memory, as part of its detection. This results
>  in a large page-map, and a much slower fork().
>  
>  To build the fuzzers, install a recent version of clang:
> -Configure with (substitute the clang binaries with the version you installed):
> +Configure with (substitute the clang binaries with the version you installed).
> +Here, enable-sanitizers, is optional but it allows us to reliably detect bugs
> +such as out-of-bounds accesses, use-after-frees, double-frees etc.
>  
> -    CC=clang-8 CXX=clang++-8 /path/to/configure --enable-fuzzing
> +    CC=clang-8 CXX=clang++-8 /path/to/configure --enable-fuzzing \
> +                                                --enable-sanitizers
>  
>  Fuzz targets are built similarly to system/softmmu:

In the future, we should maybe use "CC=clang CXX=clang++" instead, since
version numbers soon get old...

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH 4/4] docs/fuzz: add instructions for generating a coverage report

[Thomas Huth]

On 07/07/2020 06.41, Alexander Bulekov wrote:
> On 200706 1555, Alexander Bulekov wrote:
>> Signed-off-by: Alexander Bulekov <alxndr@bu.edu>
>> ---
>>  docs/devel/fuzzing.txt | 19 +++++++++++++++++++
>>  1 file changed, 19 insertions(+)
>>
>> diff --git a/docs/devel/fuzzing.txt b/docs/devel/fuzzing.txt
>> index 284d57f8fd..a9816ffce9 100644
>> --- a/docs/devel/fuzzing.txt
>> +++ b/docs/devel/fuzzing.txt
>> @@ -85,6 +85,25 @@ better coverage performance, depending on the target.
>>  Note that libFuzzer's exact behavior will depend on the version of
>>  clang and libFuzzer used to build the device fuzzers.
>>  
>> +== Generating Coverage Reports ==
>> +Code coverage is a crucial metric for evaluating a fuzzer's performance.
>> +libFuzzer's output provides a "cov: " column that provides a total number of
>> +unique blocks/edges covered. To examine coverage on a line-by-line basis we
>> +can use Clang coverage:
>> +
>> + 1. Configure libFuzzer to store a corpus of all interesting inputs (see
>> +    CORPUS_DIR above)
>> + 2. ./configure the QEMU build with:
>> +    --enable-sanitizers \
> Oops... that should be --enable-fuzzing \

I've pushed it with the fix applied to my qtest-next branch:

 https://gitlab.com/huth/qemu/-/commits/qtest-next/

  Thomas