Re: [RFC][PATCH] index-pack: add testcases found using AFL

[Vegard Nossum <vegard.nossum@oracle.com>]

On 10/03/2017 16:15, Vegard Nossum wrote:
> I've used AFL to generate a corpus of pack files that maximises the edge
> coverage for 'git index-pack'.
>
> This is a supplement to (and not a replacement for) the regular test cases
> where we know exactly what each test is checking for. These testcases are
> more useful for avoiding regressions in edge cases or as a starting point
> for future fuzzing efforts.
>
> To see the output of running 'git index-pack' on each file, you can do
> something like this:
>
>   make -C t GIT_TEST_OPTS="--run=34 --verbose" t5300-pack-object.sh
>
> I observe the following coverage changes (for t5300 only):
>
>   path                  old%  new%    pp
>   ----------------------------------------
>   builtin/index-pack.c  74.3  76.6   2.3
>   pack-write.c          79.8  80.4    .6
>   patch-delta.c         67.4  81.4  14.0
>   usage.c               26.6  35.5   8.9
>   wrapper.c             42.0  46.1   4.1
>   zlib.c                58.7  64.1   5.4

And if you add this simple patch on top (sorry, I didn't think of it
until after I'd sent the previous e-mail):

diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh
index 19e02ffc2..db705ba5c 100755
--- a/t/t5300-pack-object.sh
+++ b/t/t5300-pack-object.sh
@@ -425,8 +425,10 @@ test_expect_success 'index-pack <pack> works in 
non-repo' '
  test_expect_success 'index-pack edge coverage' '
         for pack in "$TEST_DIRECTORY"/t5300/*.pack
         do
-               rm -rf "${pack%.pack}.idx" &&
-               test_might_fail git index-pack $pack
+               rm -rf "${pack%.pack}.idx" tmp.pack tmp.idx &&
+               test_might_fail git index-pack $pack &&
+               test_might_fail git index-pack --strict $pack &&
+               test_might_fail git index-pack --stdin --fix-thin 
tmp.pack < $pack
         done
  '


you get this change to the coverage profile instead:

path                  old%  new%    pp
----------------------------------------

alloc.c               58.1  67.4   9.3
builtin/index-pack.c  74.3  80.7   6.4
commit.c              13.9  17.4   3.5
date.c                 3.5   4.2    .7
fsck.c                15.7  33.7  18.0
object.c              56.0  58.7   2.7
pack-write.c          79.8  81.4   1.6
patch-delta.c         67.4  81.4  14.0
path.c                31.6  32.1    .5
sha1_file.c           48.9  49.6    .7
tag.c                  3.7  16.8  13.1
tree.c                36.6  37.5    .9
usage.c               26.6  35.5   8.9
wrapper.c             42.0  46.1   4.1
zlib.c                58.7  64.1   5.4

Of course, it's likely some of those gains can be found in other
testcases outside t5300 -- also, coverage isn't everything. Still seems
like a nice gain with very little effort.


Vegard