Re: [RFC][PATCH] index-pack: add testcases found using AFL

[Vegard Nossum <vegard.nossum@oracle.com>]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

On 12/03/2017 13:32, Jeff King wrote:
> If people really _do_ care about coverage, arguably the AFL tests are a
> pollution of that concept. Because they are running the code, but doing
> a very perfunctory job of testing it. IOW, our coverage of "code that
> doesn't segfault or trigger ASAN" is improved, but our coverage of "code
> that has been tested to be correct" is not (and since the tests are
> lumped together, it's hard to get anything but one number).

It wouldn't be hard to separate out the testcases found by fuzzing
I've attached a patch that does just that -- none of the new testcases
are run unless you pass -f/--fuzzing in GIT_TEST_OPTS.

$ make -C t GIT_TEST_OPTS="--run=34" t5300-pack-object.sh
make: Entering directory '/home/vegard/git/git/t'
*** t5300-pack-object.sh ***
[...]
ok 34 # skip index-pack edge coverage (missing FUZZING)
[...]

$ make -C t GIT_TEST_OPTS="--run=34 -f" t5300-pack-object.sh
make: Entering directory '/home/vegard/git/git/t'
*** t5300-pack-object.sh ***
[...]
ok 34 - index-pack edge coverage
[...]

I assume automatic testing like e.g. Travis would want to enable this.

Would that help at all?


Vegard

[-- Attachment #2: 0001-test-lib-add-fuzzing-option.patch --]
[-- Type: text/x-patch, Size: 2533 bytes --]

From 04446ce562eee129588f2c92c4eef2c82ed4bb4f Mon Sep 17 00:00:00 2001
From: Vegard Nossum <vegard.nossum@oracle.com>
Date: Sun, 12 Mar 2017 14:35:25 +0100
Subject: [PATCH] test-lib: add --fuzzing option

From t/README:

	This causes additional testcases found by fuzzing to be run,
	for more exhaustive testing. Please note that these testcases
	have not been vetted for correctness, but they may uncover
	bugs introduced in code paths which are not otherwise run
	in other tests.

The -f/--fuzzing/FUZZING name is up for discussion, I just couldn't think
of anything more descriptive.
---
 t/README               | 8 ++++++++
 t/t5300-pack-object.sh | 2 +-
 t/test-lib.sh          | 6 ++++++
 3 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/t/README b/t/README
index 4982d1c52..2c56567b1 100644
--- a/t/README
+++ b/t/README
@@ -110,6 +110,14 @@ appropriately before running "make".
 	This causes additional long-running tests to be run (where
 	available), for more exhaustive testing.
 
+-f::
+--fuzzing::
+	This causes additional testcases found by fuzzing to be run,
+	for more exhaustive testing. Please note that these testcases
+	have not been vetted for correctness, but they may uncover
+	bugs introduced in code paths which are not otherwise run
+	in other tests.
+
 -r::
 --run=<test-selector>::
 	Run only the subset of tests indicated by
diff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh
index 19e02ffc2..f58d0d4bf 100755
--- a/t/t5300-pack-object.sh
+++ b/t/t5300-pack-object.sh
@@ -422,7 +422,7 @@ test_expect_success 'index-pack <pack> works in non-repo' '
 '
 
 # These pack files were generated using AFL
-test_expect_success 'index-pack edge coverage' '
+test_expect_success FUZZING 'index-pack edge coverage' '
 	for pack in "$TEST_DIRECTORY"/t5300/*.pack
 	do
 		rm -rf "${pack%.pack}.idx" &&
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 86d77c16d..35df2bd6c 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -209,6 +209,8 @@ do
 		immediate=t; shift ;;
 	-l|--l|--lo|--lon|--long|--long-|--long-t|--long-te|--long-tes|--long-test|--long-tests)
 		GIT_TEST_LONG=t; export GIT_TEST_LONG; shift ;;
+	-f|--f|--fuzzing)
+		GIT_TEST_FUZZING=t; export GIT_TEST_FUZZING; shift ;;
 	-r)
 		shift; test "$#" -ne 0 || {
 			echo 'error: -r requires an argument' >&2;
@@ -1098,6 +1100,10 @@ test_lazy_prereq EXPENSIVE '
 	test -n "$GIT_TEST_LONG"
 '
 
+test_lazy_prereq FUZZING '
+	test -n "$GIT_TEST_FUZZING"
+'
+
 test_lazy_prereq USR_BIN_TIME '
 	test -x /usr/bin/time
 '
-- 
2.12.0.rc0