[PATCH v1 0/2] test-must-fail-sigpipe

[PATCH v1 0/2] test-must-fail-sigpipe

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

As suggested by Peff I made "test-must-fail-sigpipe" a seperate topic.

diff to (former) Travis CI v7 topic:
* added helper function "list_contains" provided by Peff (thanks!)
* fix return value in "test_must_fail: command succeeded" (thanks Ramsay)
* use "-eq" instead of "=" (thanks Ramsay)
* accept SIGPIPE exit for t5504 "push with receive.fsckobjects"

Thanks,
Lars

Lars Schneider (2):
  implement test_might_fail using a refactored test_must_fail
  add "ok=sigpipe" to test_must_fail and use it to fix flaky tests

 t/t5504-fetch-receive-strict.sh |  5 ++---
 t/t5516-fetch-push.sh           |  6 ++---
 t/test-lib-functions.sh         | 50 +++++++++++++++++++++++++++++------------
 3 files changed, 41 insertions(+), 20 deletions(-)

--
2.5.1

[PATCH v1 1/2] implement test_might_fail using a refactored test_must_fail

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

Add an (optional) first parameter "ok=<special case>" to test_must_fail
and return success for "<special case>". Add "success" as
"<special case>" and use it to implement "test_might_fail". This removes
redundancies in test-lib-function.sh.

You can pass multiple <special case> arguments divided by comma (e.g.
"test_must_fail ok=success,something")

Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 t/test-lib-functions.sh | 47 +++++++++++++++++++++++++++++++++--------------
 1 file changed, 33 insertions(+), 14 deletions(-)

diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index 73e37a1..94c449a 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -569,6 +569,21 @@ test_line_count () {
 	fi
 }
 
+# Returns success if a comma separated string of keywords ($1) contains a
+# given keyword ($2).
+# Examples:
+# `list_contains "foo,bar" bar` returns 0
+# `list_contains "foo" bar` returns 1
+
+list_contains () {
+	case ",$1," in
+	*,$2,*)
+		return 0
+		;;
+	esac
+	return 1
+}
+
 # This is not among top-level (test_expect_success | test_expect_failure)
 # but is a prefix that can be used in the test script, like:
 #
@@ -582,18 +597,31 @@ test_line_count () {
 # the failure could be due to a segv.  We want a controlled failure.
 
 test_must_fail () {
+	case "$1" in
+	ok=*)
+		_test_ok=${1#ok=}
+		shift
+		;;
+	*)
+		_test_ok=
+		;;
+	esac
 	"$@"
 	exit_code=$?
-	if test $exit_code = 0; then
+	if ! list_contains "$_test_ok" success && test "$exit_code" -eq 0
+	then
 		echo >&2 "test_must_fail: command succeeded: $*"
 		return 1
-	elif test $exit_code -gt 129 && test $exit_code -le 192; then
+	elif test $exit_code -gt 129 && test $exit_code -le 192
+	then
 		echo >&2 "test_must_fail: died by signal: $*"
 		return 1
-	elif test $exit_code = 127; then
+	elif test $exit_code -eq 127
+	then
 		echo >&2 "test_must_fail: command not found: $*"
 		return 1
-	elif test $exit_code = 126; then
+	elif test $exit_code -eq 126
+	then
 		echo >&2 "test_must_fail: valgrind error: $*"
 		return 1
 	fi
@@ -612,16 +640,7 @@ test_must_fail () {
 # because we want to notice if it fails due to segv.
 
 test_might_fail () {
-	"$@"
-	exit_code=$?
-	if test $exit_code -gt 129 && test $exit_code -le 192; then
-		echo >&2 "test_might_fail: died by signal: $*"
-		return 1
-	elif test $exit_code = 127; then
-		echo >&2 "test_might_fail: command not found: $*"
-		return 1
-	fi
-	return 0
+	test_must_fail ok=success "$@"
 }
 
 # Similar to test_must_fail and test_might_fail, but check that a
-- 
2.5.1

Re: [PATCH v1 1/2] implement test_might_fail using a refactored test_must_fail

[Ramsay Jones]

On 27/11/15 09:15, larsxschneider@gmail.com wrote:
> From: Lars Schneider <larsxschneider@gmail.com>
> 
> Add an (optional) first parameter "ok=<special case>" to test_must_fail
> and return success for "<special case>". Add "success" as
> "<special case>" and use it to implement "test_might_fail". This removes
> redundancies in test-lib-function.sh.
> 
> You can pass multiple <special case> arguments divided by comma (e.g.
> "test_must_fail ok=success,something")
> 
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Jeff King <peff@peff.net>
> Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
> ---
>  t/test-lib-functions.sh | 47 +++++++++++++++++++++++++++++++++--------------
>  1 file changed, 33 insertions(+), 14 deletions(-)
> 
> diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
> index 73e37a1..94c449a 100644
> --- a/t/test-lib-functions.sh
> +++ b/t/test-lib-functions.sh
> @@ -569,6 +569,21 @@ test_line_count () {
>  	fi
>  }
>  
> +# Returns success if a comma separated string of keywords ($1) contains a
> +# given keyword ($2).
> +# Examples:
> +# `list_contains "foo,bar" bar` returns 0
> +# `list_contains "foo" bar` returns 1
> +
> +list_contains () {
> +	case ",$1," in
> +	*,$2,*)
> +		return 0
> +		;;
> +	esac
> +	return 1
> +}
> +
>  # This is not among top-level (test_expect_success | test_expect_failure)
>  # but is a prefix that can be used in the test script, like:
>  #
> @@ -582,18 +597,31 @@ test_line_count () {
>  # the failure could be due to a segv.  We want a controlled failure.
>  
>  test_must_fail () {
> +	case "$1" in
> +	ok=*)
> +		_test_ok=${1#ok=}
> +		shift
> +		;;
> +	*)
> +		_test_ok=
> +		;;
> +	esac
>  	"$@"
>  	exit_code=$?
> -	if test $exit_code = 0; then
> +	if ! list_contains "$_test_ok" success && test "$exit_code" -eq 0
> +	then

minor nit:

I would prefer this was 'if test $exit_code -eq 0 && ! list_contains ...'

ie. the test on exit code comes first (and no need for the double quotes).
The whole if..elif.. chain is about testing the exit code, with a couple
of exceptions ...

The same comment applies to the second patch with exit code 141/SIGPIPE.

ATB,
Ramsay Jones

Re: [PATCH v1 1/2] implement test_might_fail using a refactored test_must_fail

[Jeff King]

On Fri, Nov 27, 2015 at 12:37:38PM +0000, Ramsay Jones wrote:

> > -	if test $exit_code = 0; then
> > +	if ! list_contains "$_test_ok" success && test "$exit_code" -eq 0
> > +	then
> 
> minor nit:
> 
> I would prefer this was 'if test $exit_code -eq 0 && ! list_contains ...'
> 
> ie. the test on exit code comes first (and no need for the double quotes).
> The whole if..elif.. chain is about testing the exit code, with a couple
> of exceptions ...

I agree, it makes the logic a bit easier to follow. I've tweaked it
while applying; no need to re-send.

> The same comment applies to the second patch with exit code 141/SIGPIPE.

Got that one, too.

-Peff

[PATCH v1 2/2] add "ok=sigpipe" to test_must_fail and use it to fix flaky tests

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

t5516 "75 - deny fetch unreachable SHA1, allowtipsha1inwant=true" is
flaky in the following case:
1. remote upload-pack finds out "not our ref"
2. remote sends a response and closes the pipe
3. fetch-pack still tries to write commands to the remote upload-pack
4. write call in wrapper.c dies with SIGPIPE

t5504 "9 - push with transfer.fsckobjects" is flaky, too, and returns
SIGPIPE once in a while. I had to remove the final "To dst..." output
check because there is no output if the process dies with SIGPUPE.

Accept such a death-with-sigpipe also as OK when we are expecting a
failure.

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 t/t5504-fetch-receive-strict.sh | 5 ++---
 t/t5516-fetch-push.sh           | 6 +++---
 t/test-lib-functions.sh         | 3 +++
 3 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/t/t5504-fetch-receive-strict.sh b/t/t5504-fetch-receive-strict.sh
index 44f3d5f..89224ed 100755
--- a/t/t5504-fetch-receive-strict.sh
+++ b/t/t5504-fetch-receive-strict.sh
@@ -100,7 +100,7 @@ test_expect_success 'push with receive.fsckobjects' '
 		git config receive.fsckobjects true &&
 		git config transfer.fsckobjects false
 	) &&
-	test_must_fail git push --porcelain dst master:refs/heads/test >act &&
+	test_must_fail ok=sigpipe git push --porcelain dst master:refs/heads/test >act &&
 	test_cmp exp act
 '
 
@@ -111,8 +111,7 @@ test_expect_success 'push with transfer.fsckobjects' '
 		cd dst &&
 		git config transfer.fsckobjects true
 	) &&
-	test_must_fail git push --porcelain dst master:refs/heads/test >act &&
-	test_cmp exp act
+	test_must_fail ok=sigpipe git push --porcelain dst master:refs/heads/test >act
 '
 
 cat >bogus-commit <<\EOF
diff --git a/t/t5516-fetch-push.sh b/t/t5516-fetch-push.sh
index ec22c98..0a87e19 100755
--- a/t/t5516-fetch-push.sh
+++ b/t/t5516-fetch-push.sh
@@ -1162,15 +1162,15 @@ do
 		mk_empty shallow &&
 		(
 			cd shallow &&
-			test_must_fail git fetch ../testrepo/.git $SHA1_3 &&
-			test_must_fail git fetch ../testrepo/.git $SHA1_1 &&
+			test_must_fail ok=sigpipe git fetch ../testrepo/.git $SHA1_3 &&
+			test_must_fail ok=sigpipe git fetch ../testrepo/.git $SHA1_1 &&
 			git --git-dir=../testrepo/.git config uploadpack.allowreachablesha1inwant true &&
 			git fetch ../testrepo/.git $SHA1_1 &&
 			git cat-file commit $SHA1_1 &&
 			test_must_fail git cat-file commit $SHA1_2 &&
 			git fetch ../testrepo/.git $SHA1_2 &&
 			git cat-file commit $SHA1_2 &&
-			test_must_fail git fetch ../testrepo/.git $SHA1_3
+			test_must_fail ok=sigpipe git fetch ../testrepo/.git $SHA1_3
 		)
 	'
 done
diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index 94c449a..06d3fcb 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -612,6 +612,9 @@ test_must_fail () {
 	then
 		echo >&2 "test_must_fail: command succeeded: $*"
 		return 1
+	elif list_contains "$_test_ok" sigpipe && test "$exit_code" -eq 141
+	then
+		return 0
 	elif test $exit_code -gt 129 && test $exit_code -le 192
 	then
 		echo >&2 "test_must_fail: died by signal: $*"
-- 
2.5.1

Re: [PATCH v1 2/2] add "ok=sigpipe" to test_must_fail and use it to fix flaky tests

[Jeff King]

On Fri, Nov 27, 2015 at 10:15:14AM +0100, larsxschneider@gmail.com wrote:

> From: Lars Schneider <larsxschneider@gmail.com>
> 
> t5516 "75 - deny fetch unreachable SHA1, allowtipsha1inwant=true" is
> flaky in the following case:
> 1. remote upload-pack finds out "not our ref"
> 2. remote sends a response and closes the pipe
> 3. fetch-pack still tries to write commands to the remote upload-pack
> 4. write call in wrapper.c dies with SIGPIPE
> 
> t5504 "9 - push with transfer.fsckobjects" is flaky, too, and returns
> SIGPIPE once in a while. I had to remove the final "To dst..." output
> check because there is no output if the process dies with SIGPUPE.

s/PUPE/PIPE/ :)

I think it would be nice for future readers to understand a bit better
_why_ this is flaky, and why the fix is to the test suite and not to git
itself. I added this paragraph in between the two above:

    The test is flaky because the sending fetch-pack may or may not have
    finished writing its output by step (3). If it did, then we see a
    closed pipe on the next read() call. If it didn't, then we get the
    SIGPIPE from step (4) above. Both are fine, but the latter fools
    test_must_fail.

-Peff

Re: [PATCH v1 2/2] add "ok=sigpipe" to test_must_fail and use it to fix flaky tests

[Lars Schneider]

On 28 Nov 2015, at 18:10, Jeff King <peff@peff.net> wrote:

> On Fri, Nov 27, 2015 at 10:15:14AM +0100, larsxschneider@gmail.com wrote:
> 
>> From: Lars Schneider <larsxschneider@gmail.com>
>> 
>> t5516 "75 - deny fetch unreachable SHA1, allowtipsha1inwant=true" is
>> flaky in the following case:
>> 1. remote upload-pack finds out "not our ref"
>> 2. remote sends a response and closes the pipe
>> 3. fetch-pack still tries to write commands to the remote upload-pack
>> 4. write call in wrapper.c dies with SIGPIPE
>> 
>> t5504 "9 - push with transfer.fsckobjects" is flaky, too, and returns
>> SIGPIPE once in a while. I had to remove the final "To dst..." output
>> check because there is no output if the process dies with SIGPUPE.
> 
> s/PUPE/PIPE/ :)
> 
> I think it would be nice for future readers to understand a bit better
> _why_ this is flaky, and why the fix is to the test suite and not to git
> itself. I added this paragraph in between the two above:
> 
>    The test is flaky because the sending fetch-pack may or may not have
>    finished writing its output by step (3). If it did, then we see a
>    closed pipe on the next read() call. If it didn't, then we get the
>    SIGPIPE from step (4) above. Both are fine, but the latter fools
>    test_must_fail.
> 
Sounds good! Thank you :-)

- Lars