[PATCH 0/3] LSan quality of life improvements

[PATCH 0/3] LSan quality of life improvements

[Jeff King]

I was fixing some leaks the other day and came up with a few changes
that made the process a bit less painful, especially when the leaks are
hidden in sub-processes (which is most of them when you are digging into
http push/fetch, as I was).

I hope we're not too far off from a world where leaks are something that
pop up in your newly written code, and you're not slogging through
existing test scripts. But until then, I hope these might help others.

If you want to see the before/after, try:

  cd t
  ./t5550-http-fetch-dumb.sh -i

before and after this series. Before you get no leaks reported to stdout
with "-i", and way too many without it. After, you get the leaks for the
first test that generates them.

  [1/3]: test-lib: stop showing old leak logs
  [2/3]: test-lib: show leak-sanitizer logs on --immediate failure
  [3/3]: test-lib: check for leak logs after every test

 t/test-lib-functions.sh |  3 ++-
 t/test-lib.sh           | 41 ++++++-----------------------------------
 2 files changed, 8 insertions(+), 36 deletions(-)

-Peff

[PATCH 1/3] test-lib: stop showing old leak logs

[Jeff King]

We ask LSan to record the logs of all leaks in test-results/, which is
useful for finding leaks that didn't trigger a test failure.

We don't clean out the leak/ directory for each test before running it,
though. Instead, we count the number of files it has, and complain only
if we ended up with more when the script finishes. So we shouldn't
trigger any output if you've made a script leak free. But if you simply
_reduced_ the number of leaks, then there is an annoying outcome: we do
not record which logs were from this run and which were from previous
ones. So when we dump them to stdout, you get a mess of
possibly-outdated leaks. This is very confusing when you are in an
edit-compile-test cycle trying to fix leaks.

The instructions do note that you should "rm -rf test-results/" if you
want to avoid this. But I'm having trouble seeing how this cumulative
count could ever be useful. It is not even counting the number of leaks,
but rather the number of processes with at least one leak!

So let's just blow away the per-test leak/ directory before running. We
already overwrite the ".out" file in test-results/ in the same way, so
this is following that pattern.

Running "make test" isn't affected by this, since it blows away all of
test-results/ already. This only comes up when you are iterating on a
single script that you're running manually.

Signed-off-by: Jeff King <peff@peff.net>
---
 t/test-lib.sh | 35 ++---------------------------------
 1 file changed, 2 insertions(+), 33 deletions(-)

diff --git a/t/test-lib.sh b/t/test-lib.sh
index e718efe4c6..7d4471fbc5 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -322,7 +322,6 @@ TEST_RESULTS_SAN_FILE_PFX=trace
 TEST_RESULTS_SAN_DIR_SFX=leak
 TEST_RESULTS_SAN_FILE=
 TEST_RESULTS_SAN_DIR="$TEST_RESULTS_DIR/$TEST_NAME.$TEST_RESULTS_SAN_DIR_SFX"
-TEST_RESULTS_SAN_DIR_NR_LEAKS_STARTUP=
 TRASH_DIRECTORY="trash directory.$TEST_NAME$TEST_STRESS_JOB_SFX"
 test -n "$root" && TRASH_DIRECTORY="$root/$TRASH_DIRECTORY"
 case "$TRASH_DIRECTORY" in
@@ -1215,42 +1214,15 @@ test_atexit_handler () {
 	teardown_malloc_check
 }
 
-sanitize_leak_log_message_ () {
-	local new="$1" &&
-	local old="$2" &&
-	local file="$3" &&
-
-	printf "With SANITIZE=leak at exit we have %d leak logs, but started with %d
-
-This means that we have a blindspot where git is leaking but we're
-losing the exit code somewhere, or not propagating it appropriately
-upwards!
-
-See the logs at \"%s.*\";
-those logs are reproduced below." \
-	       "$new" "$old" "$file"
-}
-
 check_test_results_san_file_ () {
 	if test -z "$TEST_RESULTS_SAN_FILE"
 	then
 		return
 	fi &&
-	local old="$TEST_RESULTS_SAN_DIR_NR_LEAKS_STARTUP" &&
-	local new="$(nr_san_dir_leaks_)" &&
-
-	if test $new -le $old
+	if test "$(nr_san_dir_leaks_)" = 0
 	then
 		return
 	fi &&
-	local out="$(sanitize_leak_log_message_ "$new" "$old" "$TEST_RESULTS_SAN_FILE")" &&
-	say_color error "$out" &&
-	if test "$old" != 0
-	then
-		echo &&
-		say_color error "The logs include output from past runs to avoid" &&
-		say_color error "that remove 'test-results' between runs."
-	fi &&
 	say_color error "$(cat "$TEST_RESULTS_SAN_FILE".*)" &&
 
 	if test -n "$passes_sanitize_leak" && test "$test_failure" = 0
@@ -1586,16 +1558,13 @@ then
 		test_done
 	fi
 
+	rm -rf "$TEST_RESULTS_SAN_DIR"
 	if ! mkdir -p "$TEST_RESULTS_SAN_DIR"
 	then
 		BAIL_OUT "cannot create $TEST_RESULTS_SAN_DIR"
 	fi &&
 	TEST_RESULTS_SAN_FILE="$TEST_RESULTS_SAN_DIR/$TEST_RESULTS_SAN_FILE_PFX"
 
-	# In case "test-results" is left over from a previous
-	# run: Only report if new leaks show up.
-	TEST_RESULTS_SAN_DIR_NR_LEAKS_STARTUP=$(nr_san_dir_leaks_)
-
 	# Don't litter *.leak dirs if there was nothing to report
 	test_atexit "rmdir \"$TEST_RESULTS_SAN_DIR\" 2>/dev/null || :"
 
-- 
2.46.2.1011.gf1f9323e02

Re: [PATCH 1/3] test-lib: stop showing old leak logs

[Patrick Steinhardt]

On Tue, Sep 24, 2024 at 05:35:40PM -0400, Jeff King wrote:
> We ask LSan to record the logs of all leaks in test-results/, which is
> useful for finding leaks that didn't trigger a test failure.
> 
> We don't clean out the leak/ directory for each test before running it,
> though. Instead, we count the number of files it has, and complain only
> if we ended up with more when the script finishes. So we shouldn't
> trigger any output if you've made a script leak free. But if you simply
> _reduced_ the number of leaks, then there is an annoying outcome: we do
> not record which logs were from this run and which were from previous
> ones. So when we dump them to stdout, you get a mess of
> possibly-outdated leaks. This is very confusing when you are in an
> edit-compile-test cycle trying to fix leaks.
> 
> The instructions do note that you should "rm -rf test-results/" if you
> want to avoid this. But I'm having trouble seeing how this cumulative
> count could ever be useful. It is not even counting the number of leaks,
> but rather the number of processes with at least one leak!
> 
> So let's just blow away the per-test leak/ directory before running. We
> already overwrite the ".out" file in test-results/ in the same way, so
> this is following that pattern.
> 
> Running "make test" isn't affected by this, since it blows away all of
> test-results/ already. This only comes up when you are iterating on a
> single script that you're running manually.

I'm very much in favor of this change. I frequently re-ran tests with
leak checking enabled only to realize that, oops, I forgot to "rm -rf"
the leak directory first. So eventually I ended up with the following
command:

    $ rm -rf /tmp/git-tests/ && make -C .. -j20 SANITIZE=leak && ./t5310-pack-bitmaps.sh -ix

Every time I didn't use it I soon came to regret it.

Patrick

[PATCH 2/3] test-lib: show leak-sanitizer logs on --immediate failure

[Jeff King]

When we've compiled with SANITIZE=leak, at the end of the test script
we'll dump any collected logs to stdout. These logs have two uses:

  1. Leaks don't always cause a test snippet to fail (e.g., if they
     happen in a sub-process that we expect to return non-zero).
     Checking the logs catches these cases that we'd otherwise miss
     entirely.

  2. LSan will dump the leak info to stderr, but that is sometimes
     hidden (e.g., because it's redirected by the test, or because it's
     in a sub-process whose stderr goes elsewhere). Dumping the logs is
     the easiest way for the developer to see them.

One downside is that the set of logs for an entire script may be very
long, especially when you're trying to fix existing test scripts. You
can run with --immediate to stop at the first failing test, which means
we'll have accrued fewer logs. But we don't show the logs in that case!

Let's start doing so. This can only help case (2), of course (since it
depends on test failure). And it's somewhat weakened by the fact that
any cases of (1) will pollute the logs. But we can improve things
further in the next patch.

Signed-off-by: Jeff King <peff@peff.net>
---
 t/test-lib.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/t/test-lib.sh b/t/test-lib.sh
index 7d4471fbc5..d624ee186c 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -847,6 +847,7 @@ test_failure_ () {
 			GIT_EXIT_OK=t
 			exit 0
 		fi
+		check_test_results_san_file_ "$test_failure"
 		_error_exit
 	fi
 	finalize_test_case_output failure "$failure_label" "$@"
-- 
2.46.2.1011.gf1f9323e02

Re: [PATCH 2/3] test-lib: show leak-sanitizer logs on --immediate failure

[Patrick Steinhardt]

On Tue, Sep 24, 2024 at 05:36:36PM -0400, Jeff King wrote:
> When we've compiled with SANITIZE=leak, at the end of the test script
> we'll dump any collected logs to stdout. These logs have two uses:
> 
>   1. Leaks don't always cause a test snippet to fail (e.g., if they
>      happen in a sub-process that we expect to return non-zero).
>      Checking the logs catches these cases that we'd otherwise miss
>      entirely.
> 
>   2. LSan will dump the leak info to stderr, but that is sometimes
>      hidden (e.g., because it's redirected by the test, or because it's
>      in a sub-process whose stderr goes elsewhere). Dumping the logs is
>      the easiest way for the developer to see them.
> 
> One downside is that the set of logs for an entire script may be very
> long, especially when you're trying to fix existing test scripts. You
> can run with --immediate to stop at the first failing test, which means
> we'll have accrued fewer logs. But we don't show the logs in that case!
> 
> Let's start doing so. This can only help case (2), of course (since it
> depends on test failure). And it's somewhat weakened by the fact that
> any cases of (1) will pollute the logs. But we can improve things
> further in the next patch.

Yes, please!

Patrick

[PATCH 3/3] test-lib: check for leak logs after every test

[Jeff King]

If you are trying to find and fix leaks in a large test script, it can
be overwhelming to see the leak logs for every test at once. The
previous commit let you use "--immediate" to see the logs after the
first failing test, but this isn't always the first leak. As discussed
there, we may see leaks from previous tests that didn't happen to fail.

To catch those, let's check for any logs that appeared after each test
snippet is run, meaning that in a SANITIZE=leak build, any leak is an
immediate failure of the test snippet.

This check is mostly free in non-leak builds (just a "test -z"), and
only a few extra processes in a leak build, so I don't think the
overhead should matter (if it does, we could probably optimize for the
common "no logs" case without even spending a process).

Signed-off-by: Jeff King <peff@peff.net>
---
 t/test-lib-functions.sh |  3 ++-
 t/test-lib.sh           | 11 ++++++-----
 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index fde9bf54fc..78e054ab50 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -926,7 +926,8 @@ test_expect_success () {
 		test_body_or_stdin test_body "$2"
 		test -n "$test_skip_test_preamble" ||
 		say >&3 "expecting success of $TEST_NUMBER.$test_count '$1': $test_body"
-		if test_run_ "$test_body"
+		if test_run_ "$test_body" &&
+		   check_test_results_san_file_empty_
 		then
 			test_ok_ "$1"
 		else
diff --git a/t/test-lib.sh b/t/test-lib.sh
index d624ee186c..b1a8ee5c00 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -1215,12 +1215,13 @@ test_atexit_handler () {
 	teardown_malloc_check
 }
 
+check_test_results_san_file_empty_ () {
+	test -z "$TEST_RESULTS_SAN_FILE" ||
+	test "$(nr_san_dir_leaks_)" = 0
+}
+
 check_test_results_san_file_ () {
-	if test -z "$TEST_RESULTS_SAN_FILE"
-	then
-		return
-	fi &&
-	if test "$(nr_san_dir_leaks_)" = 0
+	if check_test_results_san_file_empty_
 	then
 		return
 	fi &&
-- 
2.46.2.1011.gf1f9323e02

Re: [PATCH 3/3] test-lib: check for leak logs after every test

[Patrick Steinhardt]

On Tue, Sep 24, 2024 at 05:38:36PM -0400, Jeff King wrote:
> If you are trying to find and fix leaks in a large test script, it can
> be overwhelming to see the leak logs for every test at once. The
> previous commit let you use "--immediate" to see the logs after the
> first failing test, but this isn't always the first leak. As discussed
> there, we may see leaks from previous tests that didn't happen to fail.
> 
> To catch those, let's check for any logs that appeared after each test
> snippet is run, meaning that in a SANITIZE=leak build, any leak is an
> immediate failure of the test snippet.
> 
> This check is mostly free in non-leak builds (just a "test -z"), and
> only a few extra processes in a leak build, so I don't think the
> overhead should matter (if it does, we could probably optimize for the
> common "no logs" case without even spending a process).

So previously, `--immediate` didn't detect tests that should have failed
because they were leaks, and now it does? Sounds like a sensible change
to me, too.

Patrick

Re: [PATCH 3/3] test-lib: check for leak logs after every test

[Jeff King]

On Thu, Sep 26, 2024 at 04:19:24PM +0200, Patrick Steinhardt wrote:

> On Tue, Sep 24, 2024 at 05:38:36PM -0400, Jeff King wrote:
> > If you are trying to find and fix leaks in a large test script, it can
> > be overwhelming to see the leak logs for every test at once. The
> > previous commit let you use "--immediate" to see the logs after the
> > first failing test, but this isn't always the first leak. As discussed
> > there, we may see leaks from previous tests that didn't happen to fail.
> > 
> > To catch those, let's check for any logs that appeared after each test
> > snippet is run, meaning that in a SANITIZE=leak build, any leak is an
> > immediate failure of the test snippet.
> > 
> > This check is mostly free in non-leak builds (just a "test -z"), and
> > only a few extra processes in a leak build, so I don't think the
> > overhead should matter (if it does, we could probably optimize for the
> > common "no logs" case without even spending a process).
> 
> So previously, `--immediate` didn't detect tests that should have failed
> because they were leaks, and now it does? Sounds like a sensible change
> to me, too.

Yes, though just to be pedantic, they are marked as failures even
without --immediate. It is just that doing so is a lot more useful with
--immediate, since otherwise we'd find the leaks at the end (but even
that it may still be useful to point to a particular test).

So it really is "if you are in a SANITIZE=leak build, generating a leak
log fails the test even if it would otherwise pass".

-Peff

Re: [PATCH 0/3] LSan quality of life improvements

[Patrick Steinhardt]

On Tue, Sep 24, 2024 at 05:34:04PM -0400, Jeff King wrote:
> I was fixing some leaks the other day and came up with a few changes
> that made the process a bit less painful, especially when the leaks are
> hidden in sub-processes (which is most of them when you are digging into
> http push/fetch, as I was).
> 
> I hope we're not too far off from a world where leaks are something that
> pop up in your newly written code, and you're not slogging through
> existing test scripts. But until then, I hope these might help others.
> 
> If you want to see the before/after, try:
> 
>   cd t
>   ./t5550-http-fetch-dumb.sh -i
> 
> before and after this series. Before you get no leaks reported to stdout
> with "-i", and way too many without it. After, you get the leaks for the
> first test that generates them.
> 
>   [1/3]: test-lib: stop showing old leak logs
>   [2/3]: test-lib: show leak-sanitizer logs on --immediate failure
>   [3/3]: test-lib: check for leak logs after every test

Thanks, all of these look like strict improvements to me! Now I wish
that I had all of these changes earlier, as that would have made my
leak-fixing sessions way more enjoyable.

Patrick