Git development
 help / color / mirror / Atom feed
* [PATCH] test-lib: run EXPENSIVE tests one at a time
@ 2026-11-19  7:40 Kaartic Sivaraam
  2026-11-19 10:07 ` Patrick Steinhardt
  2026-11-19 17:29 ` [PATCH v2] " Kaartic Sivaraam
  0 siblings, 2 replies; 5+ messages in thread
From: Kaartic Sivaraam @ 2026-11-19  7:40 UTC (permalink / raw)
  To: Git mailing list; +Cc: Patrick Steinhardt

Recently, we've acquired quite a few expensive test in our test suite.
For instance, these commits add several tests that require
multi-gigabytes:

  - 9048b48fb (t1600-index: add expensive tests for over-long
    tree paths, 2026-10-16) added tests to t1600

  - 288266c11d (xdiff: use size_t for merge buffer size computations,
    2026-10-16) added tests to t4300

  - 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE,
    2026-03-25), added tests to t4141

We also have the memory hungry 5GB filtered hash-object test in t1007.

When GIT_TEST_LONG is set, "prove --jobs" happily runs several of these
at once. On the 16GB GitLab CI runners this exhausts memory: the
sanitizer jobs have been hanging in t1007 until they hit the job
timeout ever since the new EXPENSIVE tests in t1600 started running
alongside it. Running the two under a 16GB memory cap reproduces it,
with the t1007 git process getting OOM-killed.

GitHub CI runners are no exception and observe the same behaviour.

Take a lock shared by all test scripts around each EXPENSIVE test, so
only one of them runs at a time. The lock is a directory in
TEST_RESULTS_DIR, created with mkdir(1) like the existing valgrind
symlink lock. It records the holder's PID so that a lock left behind by
a script that was killed with SIGKILL gets taken over by the next
waiter. It is released after the test and its cleanup finish, and from
the EXIT trap when a script dies or is interrupted while holding it.

Cheap tests in the same scripts keep running in parallel. Only the
expensive ones wait for each other.

Signed-off-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
---
In recent times, we've been noticing the sanitizer jobs (linux-leaks,
linux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in
our GitLab CI runners. On some digging, it turns out that this is
because of multiple memory hungry tests running in parallel.

So, I thought of suggesting this patch to serialize the expensive tests
at least to avoid the putting the CI runners under memory pressure.

The GitHub CI runners also observed the same behaviour.

Both CI runners are happy with this change.

GitHub: https://github.com/sivaraam/git/actions/runs/43281285339
GitLab: https://gitlab.com/gitlab-org/git/-/pipelines/3079929506
---
 t/README      |  4 +++-
 t/test-lib.sh | 44 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 47 insertions(+), 1 deletion(-)

diff --git a/t/README b/t/README
index fd7e8504b5..c156dd3af6 100644
--- a/t/README
+++ b/t/README
@@ -117,7 +117,9 @@ appropriately before running "make". Short options can be bundled, i.e.
 -l::
 --long-tests::
 	This causes additional long-running tests to be run (where
-	available), for more exhaustive testing.
+	available), for more exhaustive testing. These tests can use a
+	lot of memory and disk, so only one of them runs at a time, even
+	when several test scripts run in parallel (e.g. "prove --jobs").
 
 -r::
 --run=<test-selector>::
diff --git a/t/test-lib.sh b/t/test-lib.sh
index c56654b7c5..4d29172c9d 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -784,6 +784,7 @@ die () {
 	# test script run with '--immediate' fails, or when the user hits
 	# ctrl-C, i.e. when 'test_done' is not invoked at all.
 	test_atexit_handler || code=$?
+	release_expensive_lock_
 	if test -n "$GIT_EXIT_OK"
 	then
 		exit $code
@@ -1089,6 +1090,42 @@ fail_117 () {
 	return 117
 }
 
+# EXPENSIVE tests can need many gigabytes of memory or disk, so two of
+# them running at the same time under "prove --jobs" can exhaust the
+# machine. Allow only one at a time across all test scripts that share
+# this TEST_OUTPUT_DIRECTORY.
+expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
+have_expensive_lock_=
+
+acquire_expensive_lock_ () {
+	mkdir -p "$TEST_RESULTS_DIR" || return 1
+	while ! mkdir "$expensive_lock_" 2>/dev/null
+	do
+		owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
+		if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
+		then
+			# The holder was killed without releasing the lock.
+			rm -rf "$expensive_lock_"
+			continue
+		fi
+		if test -z "$expensive_lock_waited_"
+		then
+			say >&3 "waiting for another EXPENSIVE test to finish"
+			expensive_lock_waited_=t
+		fi
+		sleep 1
+	done
+	expensive_lock_waited_=
+	echo $$ >"$expensive_lock_/pid"
+	have_expensive_lock_=t
+}
+
+release_expensive_lock_ () {
+	test -n "$have_expensive_lock_" || return 0
+	rm -rf "$expensive_lock_"
+	have_expensive_lock_=
+}
+
 test_run_ () {
 	test_cleanup=:
 	expecting_failure=$2
@@ -1103,6 +1140,12 @@ test_run_ () {
 		fi
 	fi
 
+	case ",$test_prereq," in
+	*,EXPENSIVE,*)
+		acquire_expensive_lock_
+		;;
+	esac
+
 	setup_malloc_check
 	test_eval_ "$1"
 	eval_ret=$?
@@ -1115,6 +1158,7 @@ test_run_ () {
 		test_eval_ "$test_cleanup"
 		teardown_malloc_check
 	fi
+	release_expensive_lock_
 	if test "$verbose" = "t" && test -n "$HARNESS_ACTIVE"
 	then
 		echo ""

---
base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4
change-id: 20261117-ks-ci-long-test-hang-fix-008e72fd40c6


^ permalink raw reply related	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-11-19 17:33 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-19  7:40 [PATCH] test-lib: run EXPENSIVE tests one at a time Kaartic Sivaraam
2026-11-19 10:07 ` Patrick Steinhardt
2026-11-19 17:12   ` Kaartic Sivaraam
2026-11-19 17:29 ` [PATCH v2] " Kaartic Sivaraam
2026-11-19 17:33   ` Kaartic Sivaraam

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox