* [PATCH] test-lib: run EXPENSIVE tests one at a time
@ 2026-11-19 7:40 Kaartic Sivaraam
2026-11-19 10:07 ` Patrick Steinhardt
2026-11-19 17:29 ` [PATCH v2] " Kaartic Sivaraam
0 siblings, 2 replies; 5+ messages in thread
From: Kaartic Sivaraam @ 2026-11-19 7:40 UTC (permalink / raw)
To: Git mailing list; +Cc: Patrick Steinhardt
Recently, we've acquired quite a few expensive test in our test suite.
For instance, these commits add several tests that require
multi-gigabytes:
- 9048b48fb (t1600-index: add expensive tests for over-long
tree paths, 2026-10-16) added tests to t1600
- 288266c11d (xdiff: use size_t for merge buffer size computations,
2026-10-16) added tests to t4300
- 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE,
2026-03-25), added tests to t4141
We also have the memory hungry 5GB filtered hash-object test in t1007.
When GIT_TEST_LONG is set, "prove --jobs" happily runs several of these
at once. On the 16GB GitLab CI runners this exhausts memory: the
sanitizer jobs have been hanging in t1007 until they hit the job
timeout ever since the new EXPENSIVE tests in t1600 started running
alongside it. Running the two under a 16GB memory cap reproduces it,
with the t1007 git process getting OOM-killed.
GitHub CI runners are no exception and observe the same behaviour.
Take a lock shared by all test scripts around each EXPENSIVE test, so
only one of them runs at a time. The lock is a directory in
TEST_RESULTS_DIR, created with mkdir(1) like the existing valgrind
symlink lock. It records the holder's PID so that a lock left behind by
a script that was killed with SIGKILL gets taken over by the next
waiter. It is released after the test and its cleanup finish, and from
the EXIT trap when a script dies or is interrupted while holding it.
Cheap tests in the same scripts keep running in parallel. Only the
expensive ones wait for each other.
Signed-off-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
---
In recent times, we've been noticing the sanitizer jobs (linux-leaks,
linux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in
our GitLab CI runners. On some digging, it turns out that this is
because of multiple memory hungry tests running in parallel.
So, I thought of suggesting this patch to serialize the expensive tests
at least to avoid the putting the CI runners under memory pressure.
The GitHub CI runners also observed the same behaviour.
Both CI runners are happy with this change.
GitHub: https://github.com/sivaraam/git/actions/runs/43281285339
GitLab: https://gitlab.com/gitlab-org/git/-/pipelines/3079929506
---
t/README | 4 +++-
t/test-lib.sh | 44 ++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 47 insertions(+), 1 deletion(-)
diff --git a/t/README b/t/README
index fd7e8504b5..c156dd3af6 100644
--- a/t/README
+++ b/t/README
@@ -117,7 +117,9 @@ appropriately before running "make". Short options can be bundled, i.e.
-l::
--long-tests::
This causes additional long-running tests to be run (where
- available), for more exhaustive testing.
+ available), for more exhaustive testing. These tests can use a
+ lot of memory and disk, so only one of them runs at a time, even
+ when several test scripts run in parallel (e.g. "prove --jobs").
-r::
--run=<test-selector>::
diff --git a/t/test-lib.sh b/t/test-lib.sh
index c56654b7c5..4d29172c9d 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -784,6 +784,7 @@ die () {
# test script run with '--immediate' fails, or when the user hits
# ctrl-C, i.e. when 'test_done' is not invoked at all.
test_atexit_handler || code=$?
+ release_expensive_lock_
if test -n "$GIT_EXIT_OK"
then
exit $code
@@ -1089,6 +1090,42 @@ fail_117 () {
return 117
}
+# EXPENSIVE tests can need many gigabytes of memory or disk, so two of
+# them running at the same time under "prove --jobs" can exhaust the
+# machine. Allow only one at a time across all test scripts that share
+# this TEST_OUTPUT_DIRECTORY.
+expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
+have_expensive_lock_=
+
+acquire_expensive_lock_ () {
+ mkdir -p "$TEST_RESULTS_DIR" || return 1
+ while ! mkdir "$expensive_lock_" 2>/dev/null
+ do
+ owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
+ if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
+ then
+ # The holder was killed without releasing the lock.
+ rm -rf "$expensive_lock_"
+ continue
+ fi
+ if test -z "$expensive_lock_waited_"
+ then
+ say >&3 "waiting for another EXPENSIVE test to finish"
+ expensive_lock_waited_=t
+ fi
+ sleep 1
+ done
+ expensive_lock_waited_=
+ echo $$ >"$expensive_lock_/pid"
+ have_expensive_lock_=t
+}
+
+release_expensive_lock_ () {
+ test -n "$have_expensive_lock_" || return 0
+ rm -rf "$expensive_lock_"
+ have_expensive_lock_=
+}
+
test_run_ () {
test_cleanup=:
expecting_failure=$2
@@ -1103,6 +1140,12 @@ test_run_ () {
fi
fi
+ case ",$test_prereq," in
+ *,EXPENSIVE,*)
+ acquire_expensive_lock_
+ ;;
+ esac
+
setup_malloc_check
test_eval_ "$1"
eval_ret=$?
@@ -1115,6 +1158,7 @@ test_run_ () {
test_eval_ "$test_cleanup"
teardown_malloc_check
fi
+ release_expensive_lock_
if test "$verbose" = "t" && test -n "$HARNESS_ACTIVE"
then
echo ""
---
base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4
change-id: 20261117-ks-ci-long-test-hang-fix-008e72fd40c6
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH] test-lib: run EXPENSIVE tests one at a time
2026-11-19 7:40 [PATCH] test-lib: run EXPENSIVE tests one at a time Kaartic Sivaraam
@ 2026-11-19 10:07 ` Patrick Steinhardt
2026-11-19 17:12 ` Kaartic Sivaraam
2026-11-19 17:29 ` [PATCH v2] " Kaartic Sivaraam
1 sibling, 1 reply; 5+ messages in thread
From: Patrick Steinhardt @ 2026-11-19 10:07 UTC (permalink / raw)
To: Kaartic Sivaraam; +Cc: Git mailing list
On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote:
> Recently, we've acquired quite a few expensive test in our test suite.
s/test/tests/
> For instance, these commits add several tests that require
> multi-gigabytes:
Multi-gigabytes what?
[snip]
> In recent times, we've been noticing the sanitizer jobs (linux-leaks,
> linux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in
> our GitLab CI runners. On some digging, it turns out that this is
> because of multiple memory hungry tests running in parallel.
Yeah, our pipelines are currently a mess because of this, so I'd be
really happy to have this resolved soonish.
> diff --git a/t/test-lib.sh b/t/test-lib.sh
> index c56654b7c5..4d29172c9d 100644
> --- a/t/test-lib.sh
> +++ b/t/test-lib.sh
> @@ -1089,6 +1090,42 @@ fail_117 () {
> return 117
> }
>
> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of
s/can/may/
> +# them running at the same time under "prove --jobs" can exhaust the
I wouldn't single out `prove --jobs` here, as we know to run
concurrently without prove either via our plain Makefile or via Meson.
> +# machine. Allow only one at a time across all test scripts that share
> +# this TEST_OUTPUT_DIRECTORY.
> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
> +have_expensive_lock_=
> +
> +acquire_expensive_lock_ () {
> + mkdir -p "$TEST_RESULTS_DIR" || return 1
> + while ! mkdir "$expensive_lock_" 2>/dev/null
> + do
> + owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
> + if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
> + then
> + # The holder was killed without releasing the lock.
> + rm -rf "$expensive_lock_"
> + continue
Okay, if the previous owner does not exist anymore we prune the lock.
Two racing processes which try to acquire the lock may race in a way
that the "/pid" file does not exist anymore because the other process
has already deleted it. But you protect against that by checking whether
we have been able to read a PID in the first place via `test -n`.
> + fi
> + if test -z "$expensive_lock_waited_"
> + then
> + say >&3 "waiting for another EXPENSIVE test to finish"
Hmm. So file descriptor 3 points to stderr or to the test output file.
And if I remember correctly, we're indeed free to write to stderr
whatever data we want without breaking the TAP format?
> + expensive_lock_waited_=t
> + fi
> + sleep 1
> + done
> + expensive_lock_waited_=
Should this be declared as a local variable?
Overall this is a rather pragmatic approach to solve the issue. I was
thinking whether it would make sense to distinguish e.g. EXPENISVE_MEM
from EXPENSIVE_DISK, and have different locks for each of those. But I
doubt adding this kind of complexity would really be worth it in the
end.
So I'm fine with the approach. Onward to green pipelines again! Well,
until the next thing breaks next week...
Thanks!
Patrick
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] test-lib: run EXPENSIVE tests one at a time
2026-11-19 10:07 ` Patrick Steinhardt
@ 2026-11-19 17:12 ` Kaartic Sivaraam
0 siblings, 0 replies; 5+ messages in thread
From: Kaartic Sivaraam @ 2026-11-19 17:12 UTC (permalink / raw)
To: Patrick Steinhardt; +Cc: Git mailing list
On 11/19/26 15:37, Patrick Steinhardt wrote:
> On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote:
>> Recently, we've acquired quite a few expensive test in our test suite.
>
> s/test/tests/
Corrected it.
>
>> For instance, these commits add several tests that require
>> multi-gigabytes:
>
> Multi-gigabytes what?
>
Fixed it!
>> diff --git a/t/test-lib.sh b/t/test-lib.sh
>> index c56654b7c5..4d29172c9d 100644
>> --- a/t/test-lib.sh
>> +++ b/t/test-lib.sh
>> @@ -1089,6 +1090,42 @@ fail_117 () {
>> return 117
>> }
>>
>> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of
>
> s/can/may/
>
Done.
>> +# them running at the same time under "prove --jobs" can exhaust the
>
> I wouldn't single out `prove --jobs` here, as we know to run
> concurrently without prove either via our plain Makefile or via Meson.
>
Makes sense. Will remove the part mentioning 'prove --jobs' specifically.
>> +# machine. Allow only one at a time across all test scripts that share
>> +# this TEST_OUTPUT_DIRECTORY.
>> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
>> +have_expensive_lock_=
>> +
>> +acquire_expensive_lock_ () {
>> + mkdir -p "$TEST_RESULTS_DIR" || return 1
>> + while ! mkdir "$expensive_lock_" 2>/dev/null
>> + do
>> + owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
>> + if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
>> + then
>> + # The holder was killed without releasing the lock.
>> + rm -rf "$expensive_lock_"
>> + continue
>
> Okay, if the previous owner does not exist anymore we prune the lock.
> Two racing processes which try to acquire the lock may race in a way
> that the "/pid" file does not exist anymore because the other process
> has already deleted it. But you protect against that by checking whether
> we have been able to read a PID in the first place via `test -n`.
Indeed. There might still be a race in the case when two racing
processes (A and B) both see the same dead PID before either acts. A
would have deleted the lock, created a new one, written its PID. B which
still holds the stale PID would proceed to delete the lock which is
actually the live lock held by A, create a new one and write its PID.
This might happen if a holder was SIGKILLed and two or more scripts were
waiting at that exact moment, so it's rare. So, I wouldn't complicate
the code to handle this. In the worst case, we'd have 2 expensive test
cases running in place of the one we intend.
>
>> + fi
>> + if test -z "$expensive_lock_waited_"
>> + then
>> + say >&3 "waiting for another EXPENSIVE test to finish"
>
> Hmm. So file descriptor 3 points to stderr or to the test output file.
> And if I remember correctly, we're indeed free to write to stderr
> whatever data we want without breaking the TAP format?
>
Yep. With no '--verbose' or '--verbose-log' flags this message would
just go to /dev/null. I suppose that will only happen when someone runs
this manually. So, hopefully that's fine.
>> + expensive_lock_waited_=t
>> + fi
>> + sleep 1
>> + done
>> + expensive_lock_waited_=
>
> Should this be declared as a local variable?
>
Sure. Will make 'owner' local too.
> Overall this is a rather pragmatic approach to solve the issue. I was
> thinking whether it would make sense to distinguish e.g. EXPENISVE_MEM
> from EXPENSIVE_DISK, and have different locks for each of those. But I
> doubt adding this kind of complexity would really be worth it in the
> end.
>
Got it. Thank you for sharing your thoughts!
> So I'm fine with the approach. Onward to green pipelines again! Well,
> until the next thing breaks next week...
>
Totally. It feels is like we're constantly fire-fighting in the CI these
days ...
--
Sivaraam
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v2] test-lib: run EXPENSIVE tests one at a time
2026-11-19 7:40 [PATCH] test-lib: run EXPENSIVE tests one at a time Kaartic Sivaraam
2026-11-19 10:07 ` Patrick Steinhardt
@ 2026-11-19 17:29 ` Kaartic Sivaraam
2026-11-19 17:33 ` Kaartic Sivaraam
1 sibling, 1 reply; 5+ messages in thread
From: Kaartic Sivaraam @ 2026-11-19 17:29 UTC (permalink / raw)
To: Git mailing list; +Cc: Patrick Steinhardt, Kaartic Sivaraam
Recently, we've acquired quite a few expensive tests in our test suite.
For instance, these commits add several tests that require
multi-gigabytes of memory:
- 9048b48fb (t1600-index: add expensive tests for over-long
tree paths, 2026-10-16) added tests to t1600
- 288266c11d (xdiff: use size_t for merge buffer size computations,
2026-10-16) added tests to t4300
- 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE,
2026-03-25), added tests to t4141
We also have the memory hungry 5GB filtered hash-object test in t1007.
When GIT_TEST_LONG is set, "prove --jobs" happily runs several of these
at once. On the 16GB GitLab CI runners this exhausts memory: the
sanitizer jobs have been hanging in t1007 until they hit the job
timeout ever since the new EXPENSIVE tests in t1600 started running
alongside it. Running the two under a 16GB memory cap reproduces it,
with the t1007 git process getting OOM-killed.
GitHub CI runners are no exception and observe the same behaviour.
Take a lock shared by all test scripts around each EXPENSIVE test, so
only one of them runs at a time. The lock is a directory in
TEST_RESULTS_DIR, created with mkdir(1) like the existing valgrind
symlink lock. It records the holder's PID so that a lock left behind by
a script that was killed with SIGKILL gets taken over by the next
waiter. It is released after the test and its cleanup finish, and from
the EXIT trap when a script dies or is interrupted while holding it.
Cheap tests in the same scripts keep running in parallel. Only the
expensive ones wait for each other.
Signed-off-by: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
---
In recent times, we've been noticing the sanitizer jobs (linux-leaks,
linux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in
our GitLab CI runners. On some digging, it turns out that this is
because of multiple memory hungry tests running in parallel.
So, I thought of suggesting this patch to serialize the expensive tests
at least to avoid the putting the CI runners under memory pressure.
The GitHub CI runners also observed the same behaviour.
Both CI runners are happy with this change.
GitHub: https://github.com/sivaraam/git/actions/runs/43281285339
GitLab: https://gitlab.com/gitlab-org/git/-/pipelines/3079929506
---
Changes in v2:
- Corrected commit messages as per Patrick's suggestions
- Tweaked the shell function comment to avoid mentioning 'prove --jobs'
- Made the owner and waiting variables local within acquire_expensive_lock_
Link to v1: https://patch.msgid.link/20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com
---
t/README | 4 +++-
t/test-lib.sh | 45 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 48 insertions(+), 1 deletion(-)
diff --git a/t/README b/t/README
index fd7e8504b5..c156dd3af6 100644
--- a/t/README
+++ b/t/README
@@ -117,7 +117,9 @@ appropriately before running "make". Short options can be bundled, i.e.
-l::
--long-tests::
This causes additional long-running tests to be run (where
- available), for more exhaustive testing.
+ available), for more exhaustive testing. These tests can use a
+ lot of memory and disk, so only one of them runs at a time, even
+ when several test scripts run in parallel (e.g. "prove --jobs").
-r::
--run=<test-selector>::
diff --git a/t/test-lib.sh b/t/test-lib.sh
index c56654b7c5..193bf06aea 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -784,6 +784,7 @@ die () {
# test script run with '--immediate' fails, or when the user hits
# ctrl-C, i.e. when 'test_done' is not invoked at all.
test_atexit_handler || code=$?
+ release_expensive_lock_
if test -n "$GIT_EXIT_OK"
then
exit $code
@@ -1089,6 +1090,43 @@ fail_117 () {
return 117
}
+# EXPENSIVE tests may need many gigabytes of memory or disk, so two of
+# them running at the same time can exhaust the machine.
+#
+# Allow only one at a time across all test scripts that share
+# this TEST_OUTPUT_DIRECTORY.
+expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
+have_expensive_lock_=
+
+acquire_expensive_lock_ () {
+ local owner waited=
+ mkdir -p "$TEST_RESULTS_DIR" || return 1
+ while ! mkdir "$expensive_lock_" 2>/dev/null
+ do
+ owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
+ if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
+ then
+ # The holder was killed without releasing the lock.
+ rm -rf "$expensive_lock_"
+ continue
+ fi
+ if test -z "$waited"
+ then
+ say >&3 "waiting for another EXPENSIVE test to finish"
+ waited=t
+ fi
+ sleep 1
+ done
+ echo $$ >"$expensive_lock_/pid"
+ have_expensive_lock_=t
+}
+
+release_expensive_lock_ () {
+ test -n "$have_expensive_lock_" || return 0
+ rm -rf "$expensive_lock_"
+ have_expensive_lock_=
+}
+
test_run_ () {
test_cleanup=:
expecting_failure=$2
@@ -1103,6 +1141,12 @@ test_run_ () {
fi
fi
+ case ",$test_prereq," in
+ *,EXPENSIVE,*)
+ acquire_expensive_lock_
+ ;;
+ esac
+
setup_malloc_check
test_eval_ "$1"
eval_ret=$?
@@ -1115,6 +1159,7 @@ test_run_ () {
test_eval_ "$test_cleanup"
teardown_malloc_check
fi
+ release_expensive_lock_
if test "$verbose" = "t" && test -n "$HARNESS_ACTIVE"
then
echo ""
---
base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4
change-id: 20261117-ks-ci-long-test-hang-fix-008e72fd40c6
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH v2] test-lib: run EXPENSIVE tests one at a time
2026-11-19 17:29 ` [PATCH v2] " Kaartic Sivaraam
@ 2026-11-19 17:33 ` Kaartic Sivaraam
0 siblings, 0 replies; 5+ messages in thread
From: Kaartic Sivaraam @ 2026-11-19 17:33 UTC (permalink / raw)
To: Git mailing list; +Cc: Patrick Steinhardt
On 11/19/26 22:59, Kaartic Sivaraam wrote:
> Changes in v2:
> - Corrected commit messages as per Patrick's suggestions
> - Tweaked the shell function comment to avoid mentioning 'prove --jobs'
> - Made the owner and waiting variables local within acquire_expensive_lock_
>
> Link to v1: https://patch.msgid.link/20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com
I thought b4 would've included it by itself. Anyways, here's the range-diff:
1: f0664ff4ee ! 1: fbf773f39a test-lib: run EXPENSIVE tests one at a time
@@ Metadata
## Commit message ##
test-lib: run EXPENSIVE tests one at a time
- Recently, we've acquired quite a few expensive test in our
test suite.
+ Recently, we've acquired quite a few expensive tests in our
test suite.
For instance, these commits add several tests that require
- multi-gigabytes:
+ multi-gigabytes of memory:
- 9048b48fb (t1600-index: add expensive tests for over-long
tree paths, 2026-10-16) added tests to t1600
@@ t/test-lib.sh: fail_117 () {
return 117
}
-+# EXPENSIVE tests can need many gigabytes of memory or disk, so
two of
-+# them running at the same time under "prove --jobs" can exhaust the
-+# machine. Allow only one at a time across all test scripts that
share
++# EXPENSIVE tests may need many gigabytes of memory or disk, so
two of
++# them running at the same time can exhaust the machine.
++#
++# Allow only one at a time across all test scripts that share
+# this TEST_OUTPUT_DIRECTORY.
+expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
+have_expensive_lock_=
+
+acquire_expensive_lock_ () {
++ local owner waited=
+ mkdir -p "$TEST_RESULTS_DIR" || return 1
+ while ! mkdir "$expensive_lock_" 2>/dev/null
+ do
@@ t/test-lib.sh: fail_117 () {
+ rm -rf "$expensive_lock_"
+ continue
+ fi
-+ if test -z "$expensive_lock_waited_"
++ if test -z "$waited"
+ then
+ say >&3 "waiting for another EXPENSIVE test to finish"
-+ expensive_lock_waited_=t
++ waited=t
+ fi
+ sleep 1
+ done
-+ expensive_lock_waited_=
+ echo $$ >"$expensive_lock_/pid"
+ have_expensive_lock_=t
+}
--
Sivaraam
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-19 17:33 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-19 7:40 [PATCH] test-lib: run EXPENSIVE tests one at a time Kaartic Sivaraam
2026-11-19 10:07 ` Patrick Steinhardt
2026-11-19 17:12 ` Kaartic Sivaraam
2026-11-19 17:29 ` [PATCH v2] " Kaartic Sivaraam
2026-11-19 17:33 ` Kaartic Sivaraam
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox