From: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
To: Patrick Steinhardt <ps@pks.im>
Cc: Git mailing list <git@vger.kernel.org>
Subject: Re: [PATCH] test-lib: run EXPENSIVE tests one at a time
Date: Thu, 19 Nov 2026 22:42:53 +0530 [thread overview]
Message-ID: <3229c599-bdae-46ef-bcad-9557965b4b43@gmail.com> (raw)
In-Reply-To: <av7KRbOGWeV-Qof1@pks.im>
On 11/19/26 15:37, Patrick Steinhardt wrote:
> On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote:
>> Recently, we've acquired quite a few expensive test in our test suite.
>
> s/test/tests/
Corrected it.
>
>> For instance, these commits add several tests that require
>> multi-gigabytes:
>
> Multi-gigabytes what?
>
Fixed it!
>> diff --git a/t/test-lib.sh b/t/test-lib.sh
>> index c56654b7c5..4d29172c9d 100644
>> --- a/t/test-lib.sh
>> +++ b/t/test-lib.sh
>> @@ -1089,6 +1090,42 @@ fail_117 () {
>> return 117
>> }
>>
>> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of
>
> s/can/may/
>
Done.
>> +# them running at the same time under "prove --jobs" can exhaust the
>
> I wouldn't single out `prove --jobs` here, as we know to run
> concurrently without prove either via our plain Makefile or via Meson.
>
Makes sense. Will remove the part mentioning 'prove --jobs' specifically.
>> +# machine. Allow only one at a time across all test scripts that share
>> +# this TEST_OUTPUT_DIRECTORY.
>> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
>> +have_expensive_lock_=
>> +
>> +acquire_expensive_lock_ () {
>> + mkdir -p "$TEST_RESULTS_DIR" || return 1
>> + while ! mkdir "$expensive_lock_" 2>/dev/null
>> + do
>> + owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
>> + if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
>> + then
>> + # The holder was killed without releasing the lock.
>> + rm -rf "$expensive_lock_"
>> + continue
>
> Okay, if the previous owner does not exist anymore we prune the lock.
> Two racing processes which try to acquire the lock may race in a way
> that the "/pid" file does not exist anymore because the other process
> has already deleted it. But you protect against that by checking whether
> we have been able to read a PID in the first place via `test -n`.
Indeed. There might still be a race in the case when two racing
processes (A and B) both see the same dead PID before either acts. A
would have deleted the lock, created a new one, written its PID. B which
still holds the stale PID would proceed to delete the lock which is
actually the live lock held by A, create a new one and write its PID.
This might happen if a holder was SIGKILLed and two or more scripts were
waiting at that exact moment, so it's rare. So, I wouldn't complicate
the code to handle this. In the worst case, we'd have 2 expensive test
cases running in place of the one we intend.
>
>> + fi
>> + if test -z "$expensive_lock_waited_"
>> + then
>> + say >&3 "waiting for another EXPENSIVE test to finish"
>
> Hmm. So file descriptor 3 points to stderr or to the test output file.
> And if I remember correctly, we're indeed free to write to stderr
> whatever data we want without breaking the TAP format?
>
Yep. With no '--verbose' or '--verbose-log' flags this message would
just go to /dev/null. I suppose that will only happen when someone runs
this manually. So, hopefully that's fine.
>> + expensive_lock_waited_=t
>> + fi
>> + sleep 1
>> + done
>> + expensive_lock_waited_=
>
> Should this be declared as a local variable?
>
Sure. Will make 'owner' local too.
> Overall this is a rather pragmatic approach to solve the issue. I was
> thinking whether it would make sense to distinguish e.g. EXPENISVE_MEM
> from EXPENSIVE_DISK, and have different locks for each of those. But I
> doubt adding this kind of complexity would really be worth it in the
> end.
>
Got it. Thank you for sharing your thoughts!
> So I'm fine with the approach. Onward to green pipelines again! Well,
> until the next thing breaks next week...
>
Totally. It feels is like we're constantly fire-fighting in the CI these
days ...
--
Sivaraam
next prev parent reply other threads:[~2026-11-19 17:12 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 7:40 [PATCH] test-lib: run EXPENSIVE tests one at a time Kaartic Sivaraam
2026-11-19 10:07 ` Patrick Steinhardt
2026-11-19 17:12 ` Kaartic Sivaraam [this message]
2026-11-19 17:29 ` [PATCH v2] " Kaartic Sivaraam
2026-11-19 17:33 ` Kaartic Sivaraam
2026-11-20 6:28 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3229c599-bdae-46ef-bcad-9557965b4b43@gmail.com \
--to=kaartic.sivaraam@gmail.com \
--cc=git@vger.kernel.org \
--cc=ps@pks.im \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.