From: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
To: Patrick Steinhardt <ps@pks.im>
Cc: Git mailing list <git@vger.kernel.org>
Subject: Re: [PATCH] test-lib: run EXPENSIVE tests one at a time
Date: Thu, 19 Nov 2026 22:42:53 +0530 [thread overview]
Message-ID: <3229c599-bdae-46ef-bcad-9557965b4b43@gmail.com> (raw)
In-Reply-To: <av7KRbOGWeV-Qof1@pks.im>
On 11/19/26 15:37, Patrick Steinhardt wrote:
> On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote:
>> Recently, we've acquired quite a few expensive test in our test suite.
>
> s/test/tests/
Corrected it.
>
>> For instance, these commits add several tests that require
>> multi-gigabytes:
>
> Multi-gigabytes what?
>
Fixed it!
>> diff --git a/t/test-lib.sh b/t/test-lib.sh
>> index c56654b7c5..4d29172c9d 100644
>> --- a/t/test-lib.sh
>> +++ b/t/test-lib.sh
>> @@ -1089,6 +1090,42 @@ fail_117 () {
>> return 117
>> }
>>
>> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of
>
> s/can/may/
>
Done.
>> +# them running at the same time under "prove --jobs" can exhaust the
>
> I wouldn't single out `prove --jobs` here, as we know to run
> concurrently without prove either via our plain Makefile or via Meson.
>
Makes sense. Will remove the part mentioning 'prove --jobs' specifically.
>> +# machine. Allow only one at a time across all test scripts that share
>> +# this TEST_OUTPUT_DIRECTORY.
>> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
>> +have_expensive_lock_=
>> +
>> +acquire_expensive_lock_ () {
>> + mkdir -p "$TEST_RESULTS_DIR" || return 1
>> + while ! mkdir "$expensive_lock_" 2>/dev/null
>> + do
>> + owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
>> + if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
>> + then
>> + # The holder was killed without releasing the lock.
>> + rm -rf "$expensive_lock_"
>> + continue
>
> Okay, if the previous owner does not exist anymore we prune the lock.
> Two racing processes which try to acquire the lock may race in a way
> that the "/pid" file does not exist anymore because the other process
> has already deleted it. But you protect against that by checking whether
> we have been able to read a PID in the first place via `test -n`.
Indeed. There might still be a race in the case when two racing
processes (A and B) both see the same dead PID before either acts. A
would have deleted the lock, created a new one, written its PID. B which
still holds the stale PID would proceed to delete the lock which is
actually the live lock held by A, create a new one and write its PID.
This might happen if a holder was SIGKILLed and two or more scripts were
waiting at that exact moment, so it's rare. So, I wouldn't complicate
the code to handle this. In the worst case, we'd have 2 expensive test
cases running in place of the one we intend.
>
>> + fi
>> + if test -z "$expensive_lock_waited_"
>> + then
>> + say >&3 "waiting for another EXPENSIVE test to finish"
>
> Hmm. So file descriptor 3 points to stderr or to the test output file.
> And if I remember correctly, we're indeed free to write to stderr
> whatever data we want without breaking the TAP format?
>
Yep. With no '--verbose' or '--verbose-log' flags this message would
just go to /dev/null. I suppose that will only happen when someone runs
this manually. So, hopefully that's fine.
>> + expensive_lock_waited_=t
>> + fi
>> + sleep 1
>> + done
>> + expensive_lock_waited_=
>
> Should this be declared as a local variable?
>
Sure. Will make 'owner' local too.
> Overall this is a rather pragmatic approach to solve the issue. I was
> thinking whether it would make sense to distinguish e.g. EXPENISVE_MEM
> from EXPENSIVE_DISK, and have different locks for each of those. But I
> doubt adding this kind of complexity would really be worth it in the
> end.
>
Got it. Thank you for sharing your thoughts!
> So I'm fine with the approach. Onward to green pipelines again! Well,
> until the next thing breaks next week...
>
Totally. It feels is like we're constantly fire-fighting in the CI these
days ...
--
Sivaraam
next prev parent reply other threads:[~2026-11-19 17:12 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 7:40 [PATCH] test-lib: run EXPENSIVE tests one at a time Kaartic Sivaraam
2026-11-19 10:07 ` Patrick Steinhardt
2026-11-19 17:12 ` Kaartic Sivaraam [this message]
2026-11-19 17:29 ` [PATCH v2] " Kaartic Sivaraam
2026-11-19 17:33 ` Kaartic Sivaraam
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3229c599-bdae-46ef-bcad-9557965b4b43@gmail.com \
--to=kaartic.sivaraam@gmail.com \
--cc=git@vger.kernel.org \
--cc=ps@pks.im \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox