Git development
 help / color / mirror / Atom feed
From: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
To: Patrick Steinhardt <ps@pks.im>
Cc: Git mailing list <git@vger.kernel.org>
Subject: Re: [PATCH] test-lib: run EXPENSIVE tests one at a time
Date: Thu, 19 Nov 2026 22:42:53 +0530	[thread overview]
Message-ID: <3229c599-bdae-46ef-bcad-9557965b4b43@gmail.com> (raw)
In-Reply-To: <av7KRbOGWeV-Qof1@pks.im>

On 11/19/26 15:37, Patrick Steinhardt wrote:
> On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote:
>> Recently, we've acquired quite a few expensive test in our test suite.
> 
> s/test/tests/

Corrected it.

> 
>> For instance, these commits add several tests that require
>> multi-gigabytes:
> 
> Multi-gigabytes what?
>

Fixed it!
>> diff --git a/t/test-lib.sh b/t/test-lib.sh
>> index c56654b7c5..4d29172c9d 100644
>> --- a/t/test-lib.sh
>> +++ b/t/test-lib.sh
>> @@ -1089,6 +1090,42 @@ fail_117 () {
>>   	return 117
>>   }
>>   
>> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of
> 
> s/can/may/
> 

Done.

>> +# them running at the same time under "prove --jobs" can exhaust the
> 
> I wouldn't single out `prove --jobs` here, as we know to run
> concurrently without prove either via our plain Makefile or via Meson.
>

Makes sense. Will remove the part mentioning 'prove --jobs' specifically.

>> +# machine. Allow only one at a time across all test scripts that share
>> +# this TEST_OUTPUT_DIRECTORY.
>> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
>> +have_expensive_lock_=
>> +
>> +acquire_expensive_lock_ () {
>> +	mkdir -p "$TEST_RESULTS_DIR" || return 1
>> +	while ! mkdir "$expensive_lock_" 2>/dev/null
>> +	do
>> +		owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
>> +		if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
>> +		then
>> +			# The holder was killed without releasing the lock.
>> +			rm -rf "$expensive_lock_"
>> +			continue
> 
> Okay, if the previous owner does not exist anymore we prune the lock.
> Two racing processes which try to acquire the lock may race in a way
> that the "/pid" file does not exist anymore because the other process
> has already deleted it. But you protect against that by checking whether
> we have been able to read a PID in the first place via `test -n`.

Indeed. There might still be a race in the case when two racing 
processes (A and B) both see the same dead PID before either acts. A 
would have deleted the lock, created a new one, written its PID. B which 
still holds the stale PID would proceed to delete the lock which is 
actually the live lock held by A, create a new one and write its PID. 
This might happen if a holder was SIGKILLed and two or more scripts were 
waiting at that exact moment, so it's rare. So, I wouldn't complicate 
the code to handle this. In the worst case, we'd have 2 expensive test 
cases running in place of the one we intend.

> 
>> +		fi
>> +		if test -z "$expensive_lock_waited_"
>> +		then
>> +			say >&3 "waiting for another EXPENSIVE test to finish"
> 
> Hmm. So file descriptor 3 points to stderr or to the test output file.
> And if I remember correctly, we're indeed free to write to stderr
> whatever data we want without breaking the TAP format?
> 

Yep. With no '--verbose' or '--verbose-log' flags this message would 
just go to /dev/null. I suppose that will only happen when someone runs 
this manually. So, hopefully that's fine.

>> +			expensive_lock_waited_=t
>> +		fi
>> +		sleep 1
>> +	done
>> +	expensive_lock_waited_=
> 
> Should this be declared as a local variable?
>

Sure. Will make 'owner' local too.

> Overall this is a rather pragmatic approach to solve the issue. I was
> thinking whether it would make sense to distinguish e.g. EXPENISVE_MEM
> from EXPENSIVE_DISK, and have different locks for each of those. But I
> doubt adding this kind of complexity would really be worth it in the
> end.
> 

Got it. Thank you for sharing your thoughts!

> So I'm fine with the approach. Onward to green pipelines again! Well,
> until the next thing breaks next week...
> 

Totally. It feels is like we're constantly fire-fighting in the CI these 
days ...

-- 
Sivaraam


  reply	other threads:[~2026-11-19 17:12 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19  7:40 [PATCH] test-lib: run EXPENSIVE tests one at a time Kaartic Sivaraam
2026-11-19 10:07 ` Patrick Steinhardt
2026-11-19 17:12   ` Kaartic Sivaraam [this message]
2026-11-19 17:29 ` [PATCH v2] " Kaartic Sivaraam
2026-11-19 17:33   ` Kaartic Sivaraam

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=3229c599-bdae-46ef-bcad-9557965b4b43@gmail.com \
    --to=kaartic.sivaraam@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=ps@pks.im \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox