Git development
 help / color / mirror / Atom feed
From: Patrick Steinhardt <ps@pks.im>
To: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
Cc: Git mailing list <git@vger.kernel.org>
Subject: Re: [PATCH] test-lib: run EXPENSIVE tests one at a time
Date: Thu, 19 Nov 2026 11:07:01 +0100	[thread overview]
Message-ID: <av7KRbOGWeV-Qof1@pks.im> (raw)
In-Reply-To: <20261119-ks-ci-long-test-hang-fix-v1-1-45c2fff6d9f0@gmail.com>

On Thu, Nov 19, 2026 at 01:10:31PM +0530, Kaartic Sivaraam wrote:
> Recently, we've acquired quite a few expensive test in our test suite.

s/test/tests/

> For instance, these commits add several tests that require
> multi-gigabytes:

Multi-gigabytes what?

[snip]
> In recent times, we've been noticing the sanitizer jobs (linux-leaks,
> linux-reftable-leaks, linux-asan-ubsan) hang and fail with timeout in
> our GitLab CI runners. On some digging, it turns out that this is
> because of multiple memory hungry tests running in parallel.

Yeah, our pipelines are currently a mess because of this, so I'd be
really happy to have this resolved soonish.

> diff --git a/t/test-lib.sh b/t/test-lib.sh
> index c56654b7c5..4d29172c9d 100644
> --- a/t/test-lib.sh
> +++ b/t/test-lib.sh
> @@ -1089,6 +1090,42 @@ fail_117 () {
>  	return 117
>  }
>  
> +# EXPENSIVE tests can need many gigabytes of memory or disk, so two of

s/can/may/

> +# them running at the same time under "prove --jobs" can exhaust the

I wouldn't single out `prove --jobs` here, as we know to run
concurrently without prove either via our plain Makefile or via Meson.

> +# machine. Allow only one at a time across all test scripts that share
> +# this TEST_OUTPUT_DIRECTORY.
> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
> +have_expensive_lock_=
> +
> +acquire_expensive_lock_ () {
> +	mkdir -p "$TEST_RESULTS_DIR" || return 1
> +	while ! mkdir "$expensive_lock_" 2>/dev/null
> +	do
> +		owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
> +		if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
> +		then
> +			# The holder was killed without releasing the lock.
> +			rm -rf "$expensive_lock_"
> +			continue

Okay, if the previous owner does not exist anymore we prune the lock.
Two racing processes which try to acquire the lock may race in a way
that the "/pid" file does not exist anymore because the other process
has already deleted it. But you protect against that by checking whether
we have been able to read a PID in the first place via `test -n`.

> +		fi
> +		if test -z "$expensive_lock_waited_"
> +		then
> +			say >&3 "waiting for another EXPENSIVE test to finish"

Hmm. So file descriptor 3 points to stderr or to the test output file.
And if I remember correctly, we're indeed free to write to stderr
whatever data we want without breaking the TAP format?

> +			expensive_lock_waited_=t
> +		fi
> +		sleep 1
> +	done
> +	expensive_lock_waited_=

Should this be declared as a local variable?

Overall this is a rather pragmatic approach to solve the issue. I was
thinking whether it would make sense to distinguish e.g. EXPENISVE_MEM
from EXPENSIVE_DISK, and have different locks for each of those. But I
doubt adding this kind of complexity would really be worth it in the
end.

So I'm fine with the approach. Onward to green pipelines again! Well,
until the next thing breaks next week...

Thanks!

Patrick

  reply	other threads:[~2026-11-19 10:07 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19  7:40 [PATCH] test-lib: run EXPENSIVE tests one at a time Kaartic Sivaraam
2026-11-19 10:07 ` Patrick Steinhardt [this message]
2026-11-19 17:12   ` Kaartic Sivaraam
2026-11-19 17:29 ` [PATCH v2] " Kaartic Sivaraam
2026-11-19 17:33   ` Kaartic Sivaraam

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av7KRbOGWeV-Qof1@pks.im \
    --to=ps@pks.im \
    --cc=git@vger.kernel.org \
    --cc=kaartic.sivaraam@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox