Git development
 help / color / mirror / Atom feed
From: Junio C Hamano <gitster@pobox.com>
To: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
Cc: Git mailing list <git@vger.kernel.org>,  Patrick Steinhardt <ps@pks.im>
Subject: Re: [PATCH v2] test-lib: run EXPENSIVE tests one at a time
Date: Thu, 19 Nov 2026 22:28:13 -0800	[thread overview]
Message-ID: <xmqqpkw0rp8i.fsf@gitster.g> (raw)
In-Reply-To: <20261119-ks-ci-long-test-hang-fix-v2-1-37011c08380d@gmail.com> (Kaartic Sivaraam's message of "Thu, 19 Nov 2026 22:59:15 +0530")

Kaartic Sivaraam <kaartic.sivaraam@gmail.com> writes:

> Recently, we've acquired quite a few expensive tests in our test suite.
> For instance, these commits add several tests that require
> multi-gigabytes of memory:
>
>   - 9048b48fb (t1600-index: add expensive tests for over-long
>     tree paths, 2026-10-16) added tests to t1600

No such commit X-<.  Perhaps 89048b48fb (t1600-index: add expensive
tests for over-long tree paths, 2026-10-16).

>   - 288266c11d (xdiff: use size_t for merge buffer size computations,
>     2026-10-16) added tests to t4300
>
>   - 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE,
>     2026-03-25), added tests to t4141

Inconsistent use of "," after the (close parentheses).

> +# EXPENSIVE tests may need many gigabytes of memory or disk, so two of
> +# them running at the same time can exhaust the machine.
> +#
> +# Allow only one at a time across all test scripts that share
> +# this TEST_OUTPUT_DIRECTORY.
> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
> +have_expensive_lock_=
> +
> +acquire_expensive_lock_ () {
> +	local owner waited=
> +	mkdir -p "$TEST_RESULTS_DIR" || return 1
> +	while ! mkdir "$expensive_lock_" 2>/dev/null
> +	do
> +		owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
> +		if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
> +		then
> +			# The holder was killed without releasing the lock.
> +			rm -rf "$expensive_lock_"
> +			continue
> +		fi
> +		if test -z "$waited"
> +		then
> +			say >&3 "waiting for another EXPENSIVE test to finish"
> +			waited=t
> +		fi
> +		sleep 1
> +	done
> +	echo $$ >"$expensive_lock_/pid"
> +	have_expensive_lock_=t
> +}

Hmph, is the above safe when multiple processes are working with
expensive tests during normal (i.e., not involving SIGKILL)
operation, or does it still have TOCTOU race on the lock file?

 * T1 holds $expensive_lock_, finishes its last EXPENSIVE test,
   calls release_expensive_lock_ (rm -rf "$expensive_lock_"), and
   exits.

 * Waiter T2 read owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
   (obtaining T1's PID) just before T1 removed the directory.

 * Waiter T3 runs mkdir "$expensive_lock_", succeeds, writes T3's
   PID to pid, and begins its EXPENSIVE test.

 * T2 now checks test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
   against T1's exited PID, finds that T1 is gone, runs rm -rf
   "$expensive_lock_", which deletes T3's active lock.  It loops,
   succeeds on mkdir "$expensive_lock_", and runs its EXPENSIVE test
   concurrently with T3.

Also, between the time you successfully mkdir the lock directory and
the time you leave this function, you can be killed by a signal with
INT, TERM, or HUP, for which t/test-lib.sh installs a hander to call
exit, and there is a "trap die EXIT".

You may manage to create $expensive_lock_ directory, open pid file
for writing, and then get killed before you echo $$ to the pid file
or assign 't' to have_expensive_lock_ variable.

When your die handler calls release_expensive_lock_, you would leave
with the directory behind and exit.  You may not have pid file, or
the pid file may be empty.  The waiting processes will not run the
kill -0 "$owner" part (because they do not know who you are that held
the lock and left before writing $$ in pid), the waiters will wait
for the mkdir to successfully create the directory that already
exists and nobody will ever remove, no?

Between the TOCTOU race allowing multiple processes entering the
expensive region and deadlocking denying everybody from entering the
expensive region, the latter is far worse, of course.

> +release_expensive_lock_ () {
> +	test -n "$have_expensive_lock_" || return 0
> +	rm -rf "$expensive_lock_"
> +	have_expensive_lock_=
> +}

      parent reply	other threads:[~2026-11-20  6:28 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-19  7:40 [PATCH] test-lib: run EXPENSIVE tests one at a time Kaartic Sivaraam
2026-11-19 10:07 ` Patrick Steinhardt
2026-11-19 17:12   ` Kaartic Sivaraam
2026-11-19 17:29 ` [PATCH v2] " Kaartic Sivaraam
2026-11-19 17:33   ` Kaartic Sivaraam
2026-11-20  6:28   ` Junio C Hamano [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=xmqqpkw0rp8i.fsf@gitster.g \
    --to=gitster@pobox.com \
    --cc=git@vger.kernel.org \
    --cc=kaartic.sivaraam@gmail.com \
    --cc=ps@pks.im \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox