From: Junio C Hamano <gitster@pobox.com>
To: Kaartic Sivaraam <kaartic.sivaraam@gmail.com>
Cc: Git mailing list <git@vger.kernel.org>, Patrick Steinhardt <ps@pks.im>
Subject: Re: [PATCH v2] test-lib: run EXPENSIVE tests one at a time
Date: Thu, 19 Nov 2026 22:28:13 -0800 [thread overview]
Message-ID: <xmqqpkw0rp8i.fsf@gitster.g> (raw)
In-Reply-To: <20261119-ks-ci-long-test-hang-fix-v2-1-37011c08380d@gmail.com> (Kaartic Sivaraam's message of "Thu, 19 Nov 2026 22:59:15 +0530")
Kaartic Sivaraam <kaartic.sivaraam@gmail.com> writes:
> Recently, we've acquired quite a few expensive tests in our test suite.
> For instance, these commits add several tests that require
> multi-gigabytes of memory:
>
> - 9048b48fb (t1600-index: add expensive tests for over-long
> tree paths, 2026-10-16) added tests to t1600
No such commit X-<. Perhaps 89048b48fb (t1600-index: add expensive
tests for over-long tree paths, 2026-10-16).
> - 288266c11d (xdiff: use size_t for merge buffer size computations,
> 2026-10-16) added tests to t4300
>
> - 55c54fd375 (apply: reject binary patches greater than MAX_APPLY_SIZE,
> 2026-03-25), added tests to t4141
Inconsistent use of "," after the (close parentheses).
> +# EXPENSIVE tests may need many gigabytes of memory or disk, so two of
> +# them running at the same time can exhaust the machine.
> +#
> +# Allow only one at a time across all test scripts that share
> +# this TEST_OUTPUT_DIRECTORY.
> +expensive_lock_="$TEST_RESULTS_DIR/expensive.lock"
> +have_expensive_lock_=
> +
> +acquire_expensive_lock_ () {
> + local owner waited=
> + mkdir -p "$TEST_RESULTS_DIR" || return 1
> + while ! mkdir "$expensive_lock_" 2>/dev/null
> + do
> + owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
> + if test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
> + then
> + # The holder was killed without releasing the lock.
> + rm -rf "$expensive_lock_"
> + continue
> + fi
> + if test -z "$waited"
> + then
> + say >&3 "waiting for another EXPENSIVE test to finish"
> + waited=t
> + fi
> + sleep 1
> + done
> + echo $$ >"$expensive_lock_/pid"
> + have_expensive_lock_=t
> +}
Hmph, is the above safe when multiple processes are working with
expensive tests during normal (i.e., not involving SIGKILL)
operation, or does it still have TOCTOU race on the lock file?
* T1 holds $expensive_lock_, finishes its last EXPENSIVE test,
calls release_expensive_lock_ (rm -rf "$expensive_lock_"), and
exits.
* Waiter T2 read owner=$(cat "$expensive_lock_/pid" 2>/dev/null)
(obtaining T1's PID) just before T1 removed the directory.
* Waiter T3 runs mkdir "$expensive_lock_", succeeds, writes T3's
PID to pid, and begins its EXPENSIVE test.
* T2 now checks test -n "$owner" && ! kill -0 "$owner" 2>/dev/null
against T1's exited PID, finds that T1 is gone, runs rm -rf
"$expensive_lock_", which deletes T3's active lock. It loops,
succeeds on mkdir "$expensive_lock_", and runs its EXPENSIVE test
concurrently with T3.
Also, between the time you successfully mkdir the lock directory and
the time you leave this function, you can be killed by a signal with
INT, TERM, or HUP, for which t/test-lib.sh installs a hander to call
exit, and there is a "trap die EXIT".
You may manage to create $expensive_lock_ directory, open pid file
for writing, and then get killed before you echo $$ to the pid file
or assign 't' to have_expensive_lock_ variable.
When your die handler calls release_expensive_lock_, you would leave
with the directory behind and exit. You may not have pid file, or
the pid file may be empty. The waiting processes will not run the
kill -0 "$owner" part (because they do not know who you are that held
the lock and left before writing $$ in pid), the waiters will wait
for the mkdir to successfully create the directory that already
exists and nobody will ever remove, no?
Between the TOCTOU race allowing multiple processes entering the
expensive region and deadlocking denying everybody from entering the
expensive region, the latter is far worse, of course.
> +release_expensive_lock_ () {
> + test -n "$have_expensive_lock_" || return 0
> + rm -rf "$expensive_lock_"
> + have_expensive_lock_=
> +}
prev parent reply other threads:[~2026-11-20 6:28 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-19 7:40 [PATCH] test-lib: run EXPENSIVE tests one at a time Kaartic Sivaraam
2026-11-19 10:07 ` Patrick Steinhardt
2026-11-19 17:12 ` Kaartic Sivaraam
2026-11-19 17:29 ` [PATCH v2] " Kaartic Sivaraam
2026-11-19 17:33 ` Kaartic Sivaraam
2026-11-20 6:28 ` Junio C Hamano [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqqpkw0rp8i.fsf@gitster.g \
--to=gitster@pobox.com \
--cc=git@vger.kernel.org \
--cc=kaartic.sivaraam@gmail.com \
--cc=ps@pks.im \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox