git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Adam Dinwoodie <adam@dinwoodie.org>
To: git@vger.kernel.org
Cc: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
	"Fabian Stelzer" <fs@gigacodes.de>
Subject: [RFC PATCH] test-lib: allow storing counts with test harnesses
Date: Sat, 24 Dec 2022 22:52:00 +0000	[thread overview]
Message-ID: <20221224225200.1027806-1-adam@dinwoodie.org> (raw)

Currently, test result files are only stored in test-results/*.counts if
$HARNESS_ACTIVE is not set.  This dates from 8ef1abe550 (test-lib: Don't
write test-results when HARNESS_ACTIVE, 2010-08-11), where the
assumption was that if someone were using a test harness like prove,
that would track results and the count files wouldn't be required.
However, as of 49da404070 (test-lib: show missing prereq summary,
2021-11-20), those files also store the list of git test prerequisites
that were missing during the test run, which isn't something that a
generic test harness like prove can provide.

To allow folk using test harnesses to access the lists of missing
prerequisites, add a --counts argument to test-lib that will keep these
counts files even if a test harness is in use.  This means that a
subsequent call of, say, `make -C t aggregate-results` will report
useful information.

It might be preferable to do make a wider-ranging change, including
storing the missing prerequisites separately from the count files, so
the results can be reported regardless of whether the success/failure
counts are wanted, but that would be more disruptive and more work for
relatively little gain.

Signed-off-by: Adam Dinwoodie <adam@dinwoodie.org>
---

The key reason I'm submitting this as an RFC is that last paragraph:
I've tested the below patch, and it achieves what I'm after (letting me
both use prove and audit missing prerequisites), but it further embeds
the use of the "count" files for recording missing prerequisites, where
it might be preferable to do something a bit more complex that treats
the prerequisite reporting as a separate function, rather than a bolt-on
to the pass/fail/etc. counts.

 t/test-lib.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/t/test-lib.sh b/t/test-lib.sh
index 6db377f68b..bbd9ee0e34 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -157,6 +157,8 @@ parse_option () {
 	local opt="$1"
 
 	case "$opt" in
+	-c|--c|--co|--cou|--coun|--count|--counts)
+		record_counts=t ;;
 	-d|--d|--de|--deb|--debu|--debug)
 		debug=t ;;
 	-i|--i|--im|--imm|--imme|--immed|--immedi|--immedia|--immediat|--immediate)
@@ -1282,7 +1284,7 @@ test_done () {
 
 	finalize_test_output
 
-	if test -z "$HARNESS_ACTIVE"
+	if test -z "$HARNESS_ACTIVE" || test -n "$record_counts"
 	then
 		mkdir -p "$TEST_RESULTS_DIR"
 
-- 
2.39.0


             reply	other threads:[~2022-12-24 23:09 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-12-24 22:52 Adam Dinwoodie [this message]
2023-03-04 21:22 ` [PATCH] test-lib: allow storing counts with test harnesses Adam Dinwoodie
2023-03-06  8:57   ` Jeff King
2023-03-06 18:15     ` Junio C Hamano
2023-04-01 18:56   ` Elijah Newren

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20221224225200.1027806-1-adam@dinwoodie.org \
    --to=adam@dinwoodie.org \
    --cc=avarab@gmail.com \
    --cc=fs@gigacodes.de \
    --cc=git@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).