[PATCH 0/6] unit-tests: add and use TEST_RUN to simplify tests

[PATCH 0/6] unit-tests: add and use TEST_RUN to simplify tests

[René Scharfe]

The macro TEST only allows defining a test that consists of a single
expression.  This requires wrapping tests made up of one or more
statements in a function, which is a small, but avoidable hurdle.  This
series provides a new macro, TEST_RUN, that provides a way to define
tests without requiring to declare a function.

  t0080: move expected output to a file
  unit-tests: add TEST_RUN
  t-ctype: use TEST_RUN
  t-reftable-basics: use TEST_RUN
  t-strvec: use TEST_RUN
  t-strbuf: use TEST_RUN

 t/helper/test-example-tap.c      |  33 +++
 t/t0080-unit-test-output.sh      |  48 +----
 t/t0080/expect                   |  76 +++++++
 t/unit-tests/t-ctype.c           |   4 +-
 t/unit-tests/t-reftable-basics.c | 228 +++++++++-----------
 t/unit-tests/t-strbuf.c          |  79 +++----
 t/unit-tests/t-strvec.c          | 356 ++++++++++++++-----------------
 t/unit-tests/test-lib.c          |  42 +++-
 t/unit-tests/test-lib.h          |   8 +
 9 files changed, 462 insertions(+), 412 deletions(-)
 create mode 100644 t/t0080/expect

--
2.45.2

[PATCH 1/6] t0080: move expected output to a file

[René Scharfe]

Provide the expected output of "test-tool example-tap" verbatim instead
of as a here-doc, to avoid distractions due to quoting, variables
containing quotes and indentation.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/t0080-unit-test-output.sh | 48 +------------------------------------
 t/t0080/expect              | 43 +++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 47 deletions(-)
 create mode 100644 t/t0080/expect

diff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh
index 7bbb065d58..93f2defa19 100755
--- a/t/t0080-unit-test-output.sh
+++ b/t/t0080-unit-test-output.sh
@@ -6,54 +6,8 @@ TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh

 test_expect_success 'TAP output from unit tests' '
-	cat >expect <<-EOF &&
-	ok 1 - passing test
-	ok 2 - passing test and assertion return 1
-	# check "1 == 2" failed at t/helper/test-example-tap.c:77
-	#    left: 1
-	#   right: 2
-	not ok 3 - failing test
-	ok 4 - failing test and assertion return 0
-	not ok 5 - passing TEST_TODO() # TODO
-	ok 6 - passing TEST_TODO() returns 1
-	# todo check ${SQ}check(x)${SQ} succeeded at t/helper/test-example-tap.c:26
-	not ok 7 - failing TEST_TODO()
-	ok 8 - failing TEST_TODO() returns 0
-	# check "0" failed at t/helper/test-example-tap.c:31
-	# skipping test - missing prerequisite
-	# skipping check ${SQ}1${SQ} at t/helper/test-example-tap.c:33
-	ok 9 - test_skip() # SKIP
-	ok 10 - skipped test returns 1
-	# skipping test - missing prerequisite
-	ok 11 - test_skip() inside TEST_TODO() # SKIP
-	ok 12 - test_skip() inside TEST_TODO() returns 1
-	# check "0" failed at t/helper/test-example-tap.c:49
-	not ok 13 - TEST_TODO() after failing check
-	ok 14 - TEST_TODO() after failing check returns 0
-	# check "0" failed at t/helper/test-example-tap.c:57
-	not ok 15 - failing check after TEST_TODO()
-	ok 16 - failing check after TEST_TODO() returns 0
-	# check "!strcmp("\thello\\\\", "there\"\n")" failed at t/helper/test-example-tap.c:62
-	#    left: "\011hello\\\\"
-	#   right: "there\"\012"
-	# check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:63
-	#    left: "NULL"
-	#   right: NULL
-	# check "${SQ}a${SQ} == ${SQ}\n${SQ}" failed at t/helper/test-example-tap.c:64
-	#    left: ${SQ}a${SQ}
-	#   right: ${SQ}\012${SQ}
-	# check "${SQ}\\\\${SQ} == ${SQ}\\${SQ}${SQ}" failed at t/helper/test-example-tap.c:65
-	#    left: ${SQ}\\\\${SQ}
-	#   right: ${SQ}\\${SQ}${SQ}
-	not ok 17 - messages from failing string and char comparison
-	# BUG: test has no checks at t/helper/test-example-tap.c:92
-	not ok 18 - test with no checks
-	ok 19 - test with no checks returns 0
-	1..19
-	EOF
-
 	! test-tool example-tap >actual &&
-	test_cmp expect actual
+	test_cmp "$TEST_DIRECTORY"/t0080/expect actual
 '

 test_done
diff --git a/t/t0080/expect b/t/t0080/expect
new file mode 100644
index 0000000000..0cfa0dc6d8
--- /dev/null
+++ b/t/t0080/expect
@@ -0,0 +1,43 @@
+ok 1 - passing test
+ok 2 - passing test and assertion return 1
+# check "1 == 2" failed at t/helper/test-example-tap.c:77
+#    left: 1
+#   right: 2
+not ok 3 - failing test
+ok 4 - failing test and assertion return 0
+not ok 5 - passing TEST_TODO() # TODO
+ok 6 - passing TEST_TODO() returns 1
+# todo check 'check(x)' succeeded at t/helper/test-example-tap.c:26
+not ok 7 - failing TEST_TODO()
+ok 8 - failing TEST_TODO() returns 0
+# check "0" failed at t/helper/test-example-tap.c:31
+# skipping test - missing prerequisite
+# skipping check '1' at t/helper/test-example-tap.c:33
+ok 9 - test_skip() # SKIP
+ok 10 - skipped test returns 1
+# skipping test - missing prerequisite
+ok 11 - test_skip() inside TEST_TODO() # SKIP
+ok 12 - test_skip() inside TEST_TODO() returns 1
+# check "0" failed at t/helper/test-example-tap.c:49
+not ok 13 - TEST_TODO() after failing check
+ok 14 - TEST_TODO() after failing check returns 0
+# check "0" failed at t/helper/test-example-tap.c:57
+not ok 15 - failing check after TEST_TODO()
+ok 16 - failing check after TEST_TODO() returns 0
+# check "!strcmp("\thello\\", "there\"\n")" failed at t/helper/test-example-tap.c:62
+#    left: "\011hello\\"
+#   right: "there\"\012"
+# check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:63
+#    left: "NULL"
+#   right: NULL
+# check "'a' == '\n'" failed at t/helper/test-example-tap.c:64
+#    left: 'a'
+#   right: '\012'
+# check "'\\' == '\''" failed at t/helper/test-example-tap.c:65
+#    left: '\\'
+#   right: '\''
+not ok 17 - messages from failing string and char comparison
+# BUG: test has no checks at t/helper/test-example-tap.c:92
+not ok 18 - test with no checks
+ok 19 - test with no checks returns 0
+1..19
--
2.45.2

[PATCH 2/6] unit-tests: add TEST_RUN

[René Scharfe]

The macro TEST only allows defining a test that consists of a single
expression.  Add the new macro, TEST_RUN, which provides a way to define
unit tests that are made up of one or more statements.  A test started
with it implicitly ends when the next test is started or test_done() is
called.

TEST_RUN allows defining self-contained tests en bloc, a bit like
test_expect_success does for regular tests.  Unlike TEST it does not
require defining wrapper functions for test statements.

No public method is provided for ending a test explicitly, yet; let's
see if we'll ever need one.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/helper/test-example-tap.c | 33 +++++++++++++++++++++++++++++
 t/t0080/expect              | 35 ++++++++++++++++++++++++++++++-
 t/unit-tests/test-lib.c     | 42 +++++++++++++++++++++++++++++++++++--
 t/unit-tests/test-lib.h     |  8 +++++++
 4 files changed, 115 insertions(+), 3 deletions(-)

diff --git a/t/helper/test-example-tap.c b/t/helper/test-example-tap.c
index d072ad559f..7b02177a9f 100644
--- a/t/helper/test-example-tap.c
+++ b/t/helper/test-example-tap.c
@@ -92,5 +92,38 @@ int cmd__example_tap(int argc, const char **argv)
 	test_res = TEST(t_empty(), "test with no checks");
 	TEST(check_int(test_res, ==, 0), "test with no checks returns 0");

+	if (TEST_RUN("TEST_RUN passing test"))
+		check_int(1, ==, 1);
+	if (TEST_RUN("TEST_RUN failing test"))
+		check_int(1, ==, 2);
+	if (TEST_RUN("TEST_RUN passing TEST_TODO()"))
+		TEST_TODO(check(0));
+	if (TEST_RUN("TEST_RUN failing TEST_TODO()"))
+		TEST_TODO(check(1));
+	if (TEST_RUN("TEST_RUN test_skip()")) {
+		check(0);
+		test_skip("missing prerequisite");
+		check(1);
+	}
+	if (TEST_RUN("TEST_RUN test_skip() inside TEST_TODO()"))
+		TEST_TODO((test_skip("missing prerequisite"), 1));
+	if (TEST_RUN("TEST_RUN TEST_TODO() after failing check")) {
+		check(0);
+		TEST_TODO(check(0));
+	}
+	if (TEST_RUN("TEST_RUN failing check after TEST_TODO()")) {
+		check(1);
+		TEST_TODO(check(0));
+		check(0);
+	}
+	if (TEST_RUN("TEST_RUN messages from failing string and char comparison")) {
+		check_str("\thello\\", "there\"\n");
+		check_str("NULL", NULL);
+		check_char('a', ==, '\n');
+		check_char('\\', ==, '\'');
+	}
+	if (TEST_RUN("TEST_RUN test with no checks"))
+		; /* nothing */
+
 	return test_done();
 }
diff --git a/t/t0080/expect b/t/t0080/expect
index 0cfa0dc6d8..92526e1b06 100644
--- a/t/t0080/expect
+++ b/t/t0080/expect
@@ -40,4 +40,37 @@ not ok 17 - messages from failing string and char comparison
 # BUG: test has no checks at t/helper/test-example-tap.c:92
 not ok 18 - test with no checks
 ok 19 - test with no checks returns 0
-1..19
+ok 20 - TEST_RUN passing test
+# check "1 == 2" failed at t/helper/test-example-tap.c:98
+#    left: 1
+#   right: 2
+not ok 21 - TEST_RUN failing test
+not ok 22 - TEST_RUN passing TEST_TODO() # TODO
+# todo check 'check(1)' succeeded at t/helper/test-example-tap.c:102
+not ok 23 - TEST_RUN failing TEST_TODO()
+# check "0" failed at t/helper/test-example-tap.c:104
+# skipping test - missing prerequisite
+# skipping check '1' at t/helper/test-example-tap.c:106
+ok 24 - TEST_RUN test_skip() # SKIP
+# skipping test - missing prerequisite
+ok 25 - TEST_RUN test_skip() inside TEST_TODO() # SKIP
+# check "0" failed at t/helper/test-example-tap.c:111
+not ok 26 - TEST_RUN TEST_TODO() after failing check
+# check "0" failed at t/helper/test-example-tap.c:117
+not ok 27 - TEST_RUN failing check after TEST_TODO()
+# check "!strcmp("\thello\\", "there\"\n")" failed at t/helper/test-example-tap.c:120
+#    left: "\011hello\\"
+#   right: "there\"\012"
+# check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:121
+#    left: "NULL"
+#   right: NULL
+# check "'a' == '\n'" failed at t/helper/test-example-tap.c:122
+#    left: 'a'
+#   right: '\012'
+# check "'\\' == '\''" failed at t/helper/test-example-tap.c:123
+#    left: '\\'
+#   right: '\''
+not ok 28 - TEST_RUN messages from failing string and char comparison
+# BUG: test has no checks at t/helper/test-example-tap.c:125
+not ok 29 - TEST_RUN test with no checks
+1..29
diff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c
index 3c513ce59a..fc50755fae 100644
--- a/t/unit-tests/test-lib.c
+++ b/t/unit-tests/test-lib.c
@@ -16,6 +16,8 @@ static struct {
 	unsigned running :1;
 	unsigned skip_all :1;
 	unsigned todo :1;
+	char *desc;
+	char *location;
 } ctx = {
 	.lazy_plan = 1,
 	.result = RESULT_NONE,
@@ -123,9 +125,45 @@ void test_plan(int count)
 	ctx.lazy_plan = 0;
 }

-int test_done(void)
+static void test__run_maybe_end(void)
 {
+	if (ctx.running) {
+		assert(ctx.location);
+		assert(ctx.desc);
+		test__run_end(0, ctx.location, "%s", ctx.desc);
+		FREE_AND_NULL(ctx.location);
+		FREE_AND_NULL(ctx.desc);
+	}
 	assert(!ctx.running);
+	assert(!ctx.location);
+	assert(!ctx.desc);
+}
+
+int test__run(const char *location, const char *format, ...)
+{
+	va_list ap;
+	char *desc;
+
+	test__run_maybe_end();
+
+	va_start(ap, format);
+	desc = xstrvfmt(format, ap);
+	va_end(ap);
+
+	if (test__run_begin()) {
+		test__run_end(1, location, "%s", desc);
+		free(desc);
+		return 0;
+	} else {
+		ctx.location = xstrdup(location);
+		ctx.desc = desc;
+		return 1;
+	}
+}
+
+int test_done(void)
+{
+	test__run_maybe_end();

 	if (ctx.lazy_plan)
 		test_plan(ctx.count);
@@ -169,7 +207,7 @@ void test_skip_all(const char *format, ...)

 int test__run_begin(void)
 {
-	assert(!ctx.running);
+	test__run_maybe_end();

 	ctx.count++;
 	ctx.result = RESULT_NONE;
diff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h
index 2de6d715d5..6df40e3b12 100644
--- a/t/unit-tests/test-lib.h
+++ b/t/unit-tests/test-lib.h
@@ -21,6 +21,13 @@
  */
 void test_plan(int count);

+/*
+ * Start a test, returns 1 if the test was actually started or 0 if it
+ * was skipped.  The test ends when the next test starts or test_done()
+ * is called.
+ */
+#define TEST_RUN(...) test__run(TEST_LOCATION(), __VA_ARGS__)
+
 /*
  * test_done() must be called at the end of main(). It will print the
  * plan if plan() was not called at the beginning of the test program
@@ -156,6 +163,7 @@ extern union test__tmp test__tmp[2];
 int test__run_begin(void);
 __attribute__((format (printf, 3, 4)))
 int test__run_end(int, const char *, const char *, ...);
+int test__run(const char *location, const char *format, ...);
 void test__todo_begin(void);
 int test__todo_end(const char *, const char *, int);

--
2.45.2

[PATCH 3/6] t-ctype: use TEST_RUN

[René Scharfe]

Use the macro TEST_RUN instead of the internal functions
test__run_begin() and test__run_end().

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-ctype.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/t/unit-tests/t-ctype.c b/t/unit-tests/t-ctype.c
index d6ac1fe678..04a66dc105 100644
--- a/t/unit-tests/t-ctype.c
+++ b/t/unit-tests/t-ctype.c
@@ -4,15 +4,13 @@
 	size_t len = ARRAY_SIZE(string) - 1 + \
 		BUILD_ASSERT_OR_ZERO(ARRAY_SIZE(string) > 0) + \
 		BUILD_ASSERT_OR_ZERO(sizeof(string[0]) == sizeof(char)); \
-	int skip = test__run_begin(); \
-	if (!skip) { \
+	if (TEST_RUN(#class " works")) { \
 		for (int i = 0; i < 256; i++) { \
 			if (!check_int(class(i), ==, !!memchr(string, i, len)))\
 				test_msg("      i: 0x%02x", i); \
 		} \
 		check(!class(EOF)); \
 	} \
-	test__run_end(!skip, TEST_LOCATION(), #class " works"); \
 } while (0)

 #define DIGIT "0123456789"
--
2.45.2

[PATCH 4/6] t-reftable-basics: use TEST_RUN

[René Scharfe]

The macro TEST takes a single expression.  If a test requires multiple
statements then they need to be placed in a function that's called in
the TEST expression.

Remove the overhead of defining and calling single-use functions by
using TEST_RUN instead.

Run the tests in the order of definition.  We can reorder them like that
because they are independent.  Technically this changes the output, but
retains the meaning of a full run and allows for easier review e.g. with
diff option --ignore-all-space.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-reftable-basics.c | 228 ++++++++++++++-----------------
 1 file changed, 106 insertions(+), 122 deletions(-)

diff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c
index 4e80bdf16d..e6e789e741 100644
--- a/t/unit-tests/t-reftable-basics.c
+++ b/t/unit-tests/t-reftable-basics.c
@@ -20,141 +20,125 @@ static int integer_needle_lesseq(size_t i, void *_args)
 	return args->needle <= args->haystack[i];
 }

-static void test_binsearch(void)
+int cmd_main(int argc, const char *argv[])
 {
-	int haystack[] = { 2, 4, 6, 8, 10 };
-	struct {
-		int needle;
-		size_t expected_idx;
-	} testcases[] = {
-		{-9000, 0},
-		{-1, 0},
-		{0, 0},
-		{2, 0},
-		{3, 1},
-		{4, 1},
-		{7, 3},
-		{9, 4},
-		{10, 4},
-		{11, 5},
-		{9000, 5},
-	};
-
-	for (size_t i = 0; i < ARRAY_SIZE(testcases); i++) {
-		struct integer_needle_lesseq_args args = {
-			.haystack = haystack,
-			.needle = testcases[i].needle,
+	if (TEST_RUN("binary search with binsearch works")) {
+		int haystack[] = { 2, 4, 6, 8, 10 };
+		struct {
+			int needle;
+			size_t expected_idx;
+		} testcases[] = {
+			{-9000, 0},
+			{-1, 0},
+			{0, 0},
+			{2, 0},
+			{3, 1},
+			{4, 1},
+			{7, 3},
+			{9, 4},
+			{10, 4},
+			{11, 5},
+			{9000, 5},
 		};
-		size_t idx;

-		idx = binsearch(ARRAY_SIZE(haystack), &integer_needle_lesseq, &args);
-		check_int(idx, ==, testcases[i].expected_idx);
+		for (size_t i = 0; i < ARRAY_SIZE(testcases); i++) {
+			struct integer_needle_lesseq_args args = {
+				.haystack = haystack,
+				.needle = testcases[i].needle,
+			};
+			size_t idx;
+
+			idx = binsearch(ARRAY_SIZE(haystack),
+					&integer_needle_lesseq, &args);
+			check_int(idx, ==, testcases[i].expected_idx);
+		}
 	}
-}

-static void test_names_length(void)
-{
-	const char *a[] = { "a", "b", NULL };
-	check_int(names_length(a), ==, 2);
-}
-
-static void test_names_equal(void)
-{
-	const char *a[] = { "a", "b", "c", NULL };
-	const char *b[] = { "a", "b", "d", NULL };
-	const char *c[] = { "a", "b", NULL };
+	if (TEST_RUN("names_length retuns size of a NULL-terminated string array")) {
+		const char *a[] = { "a", "b", NULL };
+		check_int(names_length(a), ==, 2);
+	}

-	check(names_equal(a, a));
-	check(!names_equal(a, b));
-	check(!names_equal(a, c));
-}
+	if (TEST_RUN("names_equal compares NULL-terminated string arrays")) {
+		const char *a[] = { "a", "b", "c", NULL };
+		const char *b[] = { "a", "b", "d", NULL };
+		const char *c[] = { "a", "b", NULL };

-static void test_parse_names_normal(void)
-{
-	char in1[] = "line\n";
-	char in2[] = "a\nb\nc";
-	char **out = NULL;
-	parse_names(in1, strlen(in1), &out);
-	check_str(out[0], "line");
-	check(!out[1]);
-	free_names(out);
-
-	parse_names(in2, strlen(in2), &out);
-	check_str(out[0], "a");
-	check_str(out[1], "b");
-	check_str(out[2], "c");
-	check(!out[3]);
-	free_names(out);
-}
+		check(names_equal(a, a));
+		check(!names_equal(a, b));
+		check(!names_equal(a, c));
+	}

-static void test_parse_names_drop_empty(void)
-{
-	char in[] = "a\n\nb\n";
-	char **out = NULL;
-	parse_names(in, strlen(in), &out);
-	check_str(out[0], "a");
-	/* simply '\n' should be dropped as empty string */
-	check_str(out[1], "b");
-	check(!out[2]);
-	free_names(out);
-}
+	if (TEST_RUN("parse_names works for basic input")) {
+		char in1[] = "line\n";
+		char in2[] = "a\nb\nc";
+		char **out = NULL;
+		parse_names(in1, strlen(in1), &out);
+		check_str(out[0], "line");
+		check(!out[1]);
+		free_names(out);
+
+		parse_names(in2, strlen(in2), &out);
+		check_str(out[0], "a");
+		check_str(out[1], "b");
+		check_str(out[2], "c");
+		check(!out[3]);
+		free_names(out);
+	}

-static void test_common_prefix(void)
-{
-	struct strbuf a = STRBUF_INIT;
-	struct strbuf b = STRBUF_INIT;
-	struct {
-		const char *a, *b;
-		int want;
-	} cases[] = {
-		{"abcdef", "abc", 3},
-		{ "abc", "ab", 2 },
-		{ "", "abc", 0 },
-		{ "abc", "abd", 2 },
-		{ "abc", "pqr", 0 },
-	};
-
-	for (size_t i = 0; i < ARRAY_SIZE(cases); i++) {
-		strbuf_addstr(&a, cases[i].a);
-		strbuf_addstr(&b, cases[i].b);
-		check_int(common_prefix_size(&a, &b), ==, cases[i].want);
-		strbuf_reset(&a);
-		strbuf_reset(&b);
+	if (TEST_RUN("parse_names drops empty string")) {
+		char in[] = "a\n\nb\n";
+		char **out = NULL;
+		parse_names(in, strlen(in), &out);
+		check_str(out[0], "a");
+		/* simply '\n' should be dropped as empty string */
+		check_str(out[1], "b");
+		check(!out[2]);
+		free_names(out);
 	}
-	strbuf_release(&a);
-	strbuf_release(&b);
-}

-static void test_u24_roundtrip(void)
-{
-	uint32_t in = 0x112233;
-	uint8_t dest[3];
-	uint32_t out;
-	put_be24(dest, in);
-	out = get_be24(dest);
-	check_int(in, ==, out);
-}
+	if (TEST_RUN("common_prefix_size works")) {
+		struct strbuf a = STRBUF_INIT;
+		struct strbuf b = STRBUF_INIT;
+		struct {
+			const char *a, *b;
+			int want;
+		} cases[] = {
+			{"abcdef", "abc", 3},
+			{ "abc", "ab", 2 },
+			{ "", "abc", 0 },
+			{ "abc", "abd", 2 },
+			{ "abc", "pqr", 0 },
+		};

-static void test_u16_roundtrip(void)
-{
-	uint32_t in = 0xfef1;
-	uint8_t dest[3];
-	uint32_t out;
-	put_be16(dest, in);
-	out = get_be16(dest);
-	check_int(in, ==, out);
-}
+		for (size_t i = 0; i < ARRAY_SIZE(cases); i++) {
+			strbuf_addstr(&a, cases[i].a);
+			strbuf_addstr(&b, cases[i].b);
+			check_int(common_prefix_size(&a, &b), ==, cases[i].want);
+			strbuf_reset(&a);
+			strbuf_reset(&b);
+		}
+		strbuf_release(&a);
+		strbuf_release(&b);
+	}

-int cmd_main(int argc, const char *argv[])
-{
-	TEST(test_common_prefix(), "common_prefix_size works");
-	TEST(test_parse_names_normal(), "parse_names works for basic input");
-	TEST(test_parse_names_drop_empty(), "parse_names drops empty string");
-	TEST(test_binsearch(), "binary search with binsearch works");
-	TEST(test_names_length(), "names_length retuns size of a NULL-terminated string array");
-	TEST(test_names_equal(), "names_equal compares NULL-terminated string arrays");
-	TEST(test_u24_roundtrip(), "put_be24 and get_be24 work");
-	TEST(test_u16_roundtrip(), "put_be16 and get_be16 work");
+	if (TEST_RUN("put_be24 and get_be24 work")) {
+		uint32_t in = 0x112233;
+		uint8_t dest[3];
+		uint32_t out;
+		put_be24(dest, in);
+		out = get_be24(dest);
+		check_int(in, ==, out);
+	}
+
+	if (TEST_RUN("put_be16 and get_be16 work")) {
+		uint32_t in = 0xfef1;
+		uint8_t dest[3];
+		uint32_t out;
+		put_be16(dest, in);
+		out = get_be16(dest);
+		check_int(in, ==, out);
+	}

 	return test_done();
 }
--
2.45.2

[PATCH 5/6] t-strvec: use TEST_RUN

[René Scharfe]

The macro TEST takes a single expression.  If a test requires multiple
statements then they need to be placed in a function that's called in
the TEST expression.

Remove the overhead of defining and calling single-use functions by
using TEST_RUN instead.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-strvec.c | 356 ++++++++++++++++++----------------------
 1 file changed, 156 insertions(+), 200 deletions(-)

diff --git a/t/unit-tests/t-strvec.c b/t/unit-tests/t-strvec.c
index d4615ab06d..00ff7d4ae8 100644
--- a/t/unit-tests/t-strvec.c
+++ b/t/unit-tests/t-strvec.c
@@ -36,237 +36,193 @@ static void check_strvec_loc(const char *loc, struct strvec *vec, ...)
 	check_pointer_eq(vec->v[nr], NULL);
 }

-static void t_static_init(void)
+int cmd_main(int argc, const char **argv)
 {
-	struct strvec vec = STRVEC_INIT;
-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
-}
+	if (TEST_RUN("static initialization")) {
+		struct strvec vec = STRVEC_INIT;
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);
+	}

-static void t_dynamic_init(void)
-{
-	struct strvec vec;
-	strvec_init(&vec);
-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
-}
+	if (TEST_RUN("dynamic initialization")) {
+		struct strvec vec;
+		strvec_init(&vec);
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);
+	}

-static void t_clear(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_push(&vec, "foo");
-	strvec_clear(&vec);
-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
-}
+	if (TEST_RUN("clear")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_push(&vec, "foo");
+		strvec_clear(&vec);
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);
+	}

-static void t_push(void)
-{
-	struct strvec vec = STRVEC_INIT;
+	if (TEST_RUN("push")) {
+		struct strvec vec = STRVEC_INIT;

-	strvec_push(&vec, "foo");
-	check_strvec(&vec, "foo", NULL);
+		strvec_push(&vec, "foo");
+		check_strvec(&vec, "foo", NULL);

-	strvec_push(&vec, "bar");
-	check_strvec(&vec, "foo", "bar", NULL);
+		strvec_push(&vec, "bar");
+		check_strvec(&vec, "foo", "bar", NULL);

-	strvec_clear(&vec);
-}
+		strvec_clear(&vec);
+	}

-static void t_pushf(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushf(&vec, "foo: %d", 1);
-	check_strvec(&vec, "foo: 1", NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("pushf")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushf(&vec, "foo: %d", 1);
+		check_strvec(&vec, "foo: 1", NULL);
+		strvec_clear(&vec);
+	}

-static void t_pushl(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	check_strvec(&vec, "foo", "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("pushl")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		check_strvec(&vec, "foo", "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_pushv(void)
-{
-	const char *strings[] = {
-		"foo", "bar", "baz", NULL,
-	};
-	struct strvec vec = STRVEC_INIT;
+	if (TEST_RUN("pushv")) {
+		const char *strings[] = {
+			"foo", "bar", "baz", NULL,
+		};
+		struct strvec vec = STRVEC_INIT;

-	strvec_pushv(&vec, strings);
-	check_strvec(&vec, "foo", "bar", "baz", NULL);
+		strvec_pushv(&vec, strings);
+		check_strvec(&vec, "foo", "bar", "baz", NULL);

-	strvec_clear(&vec);
-}
+		strvec_clear(&vec);
+	}

-static void t_replace_at_head(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_replace(&vec, 0, "replaced");
-	check_strvec(&vec, "replaced", "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("replace at head")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_replace(&vec, 0, "replaced");
+		check_strvec(&vec, "replaced", "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_replace_at_tail(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_replace(&vec, 2, "replaced");
-	check_strvec(&vec, "foo", "bar", "replaced", NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("replace at tail")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_replace(&vec, 2, "replaced");
+		check_strvec(&vec, "foo", "bar", "replaced", NULL);
+		strvec_clear(&vec);
+	}

-static void t_replace_in_between(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_replace(&vec, 1, "replaced");
-	check_strvec(&vec, "foo", "replaced", "baz", NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("replace in between")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_replace(&vec, 1, "replaced");
+		check_strvec(&vec, "foo", "replaced", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_replace_with_substring(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", NULL);
-	strvec_replace(&vec, 0, vec.v[0] + 1);
-	check_strvec(&vec, "oo", NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("replace with substring")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", NULL);
+		strvec_replace(&vec, 0, vec.v[0] + 1);
+		check_strvec(&vec, "oo", NULL);
+		strvec_clear(&vec);
+	}

-static void t_remove_at_head(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_remove(&vec, 0);
-	check_strvec(&vec, "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("remove at head")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_remove(&vec, 0);
+		check_strvec(&vec, "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_remove_at_tail(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_remove(&vec, 2);
-	check_strvec(&vec, "foo", "bar", NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("remove at tail")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_remove(&vec, 2);
+		check_strvec(&vec, "foo", "bar", NULL);
+		strvec_clear(&vec);
+	}

-static void t_remove_in_between(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_remove(&vec, 1);
-	check_strvec(&vec, "foo", "baz", NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("remove in between")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_remove(&vec, 1);
+		check_strvec(&vec, "foo", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_pop_empty_array(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pop(&vec);
-	check_strvec(&vec, NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("pop with empty array")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pop(&vec);
+		check_strvec(&vec, NULL);
+		strvec_clear(&vec);
+	}

-static void t_pop_non_empty_array(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_pop(&vec);
-	check_strvec(&vec, "foo", "bar", NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("pop with non-empty array")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_pop(&vec);
+		check_strvec(&vec, "foo", "bar", NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_empty_string(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "");
-	check_strvec(&vec, NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("split empty string")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "");
+		check_strvec(&vec, NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_single_item(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "foo");
-	check_strvec(&vec, "foo", NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("split single item")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "foo");
+		check_strvec(&vec, "foo", NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_multiple_items(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "foo bar baz");
-	check_strvec(&vec, "foo", "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("split multiple items")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "foo bar baz");
+		check_strvec(&vec, "foo", "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_whitespace_only(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, " \t\n");
-	check_strvec(&vec, NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("split whitespace only")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, " \t\n");
+		check_strvec(&vec, NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_multiple_consecutive_whitespaces(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "foo\n\t bar");
-	check_strvec(&vec, "foo", "bar", NULL);
-	strvec_clear(&vec);
-}
+	if (TEST_RUN("split multiple consecutive whitespaces")) {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "foo\n\t bar");
+		check_strvec(&vec, "foo", "bar", NULL);
+		strvec_clear(&vec);
+	}

-static void t_detach(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	const char **detached;
+	if (TEST_RUN("detach")) {
+		struct strvec vec = STRVEC_INIT;
+		const char **detached;

-	strvec_push(&vec, "foo");
+		strvec_push(&vec, "foo");

-	detached = strvec_detach(&vec);
-	check_str(detached[0], "foo");
-	check_pointer_eq(detached[1], NULL);
+		detached = strvec_detach(&vec);
+		check_str(detached[0], "foo");
+		check_pointer_eq(detached[1], NULL);

-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);

-	free((char *) detached[0]);
-	free(detached);
-}
+		free((char *) detached[0]);
+		free(detached);
+	}

-int cmd_main(int argc, const char **argv)
-{
-	TEST(t_static_init(), "static initialization");
-	TEST(t_dynamic_init(), "dynamic initialization");
-	TEST(t_clear(), "clear");
-	TEST(t_push(), "push");
-	TEST(t_pushf(), "pushf");
-	TEST(t_pushl(), "pushl");
-	TEST(t_pushv(), "pushv");
-	TEST(t_replace_at_head(), "replace at head");
-	TEST(t_replace_in_between(), "replace in between");
-	TEST(t_replace_at_tail(), "replace at tail");
-	TEST(t_replace_with_substring(), "replace with substring");
-	TEST(t_remove_at_head(), "remove at head");
-	TEST(t_remove_in_between(), "remove in between");
-	TEST(t_remove_at_tail(), "remove at tail");
-	TEST(t_pop_empty_array(), "pop with empty array");
-	TEST(t_pop_non_empty_array(), "pop with non-empty array");
-	TEST(t_split_empty_string(), "split empty string");
-	TEST(t_split_single_item(), "split single item");
-	TEST(t_split_multiple_items(), "split multiple items");
-	TEST(t_split_whitespace_only(), "split whitespace only");
-	TEST(t_split_multiple_consecutive_whitespaces(), "split multiple consecutive whitespaces");
-	TEST(t_detach(), "detach");
 	return test_done();
 }
--
2.45.2

[PATCH 6/6] t-strbuf: use TEST_RUN

[René Scharfe]

The macro TEST takes a single expression.  If a test requires multiple
statements then they need to be placed in a function that's called in
the TEST expression.  The functions setup() and setup_populated() here
are used for that purpose and take another function as an argument,
making the control flow hard to follow.

Remove the overhead of these functions by using TEST_RUN instead.  Move
their duplicate post-condition checks into a new helper, t_release(),
and let t_addch() and t_addstr() accept properly typed input parameters
instead of void pointers.

Use the fully checking t_addstr() for adding initial values instead of
only doing only a length comparison -- there's no need for skipping the
other checks.

This results in test cases that look much more like strbuf usage in
production code, only with checked strbuf functions replaced by checking
wrappers.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-strbuf.c | 79 +++++++++++++++++++++--------------------
 1 file changed, 41 insertions(+), 38 deletions(-)

diff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c
index 6027dafef7..c8e39ddda7 100644
--- a/t/unit-tests/t-strbuf.c
+++ b/t/unit-tests/t-strbuf.c
@@ -1,32 +1,6 @@
 #include "test-lib.h"
 #include "strbuf.h"

-/* wrapper that supplies tests with an empty, initialized strbuf */
-static void setup(void (*f)(struct strbuf*, const void*),
-		  const void *data)
-{
-	struct strbuf buf = STRBUF_INIT;
-
-	f(&buf, data);
-	strbuf_release(&buf);
-	check_uint(buf.len, ==, 0);
-	check_uint(buf.alloc, ==, 0);
-}
-
-/* wrapper that supplies tests with a populated, initialized strbuf */
-static void setup_populated(void (*f)(struct strbuf*, const void*),
-			    const char *init_str, const void *data)
-{
-	struct strbuf buf = STRBUF_INIT;
-
-	strbuf_addstr(&buf, init_str);
-	check_uint(buf.len, ==, strlen(init_str));
-	f(&buf, data);
-	strbuf_release(&buf);
-	check_uint(buf.len, ==, 0);
-	check_uint(buf.alloc, ==, 0);
-}
-
 static int assert_sane_strbuf(struct strbuf *buf)
 {
 	/* Initialized strbufs should always have a non-NULL buffer */
@@ -66,10 +40,8 @@ static void t_dynamic_init(void)
 	strbuf_release(&buf);
 }

-static void t_addch(struct strbuf *buf, const void *data)
+static void t_addch(struct strbuf *buf, int ch)
 {
-	const char *p_ch = data;
-	const char ch = *p_ch;
 	size_t orig_alloc = buf->alloc;
 	size_t orig_len = buf->len;

@@ -85,9 +57,8 @@ static void t_addch(struct strbuf *buf, const void *data)
 	check_char(buf->buf[buf->len], ==, '\0');
 }

-static void t_addstr(struct strbuf *buf, const void *data)
+static void t_addstr(struct strbuf *buf, const char *text)
 {
-	const char *text = data;
 	size_t len = strlen(text);
 	size_t orig_alloc = buf->alloc;
 	size_t orig_len = buf->len;
@@ -105,18 +76,50 @@ static void t_addstr(struct strbuf *buf, const void *data)
 	check_str(buf->buf + orig_len, text);
 }

+static void t_release(struct strbuf *sb)
+{
+	strbuf_release(sb);
+	check_uint(sb->len, ==, 0);
+	check_uint(sb->alloc, ==, 0);
+}
+
 int cmd_main(int argc, const char **argv)
 {
 	if (!TEST(t_static_init(), "static initialization works"))
 		test_skip_all("STRBUF_INIT is broken");
 	TEST(t_dynamic_init(), "dynamic initialization works");
-	TEST(setup(t_addch, "a"), "strbuf_addch adds char");
-	TEST(setup(t_addch, ""), "strbuf_addch adds NUL char");
-	TEST(setup_populated(t_addch, "initial value", "a"),
-	     "strbuf_addch appends to initial value");
-	TEST(setup(t_addstr, "hello there"), "strbuf_addstr adds string");
-	TEST(setup_populated(t_addstr, "initial value", "hello there"),
-	     "strbuf_addstr appends string to initial value");
+
+	if (TEST_RUN("strbuf_addch adds char")) {
+		struct strbuf sb = STRBUF_INIT;
+		t_addch(&sb, 'a');
+		t_release(&sb);
+	}
+
+	if (TEST_RUN("strbuf_addch adds NUL char")) {
+		struct strbuf sb = STRBUF_INIT;
+		t_addch(&sb, '\0');
+		t_release(&sb);
+	}
+
+	if (TEST_RUN("strbuf_addch appends to initial value")) {
+		struct strbuf sb = STRBUF_INIT;
+		t_addstr(&sb, "initial value");
+		t_addch(&sb, 'a');
+		t_release(&sb);
+	}
+
+	if (TEST_RUN("strbuf_addstr adds string")) {
+		struct strbuf sb = STRBUF_INIT;
+		t_addstr(&sb, "hello there");
+		t_release(&sb);
+	}
+
+	if (TEST_RUN("strbuf_addstr appends string to initial value")) {
+		struct strbuf sb = STRBUF_INIT;
+		t_addstr(&sb, "initial value");
+		t_addstr(&sb, "hello there");
+		t_release(&sb);
+	}

 	return test_done();
 }
--
2.45.2

Re: [PATCH 1/6] t0080: move expected output to a file

[Jeff King]

On Sat, Jun 29, 2024 at 05:35:31PM +0200, René Scharfe wrote:

> Provide the expected output of "test-tool example-tap" verbatim instead
> of as a here-doc, to avoid distractions due to quoting, variables
> containing quotes and indentation.

I'm not really opposed to this patch, but I wondered...

>  test_expect_success 'TAP output from unit tests' '
> -	cat >expect <<-EOF &&
> -	ok 1 - passing test
> -	ok 2 - passing test and assertion return 1

If you could take the test input on stdin, like so:

  test_expect_success 'TAP output from unit tests' - <<-\EOT
	cat >expect <<-\EOF
	ok 1 - passing test
	ok 2 - passing test and assertion return 1
	[...]
	# check "'a' == '\n'" failed at t/helper/test-example-tap.c:64
	#    left: 'a'
	#   right: '\012'
	[...]
	EOF
  EOT

would that be preferable to moving it to its own file? I kind of like
keeping everything in the test scripts themselves so related changes can
happen side-by-side, though I admit in this case it is intimately tied
to the separate test-example-tap.c source anyway.

But I do have such an "EOT" patch which I've been meaning to send out,
since it makes many of these quoting annoyances go away (though of
course it leaves the indentation).

-Peff

Re: [PATCH 1/6] t0080: move expected output to a file

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Sat, Jun 29, 2024 at 05:35:31PM +0200, René Scharfe wrote:
>
>> Provide the expected output of "test-tool example-tap" verbatim instead
>> of as a here-doc, to avoid distractions due to quoting, variables
>> containing quotes and indentation.
>
> I'm not really opposed to this patch, but I wondered...
>
>>  test_expect_success 'TAP output from unit tests' '
>> -	cat >expect <<-EOF &&
>> -	ok 1 - passing test
>> -	ok 2 - passing test and assertion return 1
>
> If you could take the test input on stdin, like so:
>
>   test_expect_success 'TAP output from unit tests' - <<-\EOT
> 	cat >expect <<-\EOF
> 	ok 1 - passing test
> 	ok 2 - passing test and assertion return 1
> 	[...]
> 	# check "'a' == '\n'" failed at t/helper/test-example-tap.c:64
> 	#    left: 'a'
> 	#   right: '\012'
> 	[...]
> 	EOF
>   EOT
>
> would that be preferable to moving it to its own file? I kind of like
> keeping everything in the test scripts themselves so related changes can
> happen side-by-side, though I admit in this case it is intimately tied
> to the separate test-example-tap.c source anyway.

Yeah, it does feel a bit of cop-out to separate the expectation out
to an external file.  I guess I was to blame for things like t4013
but there is a valid excuse there (it would be expected that similar
tests would need to be added and one test per one expected result was
a natural way to manage hundreds of tests).

In this case, I think the fact that validating the test framework is
an oddball use case is a sufficient excuse ;-).

> But I do have such an "EOT" patch which I've been meaning to send out,
> since it makes many of these quoting annoyances go away (though of
> course it leaves the indentation).

I am not sure about your "test body comes from the standard input"
(not saying "I am not convinced it is a good idea" or even "I am
convinced it is a bad idea"---I do not know what to think about it,
not just yet).  THe above illustration does make it easier to grok
by keeping everything in one place.

Thanks.

Re: [PATCH 3/6] t-ctype: use TEST_RUN

[Josh Steadmon]

On 2024.06.29 17:44, René Scharfe wrote:
> Use the macro TEST_RUN instead of the internal functions
> test__run_begin() and test__run_end().
> 
> Signed-off-by: René Scharfe <l.s.r@web.de>

Nitpick: please expand the commit message here as you did for the
following patches.

Re: [PATCH 1/6] t0080: move expected output to a file

[René Scharfe]

Am 01.07.24 um 05:20 schrieb Jeff King:
> On Sat, Jun 29, 2024 at 05:35:31PM +0200, René Scharfe wrote:
>
>> Provide the expected output of "test-tool example-tap" verbatim instead
>> of as a here-doc, to avoid distractions due to quoting, variables
>> containing quotes and indentation.
>
> I'm not really opposed to this patch, but I wondered...
>
>>  test_expect_success 'TAP output from unit tests' '
>> -	cat >expect <<-EOF &&
>> -	ok 1 - passing test
>> -	ok 2 - passing test and assertion return 1
>
> If you could take the test input on stdin, like so:
>
>   test_expect_success 'TAP output from unit tests' - <<-\EOT
> 	cat >expect <<-\EOF
> 	ok 1 - passing test
> 	ok 2 - passing test and assertion return 1
> 	[...]
> 	# check "'a' == '\n'" failed at t/helper/test-example-tap.c:64
> 	#    left: 'a'
> 	#   right: '\012'
> 	[...]
> 	EOF
>   EOT
>
> would that be preferable to moving it to its own file? I kind of like
> keeping everything in the test scripts themselves so related changes can
> happen side-by-side, though I admit in this case it is intimately tied
> to the separate test-example-tap.c source anyway.

I can't think of an example where we keep test definitions in the same
file as the code to be tested.  It would be somewhat cool to empower the
unit test framework to test itself, but I suspect that this nesting
ability would be hard to achieve and not very useful otherwise.  And
would we be able to trust such a self-test?


We could cheese it by putting the expected output into a special comment
before (or after) the TEST invocations and letting the test script piece
them together to build the expect file, something like:

	/* expect
	ok 1 - passing test
	 */
	test_res = TEST(check_res = check_int(1, ==, 1), "passing test");

That would be a bit annoying if we change something because some
messages contain line numbers and the comments would affect those due
to their existence alone.  And there would be a ripple effect if we
change the number of output lines of a test to the output of later
tests.


The only downside of keeping the expected output of t0080 separate that
I can think of is that it might get confusing if we'd ever add more
test_expect_success calls to it, but I can't imagine why we'd want to
do that.

> But I do have such an "EOT" patch which I've been meaning to send out,
> since it makes many of these quoting annoyances go away (though of
> course it leaves the indentation).

Being able to pass the test code to test_expect_success as a here-doc or
file to avoid nested shell quoting sounds useful in general.  For t0080
we could achieve the same effect already by creating the expect file
before calling test_expect_success.  That has the downside of passing
even when the disk is full and the files are created empty, but we can
throw in a "test -s" to rule it out.

René

Re: [PATCH 6/6] t-strbuf: use TEST_RUN

[Josh Steadmon]

On 2024.06.29 17:47, René Scharfe wrote:
> The macro TEST takes a single expression.  If a test requires multiple
> statements then they need to be placed in a function that's called in
> the TEST expression.  The functions setup() and setup_populated() here
> are used for that purpose and take another function as an argument,
> making the control flow hard to follow.
> 
> Remove the overhead of these functions by using TEST_RUN instead.  Move
> their duplicate post-condition checks into a new helper, t_release(),
> and let t_addch() and t_addstr() accept properly typed input parameters
> instead of void pointers.
> 
> Use the fully checking t_addstr() for adding initial values instead of
> only doing only a length comparison -- there's no need for skipping the
> other checks.
> 
> This results in test cases that look much more like strbuf usage in
> production code, only with checked strbuf functions replaced by checking
> wrappers.
> 
> Signed-off-by: René Scharfe <l.s.r@web.de>
> ---
>  t/unit-tests/t-strbuf.c | 79 +++++++++++++++++++++--------------------
>  1 file changed, 41 insertions(+), 38 deletions(-)
> 
> diff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c
> index 6027dafef7..c8e39ddda7 100644
> --- a/t/unit-tests/t-strbuf.c
> +++ b/t/unit-tests/t-strbuf.c
> @@ -1,32 +1,6 @@
>  #include "test-lib.h"
>  #include "strbuf.h"
> 
> -/* wrapper that supplies tests with an empty, initialized strbuf */
> -static void setup(void (*f)(struct strbuf*, const void*),
> -		  const void *data)
> -{
> -	struct strbuf buf = STRBUF_INIT;
> -
> -	f(&buf, data);
> -	strbuf_release(&buf);
> -	check_uint(buf.len, ==, 0);
> -	check_uint(buf.alloc, ==, 0);
> -}
> -
> -/* wrapper that supplies tests with a populated, initialized strbuf */
> -static void setup_populated(void (*f)(struct strbuf*, const void*),
> -			    const char *init_str, const void *data)
> -{
> -	struct strbuf buf = STRBUF_INIT;
> -
> -	strbuf_addstr(&buf, init_str);
> -	check_uint(buf.len, ==, strlen(init_str));
> -	f(&buf, data);
> -	strbuf_release(&buf);
> -	check_uint(buf.len, ==, 0);
> -	check_uint(buf.alloc, ==, 0);
> -}
> -
>  static int assert_sane_strbuf(struct strbuf *buf)
>  {
>  	/* Initialized strbufs should always have a non-NULL buffer */
> @@ -66,10 +40,8 @@ static void t_dynamic_init(void)
>  	strbuf_release(&buf);
>  }
> 
> -static void t_addch(struct strbuf *buf, const void *data)
> +static void t_addch(struct strbuf *buf, int ch)
>  {
> -	const char *p_ch = data;
> -	const char ch = *p_ch;
>  	size_t orig_alloc = buf->alloc;
>  	size_t orig_len = buf->len;
> 
> @@ -85,9 +57,8 @@ static void t_addch(struct strbuf *buf, const void *data)
>  	check_char(buf->buf[buf->len], ==, '\0');
>  }
> 
> -static void t_addstr(struct strbuf *buf, const void *data)
> +static void t_addstr(struct strbuf *buf, const char *text)
>  {
> -	const char *text = data;
>  	size_t len = strlen(text);
>  	size_t orig_alloc = buf->alloc;
>  	size_t orig_len = buf->len;
> @@ -105,18 +76,50 @@ static void t_addstr(struct strbuf *buf, const void *data)
>  	check_str(buf->buf + orig_len, text);
>  }
> 
> +static void t_release(struct strbuf *sb)
> +{
> +	strbuf_release(sb);
> +	check_uint(sb->len, ==, 0);
> +	check_uint(sb->alloc, ==, 0);
> +}
> +
>  int cmd_main(int argc, const char **argv)
>  {
>  	if (!TEST(t_static_init(), "static initialization works"))
>  		test_skip_all("STRBUF_INIT is broken");
>  	TEST(t_dynamic_init(), "dynamic initialization works");

IIUC you're leaving t_static_init() as-is so that we can determine
whether or not to skip the rest of the tests, but is there a reason you
didn't convert t_dynamic_init() here?

> -	TEST(setup(t_addch, "a"), "strbuf_addch adds char");
> -	TEST(setup(t_addch, ""), "strbuf_addch adds NUL char");
> -	TEST(setup_populated(t_addch, "initial value", "a"),
> -	     "strbuf_addch appends to initial value");
> -	TEST(setup(t_addstr, "hello there"), "strbuf_addstr adds string");
> -	TEST(setup_populated(t_addstr, "initial value", "hello there"),
> -	     "strbuf_addstr appends string to initial value");
> +
> +	if (TEST_RUN("strbuf_addch adds char")) {
> +		struct strbuf sb = STRBUF_INIT;
> +		t_addch(&sb, 'a');
> +		t_release(&sb);
> +	}
> +
> +	if (TEST_RUN("strbuf_addch adds NUL char")) {
> +		struct strbuf sb = STRBUF_INIT;
> +		t_addch(&sb, '\0');
> +		t_release(&sb);
> +	}
> +
> +	if (TEST_RUN("strbuf_addch appends to initial value")) {
> +		struct strbuf sb = STRBUF_INIT;
> +		t_addstr(&sb, "initial value");
> +		t_addch(&sb, 'a');
> +		t_release(&sb);
> +	}
> +
> +	if (TEST_RUN("strbuf_addstr adds string")) {
> +		struct strbuf sb = STRBUF_INIT;
> +		t_addstr(&sb, "hello there");
> +		t_release(&sb);
> +	}
> +
> +	if (TEST_RUN("strbuf_addstr appends string to initial value")) {
> +		struct strbuf sb = STRBUF_INIT;
> +		t_addstr(&sb, "initial value");
> +		t_addstr(&sb, "hello there");
> +		t_release(&sb);
> +	}
> 
>  	return test_done();
>  }
> --
> 2.45.2

I think this commit in particular shows how TEST_RUN() is more
convenient than TEST(). (Although, arguably we shouldn't have allowed
the setup() + callback situation to start with.)

Re: [PATCH 0/6] unit-tests: add and use TEST_RUN to simplify tests

[Josh Steadmon]

On 2024.06.29 17:33, René Scharfe wrote:
> The macro TEST only allows defining a test that consists of a single
> expression.  This requires wrapping tests made up of one or more
> statements in a function, which is a small, but avoidable hurdle.  This
> series provides a new macro, TEST_RUN, that provides a way to define
> tests without requiring to declare a function.
> 
>   t0080: move expected output to a file
>   unit-tests: add TEST_RUN
>   t-ctype: use TEST_RUN
>   t-reftable-basics: use TEST_RUN
>   t-strvec: use TEST_RUN
>   t-strbuf: use TEST_RUN
> 
>  t/helper/test-example-tap.c      |  33 +++
>  t/t0080-unit-test-output.sh      |  48 +----
>  t/t0080/expect                   |  76 +++++++
>  t/unit-tests/t-ctype.c           |   4 +-
>  t/unit-tests/t-reftable-basics.c | 228 +++++++++-----------
>  t/unit-tests/t-strbuf.c          |  79 +++----
>  t/unit-tests/t-strvec.c          | 356 ++++++++++++++-----------------
>  t/unit-tests/test-lib.c          |  42 +++-
>  t/unit-tests/test-lib.h          |   8 +
>  9 files changed, 462 insertions(+), 412 deletions(-)
>  create mode 100644 t/t0080/expect
> 
> --
> 2.45.2

One small nitpick on patch #3 and a question on #6, but basically this
series looks good to me. I'll be away from email for the rest of the
week, so I'll go ahead and sign off:

Reviewed-by: Josh Steadmon <steadmon@google.com>

Re: [PATCH 3/6] t-ctype: use TEST_RUN

[René Scharfe]

Am 01.07.24 um 21:49 schrieb Josh Steadmon:
> On 2024.06.29 17:44, René Scharfe wrote:
>> Use the macro TEST_RUN instead of the internal functions
>> test__run_begin() and test__run_end().
>>
>> Signed-off-by: René Scharfe <l.s.r@web.de>
>
> Nitpick: please expand the commit message here as you did for the
> following patches.

There is not much more to it: Use of internal functions is discouraged,
so this patch replaces their calls with the new public feature.  Perhaps
a "Kinda-requested-by: Phillip Wood <phillip.wood@dunelm.org.uk>" would
be appropriate?

Unlike in the later patches the tests themselves are unchanged, so this
has no consequence for people that want to add or modify character
classifier tests.

René

Re: [PATCH 6/6] t-strbuf: use TEST_RUN

[René Scharfe]

Am 01.07.24 um 21:58 schrieb Josh Steadmon:
> On 2024.06.29 17:47, René Scharfe wrote:
>> The macro TEST takes a single expression.  If a test requires multiple
>> statements then they need to be placed in a function that's called in
>> the TEST expression.  The functions setup() and setup_populated() here
>> are used for that purpose and take another function as an argument,
>> making the control flow hard to follow.
>>
>> Remove the overhead of these functions by using TEST_RUN instead.  Move
>> their duplicate post-condition checks into a new helper, t_release(),
>> and let t_addch() and t_addstr() accept properly typed input parameters
>> instead of void pointers.
>>
>> Use the fully checking t_addstr() for adding initial values instead of
>> only doing only a length comparison -- there's no need for skipping the
>> other checks.
>>
>> This results in test cases that look much more like strbuf usage in
>> production code, only with checked strbuf functions replaced by checking
>> wrappers.
>>
>> Signed-off-by: René Scharfe <l.s.r@web.de>
>> ---
>>  t/unit-tests/t-strbuf.c | 79 +++++++++++++++++++++--------------------
>>  1 file changed, 41 insertions(+), 38 deletions(-)
>>
>> diff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c
>> index 6027dafef7..c8e39ddda7 100644
>> --- a/t/unit-tests/t-strbuf.c
>> +++ b/t/unit-tests/t-strbuf.c
>> @@ -1,32 +1,6 @@
>>  #include "test-lib.h"
>>  #include "strbuf.h"
>>
>> -/* wrapper that supplies tests with an empty, initialized strbuf */
>> -static void setup(void (*f)(struct strbuf*, const void*),
>> -		  const void *data)
>> -{
>> -	struct strbuf buf = STRBUF_INIT;
>> -
>> -	f(&buf, data);
>> -	strbuf_release(&buf);
>> -	check_uint(buf.len, ==, 0);
>> -	check_uint(buf.alloc, ==, 0);
>> -}
>> -
>> -/* wrapper that supplies tests with a populated, initialized strbuf */
>> -static void setup_populated(void (*f)(struct strbuf*, const void*),
>> -			    const char *init_str, const void *data)
>> -{
>> -	struct strbuf buf = STRBUF_INIT;
>> -
>> -	strbuf_addstr(&buf, init_str);
>> -	check_uint(buf.len, ==, strlen(init_str));
>> -	f(&buf, data);
>> -	strbuf_release(&buf);
>> -	check_uint(buf.len, ==, 0);
>> -	check_uint(buf.alloc, ==, 0);
>> -}
>> -
>>  static int assert_sane_strbuf(struct strbuf *buf)
>>  {
>>  	/* Initialized strbufs should always have a non-NULL buffer */
>> @@ -66,10 +40,8 @@ static void t_dynamic_init(void)
>>  	strbuf_release(&buf);
>>  }
>>
>> -static void t_addch(struct strbuf *buf, const void *data)
>> +static void t_addch(struct strbuf *buf, int ch)
>>  {
>> -	const char *p_ch = data;
>> -	const char ch = *p_ch;
>>  	size_t orig_alloc = buf->alloc;
>>  	size_t orig_len = buf->len;
>>
>> @@ -85,9 +57,8 @@ static void t_addch(struct strbuf *buf, const void *data)
>>  	check_char(buf->buf[buf->len], ==, '\0');
>>  }
>>
>> -static void t_addstr(struct strbuf *buf, const void *data)
>> +static void t_addstr(struct strbuf *buf, const char *text)
>>  {
>> -	const char *text = data;
>>  	size_t len = strlen(text);
>>  	size_t orig_alloc = buf->alloc;
>>  	size_t orig_len = buf->len;
>> @@ -105,18 +76,50 @@ static void t_addstr(struct strbuf *buf, const void *data)
>>  	check_str(buf->buf + orig_len, text);
>>  }
>>
>> +static void t_release(struct strbuf *sb)
>> +{
>> +	strbuf_release(sb);
>> +	check_uint(sb->len, ==, 0);
>> +	check_uint(sb->alloc, ==, 0);
>> +}
>> +
>>  int cmd_main(int argc, const char **argv)
>>  {
>>  	if (!TEST(t_static_init(), "static initialization works"))
>>  		test_skip_all("STRBUF_INIT is broken");
>>  	TEST(t_dynamic_init(), "dynamic initialization works");
>
> IIUC you're leaving t_static_init() as-is so that we can determine
> whether or not to skip the rest of the tests,

Right.  And that place might be a use case for an official version of
test__run_end(), but it's probably better to gather a few more such
candidates before drawing conclusions.

> but is there a reason you
> didn't convert t_dynamic_init() here?

Good question, I don't remember.  Perhaps an oversight or laziness.
Or just a general feeling to leave the init tests alone since the
first one doesn't map to TEST_RUN easily.  So, in a word: No.

>> -	TEST(setup(t_addch, "a"), "strbuf_addch adds char");
>> -	TEST(setup(t_addch, ""), "strbuf_addch adds NUL char");
>> -	TEST(setup_populated(t_addch, "initial value", "a"),
>> -	     "strbuf_addch appends to initial value");
>> -	TEST(setup(t_addstr, "hello there"), "strbuf_addstr adds string");
>> -	TEST(setup_populated(t_addstr, "initial value", "hello there"),
>> -	     "strbuf_addstr appends string to initial value");
>> +
>> +	if (TEST_RUN("strbuf_addch adds char")) {
>> +		struct strbuf sb = STRBUF_INIT;
>> +		t_addch(&sb, 'a');
>> +		t_release(&sb);
>> +	}
>> +
>> +	if (TEST_RUN("strbuf_addch adds NUL char")) {
>> +		struct strbuf sb = STRBUF_INIT;
>> +		t_addch(&sb, '\0');
>> +		t_release(&sb);
>> +	}
>> +
>> +	if (TEST_RUN("strbuf_addch appends to initial value")) {
>> +		struct strbuf sb = STRBUF_INIT;
>> +		t_addstr(&sb, "initial value");
>> +		t_addch(&sb, 'a');
>> +		t_release(&sb);
>> +	}
>> +
>> +	if (TEST_RUN("strbuf_addstr adds string")) {
>> +		struct strbuf sb = STRBUF_INIT;
>> +		t_addstr(&sb, "hello there");
>> +		t_release(&sb);
>> +	}
>> +
>> +	if (TEST_RUN("strbuf_addstr appends string to initial value")) {
>> +		struct strbuf sb = STRBUF_INIT;
>> +		t_addstr(&sb, "initial value");
>> +		t_addstr(&sb, "hello there");
>> +		t_release(&sb);
>> +	}
>>
>>  	return test_done();
>>  }
>> --
>> 2.45.2
>
> I think this commit in particular shows how TEST_RUN() is more
> convenient than TEST(). (Although, arguably we shouldn't have allowed
> the setup() + callback situation to start with.)

Great, thanks!

René

Re: [PATCH 1/6] t0080: move expected output to a file

[Jeff King]

On Mon, Jul 01, 2024 at 12:17:11PM -0700, Junio C Hamano wrote:

> > But I do have such an "EOT" patch which I've been meaning to send out,
> > since it makes many of these quoting annoyances go away (though of
> > course it leaves the indentation).
> 
> I am not sure about your "test body comes from the standard input"
> (not saying "I am not convinced it is a good idea" or even "I am
> convinced it is a bad idea"---I do not know what to think about it,
> not just yet).  THe above illustration does make it easier to grok
> by keeping everything in one place.

I just (re-)posted it in:

  https://lore.kernel.org/git/20240701220815.GA20293@coredump.intra.peff.net/

so you can see the improvement in some other real cases.

-Peff

Re: [PATCH 1/6] t0080: move expected output to a file

[Jeff King]

On Mon, Jul 01, 2024 at 09:51:31PM +0200, René Scharfe wrote:

> > would that be preferable to moving it to its own file? I kind of like
> > keeping everything in the test scripts themselves so related changes can
> > happen side-by-side, though I admit in this case it is intimately tied
> > to the separate test-example-tap.c source anyway.
> 
> I can't think of an example where we keep test definitions in the same
> file as the code to be tested.  It would be somewhat cool to empower the
> unit test framework to test itself, but I suspect that this nesting
> ability would be hard to achieve and not very useful otherwise.  And
> would we be able to trust such a self-test?

Yeah, I think this is really a special case, just because we're not
testing specific items, but rather sanity-checking the TAP output
itself.

I think even in the unit tests, we're mostly carrying expected output
next to the code (and the unit test harness is checking those and
producing appropriate messages). It's only this "run a bunch of tests
that might themselves fail and see if the output was sensible" that it's
hard to do this for. I.e., t0080 is special and weird.

> The only downside of keeping the expected output of t0080 separate that
> I can think of is that it might get confusing if we'd ever add more
> test_expect_success calls to it, but I can't imagine why we'd want to
> do that.

Yeah, I think it is mostly a lost cause here. We are far away from the
source code whether it is a here-doc or a separate file.

I guess I was responding more to the principle that external files are
usually just distracting and annoying (another annoyance: they inhibit
tab completion ;) ). But it is not that big a deal either way in this
case.

> Being able to pass the test code to test_expect_success as a here-doc or
> file to avoid nested shell quoting sounds useful in general.  For t0080
> we could achieve the same effect already by creating the expect file
> before calling test_expect_success.  That has the downside of passing
> even when the disk is full and the files are created empty, but we can
> throw in a "test -s" to rule it out.

Yup, agreed with everything there.

-Peff

Re: [PATCH 1/6] t0080: move expected output to a file

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Mon, Jul 01, 2024 at 12:17:11PM -0700, Junio C Hamano wrote:
>
>> > But I do have such an "EOT" patch which I've been meaning to send out,
>> > since it makes many of these quoting annoyances go away (though of
>> > course it leaves the indentation).
>> 
>> I am not sure about your "test body comes from the standard input"
>> (not saying "I am not convinced it is a good idea" or even "I am
>> convinced it is a bad idea"---I do not know what to think about it,
>> not just yet).  THe above illustration does make it easier to grok
>> by keeping everything in one place.
>
> I just (re-)posted it in:
>
>   https://lore.kernel.org/git/20240701220815.GA20293@coredump.intra.peff.net/
>
> so you can see the improvement in some other real cases.

;-)

The shells we care about (and that does not include the /bin/sh on
ancient Solaris ☹) should be OK, but "IFS= read -r line" somehow
makes me feel nervous.  Maybe I am superstitious.

Both steps look quite good.  Will queue.

Thanks.

Re: [PATCH 1/6] t0080: move expected output to a file

[Jeff King]

On Mon, Jul 01, 2024 at 04:38:45PM -0700, Junio C Hamano wrote:

> > I just (re-)posted it in:
> >
> >   https://lore.kernel.org/git/20240701220815.GA20293@coredump.intra.peff.net/
> >
> > so you can see the improvement in some other real cases.
> 
> ;-)
> 
> The shells we care about (and that does not include the /bin/sh on
> ancient Solaris ☹) should be OK, but "IFS= read -r line" somehow
> makes me feel nervous.  Maybe I am superstitious.

I don't know offhand of any case where it will fail. I'd prefer to start
with it and see if it bites us, given that it saves us a process
invocation per test (and those do add up in some cases).

I also wondered if we might be able to save the syscall-per-byte
overhead of "read" for some shells (since we know we are reading until
EOF anyway). Using "read -N" with bash would let us do that, but
obviously we'd still need to fall back to regular "read" for other
shells. I don't think it's worth the complexity unless we can really
show a measurable speedup (and I didn't seem to see one).

-Peff

Re: [PATCH 2/6] unit-tests: add TEST_RUN

[phillip.wood123]

Hi René

On 29/06/2024 16:43, René Scharfe wrote:
> The macro TEST only allows defining a test that consists of a single
> expression.  Add the new macro, TEST_RUN, which provides a way to define
> unit tests that are made up of one or more statements.  A test started
> with it implicitly ends when the next test is started or test_done() is
> called.
> 
> TEST_RUN allows defining self-contained tests en bloc, a bit like
> test_expect_success does for regular tests.  Unlike TEST it does not
> require defining wrapper functions for test statements.

There are pros and cons to not requiring one function per test. It can 
be a pain to have to write separate functions but it keeps each test 
self contained which hopefully makes it harder to have accidental 
dependencies between tests. Having separate functions for each test 
makes it easy to initialize and free resources for every test by writing 
a setup() function that initializes the resources, calls the test 
function and then frees the resources. The changes in patch 6 to use 
TEST_RUN() mean that each test now has more boilerplate to initialize 
and free the strbuf. Having each test in its own function also makes 
main() shorter and which means can quickly get an overview of all the 
test cases from it.

On the other hand having all the tests defined in main() using 
TEST_RUN() means we can just write the test body without having to 
define a separate function and then call it with TEST()

> No public method is provided for ending a test explicitly, yet; let's
> see if we'll ever need one.

This means that we do not error out if there are accidentally nested 
tests. That probably does not matter too much.


> +int test__run(const char *location, const char *format, ...)
> +{
> +	va_list ap;
> +	char *desc;
> +
> +	test__run_maybe_end();
> +
> +	va_start(ap, format);
> +	desc = xstrvfmt(format, ap);

This uses an strbuf under the hood. So far we've avoided doing that as 
we want to be able to test the strbuf implementation with this test 
framework. We don't need to support arbitrary length strings here so we 
could use a fixed array and xsnprinf() instead.

> +/*
> + * Start a test, returns 1 if the test was actually started or 0 if it
> + * was skipped.  The test ends when the next test starts or test_done()
> + * is called.
> + */
> +#define TEST_RUN(...) test__run(TEST_LOCATION(), __VA_ARGS__)

Looking ahead the plan seems to be to change most instances of TEST() to 
TEST_RUN(). If we are going to go that way perhaps we should steal 
TEST() for this macro and rename the existing TEST() macro.

I'm not very enthusiastic about requiring the test author to wrap 
TEST_RUN() in an if() statement rather than just doing that for them. It 
makes it explicit but from the test author's point of view it just feels 
like pointless boilerplate.

>   /*
>    * test_done() must be called at the end of main(). It will print the
>    * plan if plan() was not called at the beginning of the test program
> @@ -156,6 +163,7 @@ extern union test__tmp test__tmp[2];
>   int test__run_begin(void);
>   __attribute__((format (printf, 3, 4)))
>   int test__run_end(int, const char *, const char *, ...);

We should add

     __attribute__((format (printf, 2, 3), warn_unused_result))

here to catch any errors in the format string / arguments and to warn if 
TEST_RUN() isn't wrapped in an if() statement.

Best Wishes

Phillip

> +int test__run(const char *location, const char *format, ...);
>   void test__todo_begin(void);
>   int test__todo_end(const char *, const char *, int);
> 
> --
> 2.45.2

Re: [PATCH 3/6] t-ctype: use TEST_RUN

[phillip.wood123]

Hi René

On 01/07/2024 21:04, René Scharfe wrote:
> Am 01.07.24 um 21:49 schrieb Josh Steadmon:
>> On 2024.06.29 17:44, René Scharfe wrote:
>>> Use the macro TEST_RUN instead of the internal functions
>>> test__run_begin() and test__run_end().
>>>
>>> Signed-off-by: René Scharfe <l.s.r@web.de>
>>
>> Nitpick: please expand the commit message here as you did for the
>> following patches.
> 
> There is not much more to it: Use of internal functions is discouraged,
> so this patch replaces their calls with the new public feature.  Perhaps
> a "Kinda-requested-by: Phillip Wood <phillip.wood@dunelm.org.uk>" would
> be appropriate?

Perhaps something like

These tests use the internal functions test__run_begin() and 
test__run_end() which are supposed to be private to the test framework. 
Convert them to use the newly added TEST_RUN() instead.

would be sufficient?

Best Wishes

Phillip

> Unlike in the later patches the tests themselves are unchanged, so this
> has no consequence for people that want to add or modify character
> classifier tests.
> 
> René

Re: [PATCH 5/6] t-strvec: use TEST_RUN

[phillip.wood123]

Hi René

On 29/06/2024 16:46, René Scharfe wrote:
> The macro TEST takes a single expression.  If a test requires multiple
> statements then they need to be placed in a function that's called in
> the TEST expression.
> 
> Remove the overhead of defining and calling single-use functions by
> using TEST_RUN instead.

I'm not sure what the overhead is that you keep referring to - the 
compiler will inline functions that are only called once so I presume 
you mean overhead for test authors or contributors reading the code?

This is not related to you changes but while looking at this patch I 
noticed that the existing code in check_strvec_loc() contains

		if (!check_uint(vec->nr, >, nr) ||
		    !check_uint(vec->alloc, >, nr) ||
		    !check_str(vec->v[nr], str)) {
			struct strbuf msg = STRBUF_INIT;
			strbuf_addf(&msg, "strvec index %"PRIuMAX, (uintmax_t) nr);
			test_assert(loc, msg.buf, 0);
			strbuf_release(&msg);
			va_end(ap);
			return;

Which looks like it should be using test_msg() instead of writing a 
message to an strbuf and calling test_assert().

The conversion itself looks correct. It is a shame that each test has to 
have boilerplate to initalize and free the strvec but that comes from 
the existing implementation.

Best Wishes

Phillip

> Signed-off-by: René Scharfe <l.s.r@web.de>
> ---
>   t/unit-tests/t-strvec.c | 356 ++++++++++++++++++----------------------
>   1 file changed, 156 insertions(+), 200 deletions(-)
> 
> diff --git a/t/unit-tests/t-strvec.c b/t/unit-tests/t-strvec.c
> index d4615ab06d..00ff7d4ae8 100644
> --- a/t/unit-tests/t-strvec.c
> +++ b/t/unit-tests/t-strvec.c
> @@ -36,237 +36,193 @@ static void check_strvec_loc(const char *loc, struct strvec *vec, ...)
>   	check_pointer_eq(vec->v[nr], NULL);
>   }
> 
> -static void t_static_init(void)
> +int cmd_main(int argc, const char **argv)
>   {
> -	struct strvec vec = STRVEC_INIT;
> -	check_pointer_eq(vec.v, empty_strvec);
> -	check_uint(vec.nr, ==, 0);
> -	check_uint(vec.alloc, ==, 0);
> -}
> +	if (TEST_RUN("static initialization")) {
> +		struct strvec vec = STRVEC_INIT;
> +		check_pointer_eq(vec.v, empty_strvec);
> +		check_uint(vec.nr, ==, 0);
> +		check_uint(vec.alloc, ==, 0);
> +	}
> 
> -static void t_dynamic_init(void)
> -{
> -	struct strvec vec;
> -	strvec_init(&vec);
> -	check_pointer_eq(vec.v, empty_strvec);
> -	check_uint(vec.nr, ==, 0);
> -	check_uint(vec.alloc, ==, 0);
> -}
> +	if (TEST_RUN("dynamic initialization")) {
> +		struct strvec vec;
> +		strvec_init(&vec);
> +		check_pointer_eq(vec.v, empty_strvec);
> +		check_uint(vec.nr, ==, 0);
> +		check_uint(vec.alloc, ==, 0);
> +	}
> 
> -static void t_clear(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_push(&vec, "foo");
> -	strvec_clear(&vec);
> -	check_pointer_eq(vec.v, empty_strvec);
> -	check_uint(vec.nr, ==, 0);
> -	check_uint(vec.alloc, ==, 0);
> -}
> +	if (TEST_RUN("clear")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_push(&vec, "foo");
> +		strvec_clear(&vec);
> +		check_pointer_eq(vec.v, empty_strvec);
> +		check_uint(vec.nr, ==, 0);
> +		check_uint(vec.alloc, ==, 0);
> +	}
> 
> -static void t_push(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> +	if (TEST_RUN("push")) {
> +		struct strvec vec = STRVEC_INIT;
> 
> -	strvec_push(&vec, "foo");
> -	check_strvec(&vec, "foo", NULL);
> +		strvec_push(&vec, "foo");
> +		check_strvec(&vec, "foo", NULL);
> 
> -	strvec_push(&vec, "bar");
> -	check_strvec(&vec, "foo", "bar", NULL);
> +		strvec_push(&vec, "bar");
> +		check_strvec(&vec, "foo", "bar", NULL);
> 
> -	strvec_clear(&vec);
> -}
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_pushf(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_pushf(&vec, "foo: %d", 1);
> -	check_strvec(&vec, "foo: 1", NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("pushf")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_pushf(&vec, "foo: %d", 1);
> +		check_strvec(&vec, "foo: 1", NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_pushl(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> -	check_strvec(&vec, "foo", "bar", "baz", NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("pushl")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> +		check_strvec(&vec, "foo", "bar", "baz", NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_pushv(void)
> -{
> -	const char *strings[] = {
> -		"foo", "bar", "baz", NULL,
> -	};
> -	struct strvec vec = STRVEC_INIT;
> +	if (TEST_RUN("pushv")) {
> +		const char *strings[] = {
> +			"foo", "bar", "baz", NULL,
> +		};
> +		struct strvec vec = STRVEC_INIT;
> 
> -	strvec_pushv(&vec, strings);
> -	check_strvec(&vec, "foo", "bar", "baz", NULL);
> +		strvec_pushv(&vec, strings);
> +		check_strvec(&vec, "foo", "bar", "baz", NULL);
> 
> -	strvec_clear(&vec);
> -}
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_replace_at_head(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> -	strvec_replace(&vec, 0, "replaced");
> -	check_strvec(&vec, "replaced", "bar", "baz", NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("replace at head")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> +		strvec_replace(&vec, 0, "replaced");
> +		check_strvec(&vec, "replaced", "bar", "baz", NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_replace_at_tail(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> -	strvec_replace(&vec, 2, "replaced");
> -	check_strvec(&vec, "foo", "bar", "replaced", NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("replace at tail")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> +		strvec_replace(&vec, 2, "replaced");
> +		check_strvec(&vec, "foo", "bar", "replaced", NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_replace_in_between(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> -	strvec_replace(&vec, 1, "replaced");
> -	check_strvec(&vec, "foo", "replaced", "baz", NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("replace in between")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> +		strvec_replace(&vec, 1, "replaced");
> +		check_strvec(&vec, "foo", "replaced", "baz", NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_replace_with_substring(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_pushl(&vec, "foo", NULL);
> -	strvec_replace(&vec, 0, vec.v[0] + 1);
> -	check_strvec(&vec, "oo", NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("replace with substring")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_pushl(&vec, "foo", NULL);
> +		strvec_replace(&vec, 0, vec.v[0] + 1);
> +		check_strvec(&vec, "oo", NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_remove_at_head(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> -	strvec_remove(&vec, 0);
> -	check_strvec(&vec, "bar", "baz", NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("remove at head")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> +		strvec_remove(&vec, 0);
> +		check_strvec(&vec, "bar", "baz", NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_remove_at_tail(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> -	strvec_remove(&vec, 2);
> -	check_strvec(&vec, "foo", "bar", NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("remove at tail")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> +		strvec_remove(&vec, 2);
> +		check_strvec(&vec, "foo", "bar", NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_remove_in_between(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> -	strvec_remove(&vec, 1);
> -	check_strvec(&vec, "foo", "baz", NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("remove in between")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> +		strvec_remove(&vec, 1);
> +		check_strvec(&vec, "foo", "baz", NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_pop_empty_array(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_pop(&vec);
> -	check_strvec(&vec, NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("pop with empty array")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_pop(&vec);
> +		check_strvec(&vec, NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_pop_non_empty_array(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> -	strvec_pop(&vec);
> -	check_strvec(&vec, "foo", "bar", NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("pop with non-empty array")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
> +		strvec_pop(&vec);
> +		check_strvec(&vec, "foo", "bar", NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_split_empty_string(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_split(&vec, "");
> -	check_strvec(&vec, NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("split empty string")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_split(&vec, "");
> +		check_strvec(&vec, NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_split_single_item(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_split(&vec, "foo");
> -	check_strvec(&vec, "foo", NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("split single item")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_split(&vec, "foo");
> +		check_strvec(&vec, "foo", NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_split_multiple_items(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_split(&vec, "foo bar baz");
> -	check_strvec(&vec, "foo", "bar", "baz", NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("split multiple items")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_split(&vec, "foo bar baz");
> +		check_strvec(&vec, "foo", "bar", "baz", NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_split_whitespace_only(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_split(&vec, " \t\n");
> -	check_strvec(&vec, NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("split whitespace only")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_split(&vec, " \t\n");
> +		check_strvec(&vec, NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_split_multiple_consecutive_whitespaces(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	strvec_split(&vec, "foo\n\t bar");
> -	check_strvec(&vec, "foo", "bar", NULL);
> -	strvec_clear(&vec);
> -}
> +	if (TEST_RUN("split multiple consecutive whitespaces")) {
> +		struct strvec vec = STRVEC_INIT;
> +		strvec_split(&vec, "foo\n\t bar");
> +		check_strvec(&vec, "foo", "bar", NULL);
> +		strvec_clear(&vec);
> +	}
> 
> -static void t_detach(void)
> -{
> -	struct strvec vec = STRVEC_INIT;
> -	const char **detached;
> +	if (TEST_RUN("detach")) {
> +		struct strvec vec = STRVEC_INIT;
> +		const char **detached;
> 
> -	strvec_push(&vec, "foo");
> +		strvec_push(&vec, "foo");
> 
> -	detached = strvec_detach(&vec);
> -	check_str(detached[0], "foo");
> -	check_pointer_eq(detached[1], NULL);
> +		detached = strvec_detach(&vec);
> +		check_str(detached[0], "foo");
> +		check_pointer_eq(detached[1], NULL);
> 
> -	check_pointer_eq(vec.v, empty_strvec);
> -	check_uint(vec.nr, ==, 0);
> -	check_uint(vec.alloc, ==, 0);
> +		check_pointer_eq(vec.v, empty_strvec);
> +		check_uint(vec.nr, ==, 0);
> +		check_uint(vec.alloc, ==, 0);
> 
> -	free((char *) detached[0]);
> -	free(detached);
> -}
> +		free((char *) detached[0]);
> +		free(detached);
> +	}
> 
> -int cmd_main(int argc, const char **argv)
> -{
> -	TEST(t_static_init(), "static initialization");
> -	TEST(t_dynamic_init(), "dynamic initialization");
> -	TEST(t_clear(), "clear");
> -	TEST(t_push(), "push");
> -	TEST(t_pushf(), "pushf");
> -	TEST(t_pushl(), "pushl");
> -	TEST(t_pushv(), "pushv");
> -	TEST(t_replace_at_head(), "replace at head");
> -	TEST(t_replace_in_between(), "replace in between");
> -	TEST(t_replace_at_tail(), "replace at tail");
> -	TEST(t_replace_with_substring(), "replace with substring");
> -	TEST(t_remove_at_head(), "remove at head");
> -	TEST(t_remove_in_between(), "remove in between");
> -	TEST(t_remove_at_tail(), "remove at tail");
> -	TEST(t_pop_empty_array(), "pop with empty array");
> -	TEST(t_pop_non_empty_array(), "pop with non-empty array");
> -	TEST(t_split_empty_string(), "split empty string");
> -	TEST(t_split_single_item(), "split single item");
> -	TEST(t_split_multiple_items(), "split multiple items");
> -	TEST(t_split_whitespace_only(), "split whitespace only");
> -	TEST(t_split_multiple_consecutive_whitespaces(), "split multiple consecutive whitespaces");
> -	TEST(t_detach(), "detach");
>   	return test_done();
>   }
> --
> 2.45.2

Re: [PATCH 6/6] t-strbuf: use TEST_RUN

[phillip.wood123]

Hi Josh and René

On 01/07/2024 20:58, Josh Steadmon wrote:
> On 2024.06.29 17:47, René Scharfe wrote:
>
> I think this commit in particular shows how TEST_RUN() is more
> convenient than TEST(). (Although, arguably we shouldn't have allowed
> the setup() + callback situation to start with.)

I think the counterargument to that is that using TEST_RUN() makes the 
tests noisier and more error prone because each one has to be wrapped in 
an if() statement and has more boiler plate initializing and freeing the 
strbuf rather than getting that for free by calling the test function 
via setup().

Having said that I don't mind the changes in this patch if that's the 
way others want to go. Getting rid of the untyped test arguments is 
definitely a benefit of this approach.

Best Wishes

Phillip

Re: [PATCH 2/6] unit-tests: add TEST_RUN

[Junio C Hamano]

phillip.wood123@gmail.com writes:

>> No public method is provided for ending a test explicitly, yet; let's
>> see if we'll ever need one.
>
> This means that we do not error out if there are accidentally nested
> tests. That probably does not matter too much.

Isn't it more like it is impossible to create nested tests, since a
"begin" implicitly ends the current one before starting the new one?

>> @@ -156,6 +163,7 @@ extern union test__tmp test__tmp[2];
>>   int test__run_begin(void);
>>   __attribute__((format (printf, 3, 4)))
>>   int test__run_end(int, const char *, const char *, ...);
>
> We should add
>
>     __attribute__((format (printf, 2, 3), warn_unused_result))
>
> here to catch any errors in the format string / arguments and to warn
> if TEST_RUN() isn't wrapped in an if() statement.

Nice.  Especially the "unused" check is valuable.

Thanks.

Re: [PATCH 6/6] t-strbuf: use TEST_RUN

[Ghanshyam Thakkar]

Josh Steadmon <steadmon@google.com> wrote:
> > -	TEST(setup(t_addch, "a"), "strbuf_addch adds char");
> > -	TEST(setup(t_addch, ""), "strbuf_addch adds NUL char");
> > -	TEST(setup_populated(t_addch, "initial value", "a"),
> > -	     "strbuf_addch appends to initial value");
> > -	TEST(setup(t_addstr, "hello there"), "strbuf_addstr adds string");
> > -	TEST(setup_populated(t_addstr, "initial value", "hello there"),
> > -	     "strbuf_addstr appends string to initial value");
> > +
> > +	if (TEST_RUN("strbuf_addch adds char")) {
> > +		struct strbuf sb = STRBUF_INIT;
> > +		t_addch(&sb, 'a');
> > +		t_release(&sb);
> > +	}
> > +
> > +	if (TEST_RUN("strbuf_addch adds NUL char")) {
> > +		struct strbuf sb = STRBUF_INIT;
> > +		t_addch(&sb, '\0');
> > +		t_release(&sb);
> > +	}
> > +
> > +	if (TEST_RUN("strbuf_addch appends to initial value")) {
> > +		struct strbuf sb = STRBUF_INIT;
> > +		t_addstr(&sb, "initial value");
> > +		t_addch(&sb, 'a');
> > +		t_release(&sb);
> > +	}
> > +
> > +	if (TEST_RUN("strbuf_addstr adds string")) {
> > +		struct strbuf sb = STRBUF_INIT;
> > +		t_addstr(&sb, "hello there");
> > +		t_release(&sb);
> > +	}
> > +
> > +	if (TEST_RUN("strbuf_addstr appends string to initial value")) {
> > +		struct strbuf sb = STRBUF_INIT;
> > +		t_addstr(&sb, "initial value");
> > +		t_addstr(&sb, "hello there");
> > +		t_release(&sb);
> > +	}
> > 
> >  	return test_done();
> >  }
> > --
> > 2.45.2
>
> I think this commit in particular shows how TEST_RUN() is more
> convenient than TEST(). (Although, arguably we shouldn't have allowed
> the setup() + callback situation to start with.)

Could you expand a bit on why the setup() + callback thing shouldn't be
allowed? I think it is a nice way of avoiding boilerplate and having
independent state. And, I see the true potential of TEST_RUN() in
testcases defined through macros rather than replacing functions. I
actually think that the previous version with the functions was not
particularly bad, and I agree with Phillip that the previous version's
main() provided nice overview of the tests and it was easier to
verify the independence between each testcase.

Perhaps, the code snippets inside the functions are small enough to
perceive TEST_RUN() as more convenient than TEST() in this test, but,
for future reference, I definitely don't think TEST_RUN() should be
looked at as a replacement for TEST(), and more like 'when we have to
use macro magic which requires us to use internal test__run_*
functions, using TEST_RUN() is more convenient'. Patch [3/6] is a good
example of that. But, I also don't mind if patches 4, 5, or 6 get
merged as I don't see any difference between using TEST_RUN() or
TEST() in those patches, besides moving everything inside main().

Thanks.

Re: [PATCH 2/6] unit-tests: add TEST_RUN

[René Scharfe]

Am 02.07.24 um 17:51 schrieb Junio C Hamano:
> phillip.wood123@gmail.com writes:
>
>>> @@ -156,6 +163,7 @@ extern union test__tmp test__tmp[2];
>>>   int test__run_begin(void);
>>>   __attribute__((format (printf, 3, 4)))
>>>   int test__run_end(int, const char *, const char *, ...);
>>
>> We should add
>>
>>     __attribute__((format (printf, 2, 3), warn_unused_result))
>>
>> here to catch any errors in the format string / arguments and to warn
>> if TEST_RUN() isn't wrapped in an if() statement.
>
> Nice.  Especially the "unused" check is valuable.

This confused me for a moment.  I assume you both mean the new function
test__run, whose addition was cut from the citation:

   +int test__run(const char *location, const char *format, ...);

For that one the numbers make sense and the idea is good. :)

René

Re: [PATCH 2/6] unit-tests: add TEST_RUN

[René Scharfe]

Am 02.07.24 um 17:13 schrieb phillip.wood123@gmail.com:
> Hi René
>
> On 29/06/2024 16:43, René Scharfe wrote:
>> The macro TEST only allows defining a test that consists of a
>> single expression.  Add the new macro, TEST_RUN, which provides a
>> way to define unit tests that are made up of one or more
>> statements.  A test started with it implicitly ends when the next
>> test is started or test_done() is called.
>>
>> TEST_RUN allows defining self-contained tests en bloc, a bit like
>> test_expect_success does for regular tests.  Unlike TEST it does
>> not require defining wrapper functions for test statements.
>
> There are pros and cons to not requiring one function per test. It
> can be a pain to have to write separate functions but it keeps each
> test self contained which hopefully makes it harder to have
> accidental dependencies between tests. Having separate functions for
> each test makes it easy to initialize and free resources for every
> test by writing a setup() function that initializes the resources,
> calls the test function and then frees the resources.

Right.  We should use TEST and TEST_RUN when appropriate.

> The changes in patch 6 to use TEST_RUN() mean that each test now has
> more boilerplate to initialize and free the strbuf.
This makes them more similar to strbuf usage in the wild.  Using
the API idiomatically just makes more sense to me.  Not hiding
initialization and release makes the tests visibly independent.
This is not enforced by TEST_RUN, but made possible by it.

> Having each test in its own function also makes main() shorter and
> which means can quickly get an overview of all the test cases from
> it.

That's true, now you need to grep for TEST_RUN to get such an
overview.

On the other hand I find the start of the description in TEST
invocations somewhat hard to locate, as they are not vertically
aligned due to the preceding variable-length function name.  Just
saying..

>> +int test__run(const char *location, const char *format, ...)
>> +{
>> +    va_list ap;
>> +    char *desc;
>> +
>> +    test__run_maybe_end();
>> +
>> +    va_start(ap, format);
>> +    desc = xstrvfmt(format, ap);
>
> This uses an strbuf under the hood. So far we've avoided doing that
> as we want to be able to test the strbuf implementation with this
> test framework. We don't need to support arbitrary length strings
> here so we could use a fixed array and xsnprinf() instead.

Fair point.  xsnprinf() might be a bit too strict, as it doesn't
handle short buffers gracefully.  Perhaps that's OK; a developer
getting hit by that could simply increase the buffer size.

We could also let xstrvfmt() call vsnprintf(3) directly.  The code
duplication would be a bit grating, but perhaps there's some good
base function hidden in there somewhere.

> Looking ahead the plan seems to be to change most instances of TEST()
> to TEST_RUN(). If we are going to go that way perhaps we should steal
> TEST() for this macro and rename the existing TEST() macro.

Not my plan, at least -- I'm content with just having the *ability*
to keep all parts of a test together.

If we wanted to do that, though, then TEST_RUN would have to be
complemented with a blessed way to check ctx.result in order to handle
the t_static_init test of t-strbuf, which I mentioned in my earlier
reply to Josh.

Err, no, we can simply check the check_* results, like check_strvec_loc
does:

diff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c
index c8e39ddda7..c765fab53a 100644
--- a/t/unit-tests/t-strbuf.c
+++ b/t/unit-tests/t-strbuf.c
@@ -19,15 +19,6 @@ static int assert_sane_strbuf(struct strbuf *buf)
 	return check_uint(buf->len, <, buf->alloc);
 }

-static void t_static_init(void)
-{
-	struct strbuf buf = STRBUF_INIT;
-
-	check_uint(buf.len, ==, 0);
-	check_uint(buf.alloc, ==, 0);
-	check_char(buf.buf[0], ==, '\0');
-}
-
 static void t_dynamic_init(void)
 {
 	struct strbuf buf;
@@ -85,8 +76,14 @@ static void t_release(struct strbuf *sb)

 int cmd_main(int argc, const char **argv)
 {
-	if (!TEST(t_static_init(), "static initialization works"))
-		test_skip_all("STRBUF_INIT is broken");
+	if (TEST_RUN("static initialization works")) {
+		struct strbuf buf = STRBUF_INIT;
+		if (!check_uint(buf.len, ==, 0) ||
+		    !check_uint(buf.alloc, ==, 0) ||
+		    !check_char(buf.buf[0], ==, '\0'))
+			test_skip_all("STRBUF_INIT is broken");
+	}
+
 	TEST(t_dynamic_init(), "dynamic initialization works");

 	if (TEST_RUN("strbuf_addch adds char")) {

> I'm not very enthusiastic about requiring the test author to wrap
> TEST_RUN() in an if() statement rather than just doing that for them.
> It makes it explicit but from the test author's point of view it just
> feels like pointless boilerplate.

Hmm.  We can add more magic, but I suspect that it might confuse
developers and editors.

René

Re: [PATCH 5/6] t-strvec: use TEST_RUN

[René Scharfe]

Am 02.07.24 um 17:14 schrieb phillip.wood123@gmail.com:
> Hi René
>
> On 29/06/2024 16:46, René Scharfe wrote:
>> The macro TEST takes a single expression.  If a test requires
>> multiple statements then they need to be placed in a function
>> that's called in the TEST expression.
>>
>> Remove the overhead of defining and calling single-use functions
>> by using TEST_RUN instead.
>
> I'm not sure what the overhead is that you keep referring to - the
> compiler will inline functions that are only called once so I presume
> you mean overhead for test authors or contributors reading the code?

Yes, I mean the additional effort for developers to come up with
function names and write single-call functions, and for readers to
mentally connect test description and function.  Of course you still can
do that with TEST_RUN, but unlike TEST it doesn't force you.

I don't care much about changed duration of compilation or execution
here -- it shouldn't be much either way.

> This is not related to you changes but while looking at this patch I
> noticed that the existing code in check_strvec_loc() contains
>
>
>         if (!check_uint(vec->nr, >, nr) ||
>             !check_uint(vec->alloc, >, nr) ||
>             !check_str(vec->v[nr], str)) {
>             struct strbuf msg = STRBUF_INIT;
>             strbuf_addf(&msg, "strvec index %"PRIuMAX, (uintmax_t) nr);
>             test_assert(loc, msg.buf, 0);
>             strbuf_release(&msg);
>             va_end(ap);
>             return;
>
> Which looks like it should be using test_msg() instead of writing a
> message to an strbuf and calling test_assert().

Erm, yes, good find.  You know something isn't right if you see an
assert(0) or similar..

> The conversion itself looks correct. It is a shame that each test has
> to have boilerplate to initalize and free the strvec but that comes
> from the existing implementation.

I see that as a benefit: You can see what each test does at a glance;
no tricks, no dependencies.  Just exercise steps and checks.

René

Re: [PATCH 6/6] t-strbuf: use TEST_RUN

[René Scharfe]

Am 02.07.24 um 17:14 schrieb phillip.wood123@gmail.com:
> Hi Josh and René
>
> On 01/07/2024 20:58, Josh Steadmon wrote:
>> On 2024.06.29 17:47, René Scharfe wrote:
>>
>> I think this commit in particular shows how TEST_RUN() is more
>> convenient than TEST(). (Although, arguably we shouldn't have allowed
>> the setup() + callback situation to start with.)
>
> I think the counterargument to that is that using TEST_RUN() makes
> the tests noisier and more error prone because each one has to be
> wrapped in an if() statement and has more boiler plate initializing
> and freeing the strbuf rather than getting that for free by calling
> the test function via setup().

I guess these are two sides of the same coin.  Explicit initialization
and cleanup is closer to what real strbuf users do, more idiomatic.
Which helps to see what the tests are actually doing.

The wrapping if is less annoying than a wrapping function that I have
to name and call.  The condition is just !ctx.skip_all, though, (for
now at least) so we could do something like:

   #define TEST_START(...) if (!TEST_RUN(__VA_ARGS__)) return test_done()

... and tests could look like that then:

   TEST_START("this and that");
   check(this);
   check(that);

   TEST_START("whatever");
   check(whatever);

Perhaps a bit too much magic, though?

René

Re: [PATCH 6/6] t-strbuf: use TEST_RUN

[René Scharfe]

Am 02.07.24 um 19:29 schrieb Ghanshyam Thakkar:
> Josh Steadmon <steadmon@google.com> wrote:
>>> -	TEST(setup(t_addch, "a"), "strbuf_addch adds char");
>>> -	TEST(setup(t_addch, ""), "strbuf_addch adds NUL char");
>>> -	TEST(setup_populated(t_addch, "initial value", "a"),
>>> -	     "strbuf_addch appends to initial value");
>>> -	TEST(setup(t_addstr, "hello there"), "strbuf_addstr adds string");
>>> -	TEST(setup_populated(t_addstr, "initial value", "hello there"),
>>> -	     "strbuf_addstr appends string to initial value");
>>> +
>>> +	if (TEST_RUN("strbuf_addch adds char")) {
>>> +		struct strbuf sb = STRBUF_INIT;
>>> +		t_addch(&sb, 'a');
>>> +		t_release(&sb);
>>> +	}
>>> +
>>> +	if (TEST_RUN("strbuf_addch adds NUL char")) {
>>> +		struct strbuf sb = STRBUF_INIT;
>>> +		t_addch(&sb, '\0');
>>> +		t_release(&sb);
>>> +	}
>>> +
>>> +	if (TEST_RUN("strbuf_addch appends to initial value")) {
>>> +		struct strbuf sb = STRBUF_INIT;
>>> +		t_addstr(&sb, "initial value");
>>> +		t_addch(&sb, 'a');
>>> +		t_release(&sb);
>>> +	}
>>> +
>>> +	if (TEST_RUN("strbuf_addstr adds string")) {
>>> +		struct strbuf sb = STRBUF_INIT;
>>> +		t_addstr(&sb, "hello there");
>>> +		t_release(&sb);
>>> +	}
>>> +
>>> +	if (TEST_RUN("strbuf_addstr appends string to initial value")) {
>>> +		struct strbuf sb = STRBUF_INIT;
>>> +		t_addstr(&sb, "initial value");
>>> +		t_addstr(&sb, "hello there");
>>> +		t_release(&sb);
>>> +	}
>>>
>>>  	return test_done();
>>>  }
>>> --
>>> 2.45.2
>>
>> I think this commit in particular shows how TEST_RUN() is more
>> convenient than TEST(). (Although, arguably we shouldn't have allowed
>> the setup() + callback situation to start with.)
>
> Could you expand a bit on why the setup() + callback thing shouldn't be
> allowed? I think it is a nice way of avoiding boilerplate and having
> independent state. And, I see the true potential of TEST_RUN() in
> testcases defined through macros rather than replacing functions. I
> actually think that the previous version with the functions was not
> particularly bad, and I agree with Phillip that the previous version's
> main() provided nice overview of the tests and it was easier to
> verify the independence between each testcase.

Each test uses its own strbuf and the t_ functions don't use global or
static variables, so how does the doubt about their independence creep
in?

With tests specified as above I now notice that we never check the
resulting string.  t_addch and t_addstr check the tail matches their
argument, but strbuf_addch and strbuf_addstr could overwrite the old
content and we wouldn't notice.  Perhaps it's not worth checking, but
now that tests look more like test_expect_success and all steps are
visible I somehow miss the equivalent of test_cmp; didn't see that
in the original version.

> Perhaps, the code snippets inside the functions are small enough to
> perceive TEST_RUN() as more convenient than TEST() in this test, but,
> for future reference, I definitely don't think TEST_RUN() should be
> looked at as a replacement for TEST(), and more like 'when we have to
> use macro magic which requires us to use internal test__run_*
> functions, using TEST_RUN() is more convenient. Patch [3/6] is a good> example of that.

Sure, not all tests will be improved by using TEST_RUN.  If TEST fits
better, use it.

> But, I also don't mind if patches 4, 5, or 6 get
> merged as I don't see any difference between using TEST_RUN() or
> TEST() in those patches, besides moving everything inside main().

The difference is that in the original version test description and
definition are separated, only linked by a function name.  The new
version brings them together and does away with function name.  A small
change, for sure, just to get rid of the artificial divide and the need
for that link.

René

Re: [PATCH 3/6] t-ctype: use TEST_RUN

[René Scharfe]

Am 02.07.24 um 17:14 schrieb phillip.wood123@gmail.com:
> Hi René
>
> On 01/07/2024 21:04, René Scharfe wrote:
>> Am 01.07.24 um 21:49 schrieb Josh Steadmon:
>>> On 2024.06.29 17:44, René Scharfe wrote:
>>>> Use the macro TEST_RUN instead of the internal functions
>>>> test__run_begin() and test__run_end().
>>>>
>>>> Signed-off-by: René Scharfe <l.s.r@web.de>
>>>
>>> Nitpick: please expand the commit message here as you did for the
>>> following patches.
>>
>> There is not much more to it: Use of internal functions is discouraged,
>> so this patch replaces their calls with the new public feature.  Perhaps
>> a "Kinda-requested-by: Phillip Wood <phillip.wood@dunelm.org.uk>" would
>> be appropriate?
>
> Perhaps something like
>
> These tests use the internal functions test__run_begin() and
> test__run_end() which are supposed to be private to the test
> framework. Convert them to use the newly added TEST_RUN() instead.
> would be sufficient?

That works for me.

René

Re: [PATCH 6/6] t-strbuf: use TEST_RUN

[Ghanshyam Thakkar]

René Scharfe <l.s.r@web.de> wrote:
> Am 02.07.24 um 19:29 schrieb Ghanshyam Thakkar:
> > Josh Steadmon <steadmon@google.com> wrote:
> >>> -	TEST(setup(t_addch, "a"), "strbuf_addch adds char");
> >>> -	TEST(setup(t_addch, ""), "strbuf_addch adds NUL char");
> >>> -	TEST(setup_populated(t_addch, "initial value", "a"),
> >>> -	     "strbuf_addch appends to initial value");
> >>> -	TEST(setup(t_addstr, "hello there"), "strbuf_addstr adds string");
> >>> -	TEST(setup_populated(t_addstr, "initial value", "hello there"),
> >>> -	     "strbuf_addstr appends string to initial value");
> >>> +
> >>> +	if (TEST_RUN("strbuf_addch adds char")) {
> >>> +		struct strbuf sb = STRBUF_INIT;
> >>> +		t_addch(&sb, 'a');
> >>> +		t_release(&sb);
> >>> +	}
> >>> +
> >>> +	if (TEST_RUN("strbuf_addch adds NUL char")) {
> >>> +		struct strbuf sb = STRBUF_INIT;
> >>> +		t_addch(&sb, '\0');
> >>> +		t_release(&sb);
> >>> +	}
> >>> +
> >>> +	if (TEST_RUN("strbuf_addch appends to initial value")) {
> >>> +		struct strbuf sb = STRBUF_INIT;
> >>> +		t_addstr(&sb, "initial value");
> >>> +		t_addch(&sb, 'a');
> >>> +		t_release(&sb);
> >>> +	}
> >>> +
> >>> +	if (TEST_RUN("strbuf_addstr adds string")) {
> >>> +		struct strbuf sb = STRBUF_INIT;
> >>> +		t_addstr(&sb, "hello there");
> >>> +		t_release(&sb);
> >>> +	}
> >>> +
> >>> +	if (TEST_RUN("strbuf_addstr appends string to initial value")) {
> >>> +		struct strbuf sb = STRBUF_INIT;
> >>> +		t_addstr(&sb, "initial value");
> >>> +		t_addstr(&sb, "hello there");
> >>> +		t_release(&sb);
> >>> +	}
> >>>
> >>>  	return test_done();
> >>>  }
> >>> --
> >>> 2.45.2
> >>
> >> I think this commit in particular shows how TEST_RUN() is more
> >> convenient than TEST(). (Although, arguably we shouldn't have allowed
> >> the setup() + callback situation to start with.)
> >
> > Could you expand a bit on why the setup() + callback thing shouldn't be
> > allowed? I think it is a nice way of avoiding boilerplate and having
> > independent state. And, I see the true potential of TEST_RUN() in
> > testcases defined through macros rather than replacing functions. I
> > actually think that the previous version with the functions was not
> > particularly bad, and I agree with Phillip that the previous version's
> > main() provided nice overview of the tests and it was easier to
> > verify the independence between each testcase.
>
> Each test uses its own strbuf and the t_ functions don't use global or
> static variables, so how does the doubt about their independence creep
> in?

Ah, apologies. I should clarify that I meant in general terms about the
future uses of TEST_RUN() and not about this particular patch. But I see
it being less of a problem now that I think about it more. And for the
record, I see no problems in this patch. But on a side note, with what
Phillip was suggesting to remove having TEST_RUN() inside if(), it
would definitely make verifying state independence more harder.

<snip>
> > But, I also don't mind if patches 4, 5, or 6 get
> > merged as I don't see any difference between using TEST_RUN() or
> > TEST() in those patches, besides moving everything inside main().
>
> The difference is that in the original version test description and
> definition are separated, only linked by a function name. The new
> version brings them together and does away with function name. A small
> change, for sure, just to get rid of the artificial divide and the need
> for that link.

Yeah, but I didn't mind that divide (and I don't mind bringing them
together either). :)

Thanks.

Re: [PATCH 6/6] t-strbuf: use TEST_RUN

[phillip.wood123]

On 02/07/2024 21:55, René Scharfe wrote:
> Am 02.07.24 um 17:14 schrieb phillip.wood123@gmail.com:
>> Hi Josh and René
>>
>> On 01/07/2024 20:58, Josh Steadmon wrote:
>>> On 2024.06.29 17:47, René Scharfe wrote:
>>>
>>> I think this commit in particular shows how TEST_RUN() is more
>>> convenient than TEST(). (Although, arguably we shouldn't have allowed
>>> the setup() + callback situation to start with.)
>>
>> I think the counterargument to that is that using TEST_RUN() makes
>> the tests noisier and more error prone because each one has to be
>> wrapped in an if() statement and has more boiler plate initializing
>> and freeing the strbuf rather than getting that for free by calling
>> the test function via setup().
> 
> I guess these are two sides of the same coin.  Explicit initialization
> and cleanup is closer to what real strbuf users do, more idiomatic.
> Which helps to see what the tests are actually doing.
> 
> The wrapping if is less annoying than a wrapping function that I have
> to name and call.  The condition is just !ctx.skip_all, though, (for
> now at least) so we could do something like:
> 
>     #define TEST_START(...) if (!TEST_RUN(__VA_ARGS__)) return test_done()
> 
> ... and tests could look like that then:
> 
>     TEST_START("this and that");
>     check(this);
>     check(that);
> 
>     TEST_START("whatever");
>     check(whatever);
> 
> Perhaps a bit too much magic, though?

I think so. It's an interesting idea, but at some point someone will 
probably want to be able to run a subset of the tests in a file like 
"--run" does for the integration tests so I don't think we want to 
assume we can skip all the tests just because any particular test should 
be skipped.

Best Wishes

Phillip

> René

Re: [PATCH 2/6] unit-tests: add TEST_RUN

[phillip.wood123]

Hi René

On 02/07/2024 21:55, René Scharfe wrote:
> Am 02.07.24 um 17:13 schrieb phillip.wood123@gmail.com:
>> On 29/06/2024 16:43, René Scharfe wrote:
>>> The macro TEST only allows defining a test that consists of a
>>> single expression.  Add the new macro, TEST_RUN, which provides a
>>> way to define unit tests that are made up of one or more
>>> statements.  A test started with it implicitly ends when the next
>>> test is started or test_done() is called.
>>>
>>> TEST_RUN allows defining self-contained tests en bloc, a bit like
>>> test_expect_success does for regular tests.  Unlike TEST it does
>>> not require defining wrapper functions for test statements.
>>
>> There are pros and cons to not requiring one function per test. It
>> can be a pain to have to write separate functions but it keeps each
>> test self contained which hopefully makes it harder to have
>> accidental dependencies between tests. Having separate functions for
>> each test makes it easy to initialize and free resources for every
>> test by writing a setup() function that initializes the resources,
>> calls the test function and then frees the resources.
> 
> Right.  We should use TEST and TEST_RUN when appropriate.
> 
>> The changes in patch 6 to use TEST_RUN() mean that each test now has
>> more boilerplate to initialize and free the strbuf. > This makes them more similar to strbuf usage in the wild.  Using
> the API idiomatically just makes more sense to me.

I see what you mean. I think it only looks idiomatic if you're already 
familiar with the api though as the test bodies call wrappers rather 
than using the strbuf api directly. I think that reduces its value as an 
example of idomatic usage for someone who is not familiar with the 
strbuf api.

>  Not hiding
> initialization and release makes the tests visibly independent.
> This is not enforced by TEST_RUN, but made possible by it.
> 
>> Having each test in its own function also makes main() shorter and
>> which means can quickly get an overview of all the test cases from
>> it.
> 
> That's true, now you need to grep for TEST_RUN to get such an
> overview.
> 
> On the other hand I find the start of the description in TEST
> invocations somewhat hard to locate, as they are not vertically
> aligned due to the preceding variable-length function name.  Just
> saying..

Yes I really wanted the first argument of TEST to be the description but 
that isn't easy to do while supporting printf style format strings.

>>> +int test__run(const char *location, const char *format, ...)
>>> +{
>>> +    va_list ap;
>>> +    char *desc;
>>> +
>>> +    test__run_maybe_end();
>>> +
>>> +    va_start(ap, format);
>>> +    desc = xstrvfmt(format, ap);
>>
>> This uses an strbuf under the hood. So far we've avoided doing that
>> as we want to be able to test the strbuf implementation with this
>> test framework. We don't need to support arbitrary length strings
>> here so we could use a fixed array and xsnprinf() instead.
> 
> Fair point.  xsnprinf() might be a bit too strict, as it doesn't
> handle short buffers gracefully.  Perhaps that's OK; a developer
> getting hit by that could simply increase the buffer size.

I think so.

> We could also let xstrvfmt() call vsnprintf(3) directly.  The code
> duplication would be a bit grating, but perhaps there's some good
> base function hidden in there somewhere.

Oh, interesting - maybe something like

char* xstrvfmt(const char *fmt, ...)
{
	va_list ap, aq;

	va_start(ap, fmt);
	va_copy(aq, ap);
	len = vnsprintf(NULL, 0, fmt, ap);
	if (len < 0)
		BUG(...)
	buf = xmalloc(len + 1);
	if (vnsprintf(buf, len + 1, fmt, aq) != len)
		BUG(...)
	va_end(aq);
	va_end(ap);

	return buf;
}

>> Looking ahead the plan seems to be to change most instances of TEST()
>> to TEST_RUN(). If we are going to go that way perhaps we should steal
>> TEST() for this macro and rename the existing TEST() macro.
> 
> Not my plan, at least -- I'm content with just having the *ability*
> to keep all parts of a test together.

That sounds sensible to me

> +	if (TEST_RUN("static initialization works")) {
> +		struct strbuf buf = STRBUF_INIT;
> +		if (!check_uint(buf.len, ==, 0) ||
> +		    !check_uint(buf.alloc, ==, 0) ||
> +		    !check_char(buf.buf[0], ==, '\0'))
> +			test_skip_all("STRBUF_INIT is broken");
> +	}

that's a nice use of test_skip_all()

>> I'm not very enthusiastic about requiring the test author to wrap
>> TEST_RUN() in an if() statement rather than just doing that for them.
>> It makes it explicit but from the test author's point of view it just
>> feels like pointless boilerplate.
> 
> Hmm.  We can add more magic, but I suspect that it might confuse
> developers and editors.

To me its confusing to have to wrap TEST_RUN() in an if() statement 
until one realizes that the test might be skipped. If we document that 
the test body should be enclosed in braces I don't think it should 
confuse developers or editors and will keep the tests a bit cleaner.

Best Wishes

Phillip

Re: [PATCH 2/6] unit-tests: add TEST_RUN

[René Scharfe]

Am 05.07.24 um 11:42 schrieb phillip.wood123@gmail.com:
> On 02/07/2024 21:55, René Scharfe wrote:
>> Am 02.07.24 um 17:13 schrieb phillip.wood123@gmail.com:
>>
>>> The changes in patch 6 to use TEST_RUN() mean that each test now has
>>> more boilerplate to initialize and free the strbuf.
>>
>> This makes them more similar to strbuf usage in the wild.  Using
>> the API idiomatically just makes more sense to me.
>
> I see what you mean. I think it only looks idiomatic if you're
> already familiar with the api though as the test bodies call wrappers
> rather than using the strbuf api directly. I think that reduces its
> value as an example of idomatic usage for someone who is not familiar
> with the strbuf api.

In early versions I used the original names by adding these just before
main():

   #define strbuf_addch(x, y)  t_addch(x, y)
   #define strbuf_addstr(x, y) t_addstr(x, y)
   #define strbuf_release(x)   t_release(x)

This allowed normal looking code to be used in tests, with checks
injected behind the scenes.  Rejected it for v1 because it offers no
structural improvements, just optics.  It does allow to forget the
checked versions when writing tests, though, so perhaps it's still
worth doing.

>> We could also let xstrvfmt() call vsnprintf(3) directly.  The code
>> duplication would be a bit grating, but perhaps there's some good
>> base function hidden in there somewhere.
>
> Oh, interesting - maybe something like
>
> char* xstrvfmt(const char *fmt, ...)
> {
>     va_list ap, aq;
>
>     va_start(ap, fmt);
>     va_copy(aq, ap);
>     len = vnsprintf(NULL, 0, fmt, ap);
>     if (len < 0)
>         BUG(...)
>     buf = xmalloc(len + 1);
>     if (vnsprintf(buf, len + 1, fmt, aq) != len)
>         BUG(...)
>     va_end(aq);
>     va_end(ap);
>
>     return buf;
> }

Yes.  Though the current version allocates 65 bytes in the first try
and only needs to call vnsprintf(3) once if the output fits in.  No
longer doing that might affect the performance of some callers in a
noticeable way.

>>> I'm not very enthusiastic about requiring the test author to wrap
>>> TEST_RUN() in an if() statement rather than just doing that for them.
>>> It makes it explicit but from the test author's point of view it just
>>> feels like pointless boilerplate.
>>
>> Hmm.  We can add more magic, but I suspect that it might confuse
>> developers and editors.
>
> To me its confusing to have to wrap TEST_RUN() in an if() statement
> until one realizes that the test might be skipped. If we document
> that the test body should be enclosed in braces I don't think it
> should confuse developers or editors and will keep the tests a bit
> cleaner.

You don't need braces in either case.  I.e. something like

	if (TEST_RUN("foo"))
		foo();

works fine.  And

	#define IF_TEST_RUN(...) if (TEST_RUN(__VA_ARGS__))
	IF_TEST_RUN("foo")
		foo();

works fine as well.

The confusion that I'm afraid of is that we'd effectively invent a new
control flow keyword here, which is unusual.  There are precedents,
though: foreach macros like for_each_string_list_item.  We tell
clang-format about them using its option ForEachMacros.  I see it also
has an option IfMacros since version 13 (unused by us so far).  Hmm.

René

Re: [PATCH 2/6] unit-tests: add TEST_RUN

[René Scharfe]

Am 05.07.24 um 20:01 schrieb René Scharfe:
> Am 05.07.24 um 11:42 schrieb phillip.wood123@gmail.com:
>> On 02/07/2024 21:55, René Scharfe wrote:
>>> Am 02.07.24 um 17:13 schrieb phillip.wood123@gmail.com:
>>>
>>>> I'm not very enthusiastic about requiring the test author to wrap
>>>> TEST_RUN() in an if() statement rather than just doing that for them.
>>>> It makes it explicit but from the test author's point of view it just
>>>> feels like pointless boilerplate.
>>>
>>> Hmm.  We can add more magic, but I suspect that it might confuse
>>> developers and editors.
>>
>> To me its confusing to have to wrap TEST_RUN() in an if() statement
>> until one realizes that the test might be skipped. If we document
>> that the test body should be enclosed in braces I don't think it
>> should confuse developers or editors and will keep the tests a bit
>> cleaner.
>
> You don't need braces in either case.  I.e. something like
>
> 	if (TEST_RUN("foo"))
> 		foo();
>
> works fine.  And
>
> 	#define IF_TEST_RUN(...) if (TEST_RUN(__VA_ARGS__))
> 	IF_TEST_RUN("foo")
> 		foo();
>
> works fine as well.
>
> The confusion that I'm afraid of is that we'd effectively invent a new
> control flow keyword here, which is unusual.  There are precedents,
> though: foreach macros like for_each_string_list_item.  We tell
> clang-format about them using its option ForEachMacros.  I see it also
> has an option IfMacros since version 13 (unused by us so far).  Hmm.

Hmm, again.  I can see the appeal.  How to call it?  Given that the
foreach macros have lower-case names, perhaps to indicate that they act
as control flow statements, not function-like macro calls, would we want
lower case here as well?

	#define test(...) if (TEST_RUN(__VA_ARGS__))

	        test ("passing test")
	                check(1);

		test ("split single item") {
			struct strvec vec = STRVEC_INIT;
			strvec_split(&vec, "foo");
			check_strvec(&vec, "foo", NULL);
			strvec_clear(&vec);
		}

Can't get much cleaner than that.  Requires learning that this macro is
not function-like, but it's probably not too much to ask.

René

Re: [PATCH 2/6] unit-tests: add TEST_RUN

[phillip.wood123]

Hi René

On 05/07/2024 19:01, René Scharfe wrote:
> Am 05.07.24 um 11:42 schrieb phillip.wood123@gmail.com:
>> On 02/07/2024 21:55, René Scharfe wrote:
>>> Am 02.07.24 um 17:13 schrieb phillip.wood123@gmail.com:
>>>
>>>> The changes in patch 6 to use TEST_RUN() mean that each test now has
>>>> more boilerplate to initialize and free the strbuf.
>>>
>>> This makes them more similar to strbuf usage in the wild.  Using
>>> the API idiomatically just makes more sense to me.
>>
>> I see what you mean. I think it only looks idiomatic if you're
>> already familiar with the api though as the test bodies call wrappers
>> rather than using the strbuf api directly. I think that reduces its
>> value as an example of idomatic usage for someone who is not familiar
>> with the strbuf api.
> 
> In early versions I used the original names by adding these just before
> main():
> 
>     #define strbuf_addch(x, y)  t_addch(x, y)
>     #define strbuf_addstr(x, y) t_addstr(x, y)
>     #define strbuf_release(x)   t_release(x)
> 
> This allowed normal looking code to be used in tests, with checks
> injected behind the scenes.  Rejected it for v1 because it offers no
> structural improvements, just optics.  It does allow to forget the
> checked versions when writing tests, though, so perhaps it's still
> worth doing.

It also obscures what the test is doing though. It's nice if unit tests 
show how to use an API but I don't think we should let that be our 
overriding concern.

>>> We could also let xstrvfmt() call vsnprintf(3) directly.  The code
>>> duplication would be a bit grating, but perhaps there's some good
>>> base function hidden in there somewhere.
>>
>> Oh, interesting - maybe something like
>>
>> char* xstrvfmt(const char *fmt, ...)
>> {
>>      va_list ap, aq;
>>
>>      va_start(ap, fmt);
>>      va_copy(aq, ap);
>>      len = vnsprintf(NULL, 0, fmt, ap);
>>      if (len < 0)
>>          BUG(...)
>>      buf = xmalloc(len + 1);
>>      if (vnsprintf(buf, len + 1, fmt, aq) != len)
>>          BUG(...)
>>      va_end(aq);
>>      va_end(ap);
>>
>>      return buf;
>> }
> 
> Yes.  Though the current version allocates 65 bytes in the first try
> and only needs to call vnsprintf(3) once if the output fits in.  No
> longer doing that might affect the performance of some callers in a
> noticeable way.

Good point

>>>> I'm not very enthusiastic about requiring the test author to wrap
>>>> TEST_RUN() in an if() statement rather than just doing that for them.
>>>> It makes it explicit but from the test author's point of view it just
>>>> feels like pointless boilerplate.
>>>
>>> Hmm.  We can add more magic, but I suspect that it might confuse
>>> developers and editors.
>>
>> To me its confusing to have to wrap TEST_RUN() in an if() statement
>> until one realizes that the test might be skipped. If we document
>> that the test body should be enclosed in braces I don't think it
>> should confuse developers or editors and will keep the tests a bit
>> cleaner.
> 
> You don't need braces in either case.  I.e. something like
> 
> 	if (TEST_RUN("foo"))
> 		foo();
> 
> works fine.  And
> 
> 	#define IF_TEST_RUN(...) if (TEST_RUN(__VA_ARGS__))
> 	IF_TEST_RUN("foo")
> 		foo();
> 
> works fine as well.

Indeed. The reason I suggested requiring braces was to help editors 
indent the code correctly and because it gives a nice visual cue to see 
what is in each test.

Best Wishes

Phillip

> The confusion that I'm afraid of is that we'd effectively invent a new
> control flow keyword here, which is unusual.  There are precedents,
> though: foreach macros like for_each_string_list_item.  We tell
> clang-format about them using its option ForEachMacros.  I see it also
> has an option IfMacros since version 13 (unused by us so far).  Hmm.
> 
> René

Re: [PATCH 2/6] unit-tests: add TEST_RUN

[phillip.wood123]

Hi René

On 07/07/2024 08:20, René Scharfe wrote:
> Hmm, again.  I can see the appeal.  How to call it?  Given that the
> foreach macros have lower-case names, perhaps to indicate that they act
> as control flow statements, not function-like macro calls, would we want
> lower case here as well?

I'd automatically assumed we'd want an uppercase name to signal that it 
was a pre-processor macro but I've not really spent any time thinking 
about it.

> 	#define test(...) if (TEST_RUN(__VA_ARGS__))
> 
> 	        test ("passing test")
> 	                check(1);
> 
> 		test ("split single item") {
> 			struct strvec vec = STRVEC_INIT;
> 			strvec_split(&vec, "foo");
> 			check_strvec(&vec, "foo", NULL);
> 			strvec_clear(&vec);
> 		}
> 
> Can't get much cleaner than that.

Yes, I like it

> Requires learning that this macro is
> not function-like, but it's probably not too much to ask.

For someone new to the project it should hopefully be pretty clear how 
to use it from the existing tests once we have a few more test files 
that use it. Maybe an uppercase name would signal that it is special?

Best Wishes

Phillip

Re: [PATCH 2/6] unit-tests: add TEST_RUN

[Junio C Hamano]

phillip.wood123@gmail.com writes:

> Hi René
>
> On 07/07/2024 08:20, René Scharfe wrote:
>> Hmm, again.  I can see the appeal.  How to call it?  Given that the
>> foreach macros have lower-case names, perhaps to indicate that they act
>> as control flow statements, not function-like macro calls, would we want
>> lower case here as well?
>
> I'd automatically assumed we'd want an uppercase name to signal that
> it was a pre-processor macro but I've not really spent any time
> thinking about it.
>
>> 	#define test(...) if (TEST_RUN(__VA_ARGS__))
>> 	        test ("passing test")
>> 	                check(1);
>> 		test ("split single item") {
>> 			struct strvec vec = STRVEC_INIT;
>> 			strvec_split(&vec, "foo");
>> 			check_strvec(&vec, "foo", NULL);
>> 			strvec_clear(&vec);
>> 		}
>> Can't get much cleaner than that.
>
> Yes, I like it

Yeah, if you squint your eyes hard, it starts to look a bit like
test_expect_success we use in the test suite ;-)

Isn't this introducing a new control structure to the language?

A macro that is a conditional switch (aka "if"-like statement),
having "if" in the name somewhere, and a macro that wrapts a loop
around a block (aka "for/while" like statement), having "for" in the
name somewhere, might be less confusing for the uninitiated.

> For someone new to the project it should hopefully be pretty clear how
> to use it from the existing tests once we have a few more test files
> that use it. Maybe an uppercase name would signal that it is special?

As to the cases, I personally prefer lowercase names whose semantics
is very clear, just like e.g., kh_foreach() makes it clear with
"foreach" that it iterates over the set.  But the above "test" may
not qualify---it is not making it sufficiently clear what control
structure it is creating with its name.

Thanks.

Re: [PATCH 6/6] t-strbuf: use TEST_RUN

[Josh Steadmon]

On 2024.07.02 22:59, Ghanshyam Thakkar wrote:
> Josh Steadmon <steadmon@google.com> wrote:
> > > -	TEST(setup(t_addch, "a"), "strbuf_addch adds char");
> > > -	TEST(setup(t_addch, ""), "strbuf_addch adds NUL char");
> > > -	TEST(setup_populated(t_addch, "initial value", "a"),
> > > -	     "strbuf_addch appends to initial value");
> > > -	TEST(setup(t_addstr, "hello there"), "strbuf_addstr adds string");
> > > -	TEST(setup_populated(t_addstr, "initial value", "hello there"),
> > > -	     "strbuf_addstr appends string to initial value");
> > > +
> > > +	if (TEST_RUN("strbuf_addch adds char")) {
> > > +		struct strbuf sb = STRBUF_INIT;
> > > +		t_addch(&sb, 'a');
> > > +		t_release(&sb);
> > > +	}
> > > +
> > > +	if (TEST_RUN("strbuf_addch adds NUL char")) {
> > > +		struct strbuf sb = STRBUF_INIT;
> > > +		t_addch(&sb, '\0');
> > > +		t_release(&sb);
> > > +	}
> > > +
> > > +	if (TEST_RUN("strbuf_addch appends to initial value")) {
> > > +		struct strbuf sb = STRBUF_INIT;
> > > +		t_addstr(&sb, "initial value");
> > > +		t_addch(&sb, 'a');
> > > +		t_release(&sb);
> > > +	}
> > > +
> > > +	if (TEST_RUN("strbuf_addstr adds string")) {
> > > +		struct strbuf sb = STRBUF_INIT;
> > > +		t_addstr(&sb, "hello there");
> > > +		t_release(&sb);
> > > +	}
> > > +
> > > +	if (TEST_RUN("strbuf_addstr appends string to initial value")) {
> > > +		struct strbuf sb = STRBUF_INIT;
> > > +		t_addstr(&sb, "initial value");
> > > +		t_addstr(&sb, "hello there");
> > > +		t_release(&sb);
> > > +	}
> > > 
> > >  	return test_done();
> > >  }
> > > --
> > > 2.45.2
> >
> > I think this commit in particular shows how TEST_RUN() is more
> > convenient than TEST(). (Although, arguably we shouldn't have allowed
> > the setup() + callback situation to start with.)
> 
> Could you expand a bit on why the setup() + callback thing shouldn't be
> allowed? I think it is a nice way of avoiding boilerplate and having
> independent state.

It's a matter of taste rather than strict principles, I suppose, but I
think that test cases should focus on being readable rather than
avoiding duplication. Production code can follow a Don't Repeat Yourself
philosophy, but when a test breaks, it's better to be able to look at
only the failing test and quickly see what's wrong, rather than having
to work out which functions call which callbacks and what gets verified
where. Also, since we don't have tests for our tests, it's important
that the tests are easy for people to manually verify them.

I also agree with René's point about the test cases matching real-world
API usage.

I do agree that having separate functions improves isolation, but
hopefully we can encourage test authors to define local variables in
their TEST_RUN blocks.

> And, I see the true potential of TEST_RUN() in
> testcases defined through macros rather than replacing functions. I
> actually think that the previous version with the functions was not
> particularly bad, and I agree with Phillip that the previous version's
> main() provided nice overview of the tests and it was easier to
> verify the independence between each testcase.
> 
> Perhaps, the code snippets inside the functions are small enough to
> perceive TEST_RUN() as more convenient than TEST() in this test, but,
> for future reference, I definitely don't think TEST_RUN() should be
> looked at as a replacement for TEST(), and more like 'when we have to
> use macro magic which requires us to use internal test__run_*
> functions, using TEST_RUN() is more convenient'. Patch [3/6] is a good
> example of that. But, I also don't mind if patches 4, 5, or 6 get
> merged as I don't see any difference between using TEST_RUN() or
> TEST() in those patches, besides moving everything inside main().
> 
> Thanks.

Re: [PATCH 6/6] t-strbuf: use TEST_RUN

[Ghanshyam Thakkar]

Josh Steadmon <steadmon@google.com> wrote:
> On 2024.07.02 22:59, Ghanshyam Thakkar wrote:
> > Josh Steadmon <steadmon@google.com> wrote:
> > > I think this commit in particular shows how TEST_RUN() is more
> > > convenient than TEST(). (Although, arguably we shouldn't have allowed
> > > the setup() + callback situation to start with.)
> > 
> > Could you expand a bit on why the setup() + callback thing shouldn't be
> > allowed? I think it is a nice way of avoiding boilerplate and having
> > independent state.
>
> It's a matter of taste rather than strict principles, I suppose, but I
> think that test cases should focus on being readable rather than
> avoiding duplication. Production code can follow a Don't Repeat Yourself
> philosophy, but when a test breaks, it's better to be able to look at
> only the failing test and quickly see what's wrong, rather than having
> to work out which functions call which callbacks and what gets verified
> where. Also, since we don't have tests for our tests, it's important
> that the tests are easy for people to manually verify them.

Makes sense. But the manner in which callbacks are used in these tests,
which usually only initialize a certain datastructure and then release them at
the end, I fail to see them as more harder to manually verify. But it is
very easy to stretch the line and perhaps do more than initialization in
the setup(), so I can see how disallowing them would get rid of the need of
judging every time wether they are readable or not. Maybe a 'best
practices' document would help in that regard? 

> I also agree with René's point about the test cases matching real-world
> API usage.
>
> I do agree that having separate functions improves isolation, but
> hopefully we can encourage test authors to define local variables in
> their TEST_RUN blocks.

Makes sense.

Thanks.

Re: [PATCH 6/6] t-strbuf: use TEST_RUN

[Phillip Wood]

On 02/07/2024 16:14, phillip.wood123@gmail.com wrote:
> Getting rid of the untyped test arguments is 
> definitely a benefit of this approach.

That got me thinking how we might make type-safe setup()
functions. The diff below shows how we could define a macro to
generate the functions. DEFINE_SETUP_FN(char, ch) defines setup_ch()
that takes a test function and a char that is passed to the test with
the initialized strbuf. I'm not sure that's the way we want to go for
this test file but I thought I'd post it in case it is useful for
future tests.

Best Wishes

Phillip

---- >8 ----
diff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c
index 6027dafef70..8fc9a8b38df 100644
--- a/t/unit-tests/t-strbuf.c
+++ b/t/unit-tests/t-strbuf.c
@@ -1,27 +1,60 @@
  #include "test-lib.h"
  #include "strbuf.h"
  
-/* wrapper that supplies tests with an empty, initialized strbuf */
-static void setup(void (*f)(struct strbuf*, const void*),
-		  const void *data)
-{
-	struct strbuf buf = STRBUF_INIT;
-
-	f(&buf, data);
-	strbuf_release(&buf);
-	check_uint(buf.len, ==, 0);
-	check_uint(buf.alloc, ==, 0);
-}
-
-/* wrapper that supplies tests with a populated, initialized strbuf */
-static void setup_populated(void (*f)(struct strbuf*, const void*),
-			    const char *init_str, const void *data)
-{
-	struct strbuf buf = STRBUF_INIT;
-
-	strbuf_addstr(&buf, init_str);
-	check_uint(buf.len, ==, strlen(init_str));
-	f(&buf, data);
+/*
+ * Define a type safe wrapper function that supplies test functions
+ * with an initialized strbuf populated with an optional string and
+ * some data and then frees the strbuf when the test function
+ * returns. For example given the test function
+ *
+ *      t_foo(struct strbuf *buf, struct foo *data)
+ *
+ * the type safe wrapper function
+ *
+ *     setup_foo(void(*)(struct strbuf*, const struct foo*),
+ *     		 const char *init_str, const struct foo*)
+ *
+ * can be defined with
+ *
+ *     DEFINE_SETUP_FN(const struct foo*, foo)
+ *
+ * and used to run t_foo() with
+ *
+ *     TEST(setup_foo(t_foo, "initial string", &my_foo), "test foo");
+ */
+#define DEFINE_SETUP_FN(type, suffix) \
+	static void marshal_##suffix(void(*test_fn)(void),		     \
+				      struct strbuf *buf, const void *ctx)   \
+	{								     \
+		type data = *(type *)ctx;				     \
+		((void(*)(struct strbuf*, type)) test_fn)(buf, data);	     \
+	}								     \
+									     \
+	static void setup_##suffix(void(*test_fn)(struct strbuf*, type),     \
+				   const char *init_str, type data)	     \
+	{								     \
+		void *ctx = &data;					     \
+		do_setup(init_str, (void(*)(void)) test_fn, ctx,	     \
+			 marshal_##suffix);				     \
+	}
+
+/*
+ * Helper function for DEFINE_SETUP_FN() that initializes the strbuf,
+ * calls the test function and releases the strbuf
+ */
+static void do_setup(const char* init_str, void(*f)(void), const void *ctx,
+		     void(*marshal)(void(*)(void), struct strbuf*, const void*))
+{
+	struct strbuf buf = STRBUF_INIT;
+
+	if (init_str) {
+		strbuf_addstr(&buf, init_str);
+		if (!check_uint(buf.len, ==, strlen(init_str))) {
+			strbuf_release(&buf);
+			return;
+		}
+	}
+	marshal(f, &buf, ctx);
  	strbuf_release(&buf);
  	check_uint(buf.len, ==, 0);
  	check_uint(buf.alloc, ==, 0);
@@ -66,10 +99,9 @@ static void t_dynamic_init(void)
  	strbuf_release(&buf);
  }
  
-static void t_addch(struct strbuf *buf, const void *data)
+DEFINE_SETUP_FN(char, ch)
+static void t_addch(struct strbuf *buf, char ch)
  {
-	const char *p_ch = data;
-	const char ch = *p_ch;
  	size_t orig_alloc = buf->alloc;
  	size_t orig_len = buf->len;
  
@@ -85,9 +117,9 @@ static void t_addch(struct strbuf *buf, const void *data)
  	check_char(buf->buf[buf->len], ==, '\0');
  }
  
-static void t_addstr(struct strbuf *buf, const void *data)
+DEFINE_SETUP_FN(const char*, str)
+static void t_addstr(struct strbuf *buf, const char *text)
  {
-	const char *text = data;
  	size_t len = strlen(text);
  	size_t orig_alloc = buf->alloc;
  	size_t orig_len = buf->len;
@@ -110,12 +142,12 @@ int cmd_main(int argc, const char **argv)
  	if (!TEST(t_static_init(), "static initialization works"))
  		test_skip_all("STRBUF_INIT is broken");
  	TEST(t_dynamic_init(), "dynamic initialization works");
-	TEST(setup(t_addch, "a"), "strbuf_addch adds char");
-	TEST(setup(t_addch, ""), "strbuf_addch adds NUL char");
-	TEST(setup_populated(t_addch, "initial value", "a"),
+	TEST(setup_ch(t_addch, NULL, 'a'), "strbuf_addch adds char");
+	TEST(setup_ch(t_addch, NULL, '\0'), "strbuf_addch adds NUL char");
+	TEST(setup_ch(t_addch, "initial value", 'a'),
  	     "strbuf_addch appends to initial value");
-	TEST(setup(t_addstr, "hello there"), "strbuf_addstr adds string");
-	TEST(setup_populated(t_addstr, "initial value", "hello there"),
+	TEST(setup_str(t_addstr, NULL, "hello there"), "strbuf_addstr adds string");
+	TEST(setup_str(t_addstr, "initial value", "hello there"),
  	     "strbuf_addstr appends string to initial value");
  
  	return test_done();

Re: [PATCH 0/6] unit-tests: add and use TEST_RUN to simplify tests

[Junio C Hamano]

René Scharfe <l.s.r@web.de> writes:

> The macro TEST only allows defining a test that consists of a single
> expression.  This requires wrapping tests made up of one or more
> statements in a function, which is a small, but avoidable hurdle.  This
> series provides a new macro, TEST_RUN, that provides a way to define
> tests without requiring to declare a function.
>
>   t0080: move expected output to a file
>   unit-tests: add TEST_RUN
>   t-ctype: use TEST_RUN
>   t-reftable-basics: use TEST_RUN
>   t-strvec: use TEST_RUN
>   t-strbuf: use TEST_RUN
>
>  t/helper/test-example-tap.c      |  33 +++
>  t/t0080-unit-test-output.sh      |  48 +----
>  t/t0080/expect                   |  76 +++++++
>  t/unit-tests/t-ctype.c           |   4 +-
>  t/unit-tests/t-reftable-basics.c | 228 +++++++++-----------
>  t/unit-tests/t-strbuf.c          |  79 +++----
>  t/unit-tests/t-strvec.c          | 356 ++++++++++++++-----------------
>  t/unit-tests/test-lib.c          |  42 +++-
>  t/unit-tests/test-lib.h          |   8 +
>  9 files changed, 462 insertions(+), 412 deletions(-)
>  create mode 100644 t/t0080/expect

So, looking back the discussion list on

  https://lore.kernel.org/git/85b6b8a9-ee5f-42ab-bcbc-49976b30ef33@web.de/

any loose ends still need to be addressed?  I didn't spot any
myself, so I am willing to merge it to 'next' soonish, but please
stop me if there were something I missed.

Thanks.

Re: [PATCH 0/6] unit-tests: add and use TEST_RUN to simplify tests

[Phillip Wood]

On 10/07/2024 23:13, Junio C Hamano wrote:
> René Scharfe <l.s.r@web.de> writes:
>
> So, looking back the discussion list on
> 
>    https://lore.kernel.org/git/85b6b8a9-ee5f-42ab-bcbc-49976b30ef33@web.de/
> 
> any loose ends still need to be addressed?  I didn't spot any
> myself, so I am willing to merge it to 'next' soonish, but please
> stop me if there were something I missed.

I thought René was planning a re-roll to avoid using xstrfmt() in Patch 
2 c.f <97390954-49bc-48c4-bab1-95be10717aca@web.de>. Also I'm not sure 
we've reached a conclusion on whether to include the "if" in the macro 
or require the user to write "if(TEST_RUN(...))". My impression is that 
there is a consensus building around having the macro include the "if" 
but we haven't decided what to call it c.f. 
<62d221cc-532a-4a6d-8e96-b5a246ddeb1b@web.de>

Best Wishes

Phillip

Re: [PATCH 0/6] unit-tests: add and use TEST_RUN to simplify tests

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

> On 10/07/2024 23:13, Junio C Hamano wrote:
>> René Scharfe <l.s.r@web.de> writes:
>>
>> So, looking back the discussion list on
>>    https://lore.kernel.org/git/85b6b8a9-ee5f-42ab-bcbc-49976b30ef33@web.de/
>> any loose ends still need to be addressed?  I didn't spot any
>> myself, so I am willing to merge it to 'next' soonish, but please
>> stop me if there were something I missed.
>
> I thought René was planning a re-roll to avoid using xstrfmt() in
> Patch 2 c.f <97390954-49bc-48c4-bab1-95be10717aca@web.de>. Also I'm
> not sure we've reached a conclusion on whether to include the "if" in
> the macro or require the user to write "if(TEST_RUN(...))". My
> impression is that there is a consensus building around having the
> macro include the "if" but we haven't decided what to call it
> c.f. <62d221cc-532a-4a6d-8e96-b5a246ddeb1b@web.de>
>
> Best Wishes

Thanks.  Very much appreciated.

Re: [PATCH 2/6] unit-tests: add TEST_RUN

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

>> I'd automatically assumed we'd want an uppercase name to signal that
>> it was a pre-processor macro but I've not really spent any time
>> thinking about it.
>>
>>> 	#define test(...) if (TEST_RUN(__VA_ARGS__))
>>> 	        test ("passing test")
>>> 	                check(1);
> ...
> Isn't this introducing a new control structure to the language?
>
> A macro that is a conditional switch (aka "if"-like statement),
> having "if" in the name somewhere, and a macro that wrapts a loop
> around a block (aka "for/while" like statement), having "for" in the
> name somewhere, might be less confusing for the uninitiated.

So, perhaps test_if_XXXXXX() but it is not quite clear to me when
TEST_RUN() wants to return true, so I cannot come up with an
appropriate value to fill the XXXXXX part.  If this is about
honoring GIT_SKIP_TESTS or something similar, then I may suggest
test_if_enabled(), but that does not seem like it.  So...

Re: [PATCH 2/6] unit-tests: add TEST_RUN

[Phillip Wood]

On 11/07/2024 16:34, Junio C Hamano wrote:
> Junio C Hamano <gitster@pobox.com> writes:
> 
> So, perhaps test_if_XXXXXX() but it is not quite clear to me when
> TEST_RUN() wants to return true, so I cannot come up with an
> appropriate value to fill the XXXXXX part.  If this is about
> honoring GIT_SKIP_TESTS or something similar, then I may suggest
> test_if_enabled(), but that does not seem like it.  So...

TEST_RUN() returns true if the test has not been skipped i.e. the test 
should be run. At the moment the only way to skip a test is to call 
test_skip_all() in a previous test. In the future I expect we'll add 
something like the "--run" option we have for the integration tests.

Best Wishes

Phillip

Re: [PATCH 2/6] unit-tests: add TEST_RUN

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

> On 11/07/2024 16:34, Junio C Hamano wrote:
>> Junio C Hamano <gitster@pobox.com> writes:
>> So, perhaps test_if_XXXXXX() but it is not quite clear to me when
>> TEST_RUN() wants to return true, so I cannot come up with an
>> appropriate value to fill the XXXXXX part.  If this is about
>> honoring GIT_SKIP_TESTS or something similar, then I may suggest
>> test_if_enabled(), but that does not seem like it.  So...
>
> TEST_RUN() returns true if the test has not been skipped i.e. the test
> should be run. At the moment the only way to skip a test is to call
> test_skip_all() in a previous test. In the future I expect we'll add
> something like the "--run" option we have for the integration tests.

Thanks, so test_if_enabled() is not all that off the mark after all.

Re: [PATCH 0/6] unit-tests: add and use TEST_RUN to simplify tests

[René Scharfe]

Am 11.07.24 um 12:05 schrieb Phillip Wood:
> On 10/07/2024 23:13, Junio C Hamano wrote:
>> René Scharfe <l.s.r@web.de> writes:
>>
>> So, looking back the discussion list on
>>
>>    https://lore.kernel.org/git/85b6b8a9-ee5f-42ab-bcbc-49976b30ef33@web.de/
>>
>> any loose ends still need to be addressed?  I didn't spot any
>> myself, so I am willing to merge it to 'next' soonish, but please
>> stop me if there were something I missed.
>
> I thought René was planning a re-roll to avoid using xstrfmt() in
> Patch 2 c.f <97390954-49bc-48c4-bab1-95be10717aca@web.de>. Also I'm
> not sure we've reached a conclusion on whether to include the "if" in
> the macro or require the user to write "if(TEST_RUN(...))". My
> impression is that there is a consensus building around having the
> macro include the "if" but we haven't decided what to call it c.f.
> <62d221cc-532a-4a6d-8e96-b5a246ddeb1b@web.de>
Right, thanks, and I'm back now from a surprisingly exhausting business
trip.  Will send v3 soonish.

René

Re: [PATCH 6/6] t-strbuf: use TEST_RUN

[René Scharfe]

Am 10.07.24 um 15:55 schrieb Phillip Wood:
> On 02/07/2024 16:14, phillip.wood123@gmail.com wrote:
>> Getting rid of the untyped test arguments is definitely a benefit of this approach.
>
> That got me thinking how we might make type-safe setup()
> functions. The diff below shows how we could define a macro to
> generate the functions. DEFINE_SETUP_FN(char, ch) defines setup_ch()
> that takes a test function and a char that is passed to the test with
> the initialized strbuf. I'm not sure that's the way we want to go for
> this test file but I thought I'd post it in case it is useful for
> future tests.

I'm sympathetic with the idea of enhancing the weak type system of C to
avoid mistakes.  Hiding the use of void pointers behind typed wrappers
can be useful.  Not using void pointers is even better and clearer,
though, where possible.

Thought about using such wrappers around qsort(3) and friends long ago
as well as parse_options() more recently, but didn't arrive at mergeable
patches, yet.

Initialization and destruction seem easy and familiar enough to not
warrant the use of void pointers.  No need to get fancy just for basic
necessities IMHO.

René

Re: [PATCH 6/6] t-strbuf: use TEST_RUN

[Ghanshyam Thakkar]

Phillip Wood <phillip.wood123@gmail.com> wrote:
> On 02/07/2024 16:14, phillip.wood123@gmail.com wrote:
> > Getting rid of the untyped test arguments is 
> > definitely a benefit of this approach.
>
> That got me thinking how we might make type-safe setup()
> functions. The diff below shows how we could define a macro to
> generate the functions. DEFINE_SETUP_FN(char, ch) defines setup_ch()
> that takes a test function and a char that is passed to the test with
> the initialized strbuf. I'm not sure that's the way we want to go for
> this test file but I thought I'd post it in case it is useful for
> future tests.
>
> Best Wishes
>
> Phillip

This seems interesting if we want keep using setup() architecture. But a
bit too much if we have to keep doing this in every test where we need
setup(). Maybe some of it can be abstracted away in test-lib? Anyway, I was
a bit surprised that we didn't just use 'const char *' instead of 'const void *',
as we pass a string to all of them. So, I don't see the value of using 'void *'
in the first place in this test.

Thanks.

> ---- >8 ----
> diff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c
> index 6027dafef70..8fc9a8b38df 100644
> --- a/t/unit-tests/t-strbuf.c
> +++ b/t/unit-tests/t-strbuf.c
> @@ -1,27 +1,60 @@
> #include "test-lib.h"
> #include "strbuf.h"
>   
> -/* wrapper that supplies tests with an empty, initialized strbuf */
> -static void setup(void (*f)(struct strbuf*, const void*),
> - const void *data)
> -{
> - struct strbuf buf = STRBUF_INIT;
> -
> - f(&buf, data);
> - strbuf_release(&buf);
> - check_uint(buf.len, ==, 0);
> - check_uint(buf.alloc, ==, 0);
> -}
> -
> -/* wrapper that supplies tests with a populated, initialized strbuf */
> -static void setup_populated(void (*f)(struct strbuf*, const void*),
> - const char *init_str, const void *data)
> -{
> - struct strbuf buf = STRBUF_INIT;
> -
> - strbuf_addstr(&buf, init_str);
> - check_uint(buf.len, ==, strlen(init_str));
> - f(&buf, data);
> +/*
> + * Define a type safe wrapper function that supplies test functions
> + * with an initialized strbuf populated with an optional string and
> + * some data and then frees the strbuf when the test function
> + * returns. For example given the test function
> + *
> + * t_foo(struct strbuf *buf, struct foo *data)
> + *
> + * the type safe wrapper function
> + *
> + * setup_foo(void(*)(struct strbuf*, const struct foo*),
> + * const char *init_str, const struct foo*)
> + *
> + * can be defined with
> + *
> + * DEFINE_SETUP_FN(const struct foo*, foo)
> + *
> + * and used to run t_foo() with
> + *
> + * TEST(setup_foo(t_foo, "initial string", &my_foo), "test foo");
> + */
> +#define DEFINE_SETUP_FN(type, suffix) \
> + static void marshal_##suffix(void(*test_fn)(void), \
> + struct strbuf *buf, const void *ctx) \
> + { \
> + type data = *(type *)ctx; \
> + ((void(*)(struct strbuf*, type)) test_fn)(buf, data); \
> + } \
> + \
> + static void setup_##suffix(void(*test_fn)(struct strbuf*, type), \
> + const char *init_str, type data) \
> + { \
> + void *ctx = &data; \
> + do_setup(init_str, (void(*)(void)) test_fn, ctx, \
> + marshal_##suffix); \
> + }
> +
> +/*
> + * Helper function for DEFINE_SETUP_FN() that initializes the strbuf,
> + * calls the test function and releases the strbuf
> + */
> +static void do_setup(const char* init_str, void(*f)(void), const void
> *ctx,
> + void(*marshal)(void(*)(void), struct strbuf*, const void*))
> +{
> + struct strbuf buf = STRBUF_INIT;
> +
> + if (init_str) {
> + strbuf_addstr(&buf, init_str);
> + if (!check_uint(buf.len, ==, strlen(init_str))) {
> + strbuf_release(&buf);
> + return;
> + }
> + }
> + marshal(f, &buf, ctx);
> strbuf_release(&buf);
> check_uint(buf.len, ==, 0);
> check_uint(buf.alloc, ==, 0);
> @@ -66,10 +99,9 @@ static void t_dynamic_init(void)
> strbuf_release(&buf);
> }
>   
> -static void t_addch(struct strbuf *buf, const void *data)
> +DEFINE_SETUP_FN(char, ch)
> +static void t_addch(struct strbuf *buf, char ch)
> {
> - const char *p_ch = data;
> - const char ch = *p_ch;
> size_t orig_alloc = buf->alloc;
> size_t orig_len = buf->len;
>   
> @@ -85,9 +117,9 @@ static void t_addch(struct strbuf *buf, const void
> *data)
> check_char(buf->buf[buf->len], ==, '\0');
> }
>   
> -static void t_addstr(struct strbuf *buf, const void *data)
> +DEFINE_SETUP_FN(const char*, str)
> +static void t_addstr(struct strbuf *buf, const char *text)
> {
> - const char *text = data;
> size_t len = strlen(text);
> size_t orig_alloc = buf->alloc;
> size_t orig_len = buf->len;
> @@ -110,12 +142,12 @@ int cmd_main(int argc, const char **argv)
> if (!TEST(t_static_init(), "static initialization works"))
> test_skip_all("STRBUF_INIT is broken");
> TEST(t_dynamic_init(), "dynamic initialization works");
> - TEST(setup(t_addch, "a"), "strbuf_addch adds char");
> - TEST(setup(t_addch, ""), "strbuf_addch adds NUL char");
> - TEST(setup_populated(t_addch, "initial value", "a"),
> + TEST(setup_ch(t_addch, NULL, 'a'), "strbuf_addch adds char");
> + TEST(setup_ch(t_addch, NULL, '\0'), "strbuf_addch adds NUL char");
> + TEST(setup_ch(t_addch, "initial value", 'a'),
> "strbuf_addch appends to initial value");
> - TEST(setup(t_addstr, "hello there"), "strbuf_addstr adds string");
> - TEST(setup_populated(t_addstr, "initial value", "hello there"),
> + TEST(setup_str(t_addstr, NULL, "hello there"), "strbuf_addstr adds
> string");
> + TEST(setup_str(t_addstr, "initial value", "hello there"),
> "strbuf_addstr appends string to initial value");
>   
> return test_done();

[PATCH v2 0/6] unit-tests: add and use for_test to simplify tests

[René Scharfe]

Changes since v1:
- replace "if (TEST_RUN(...))" with "for_test (...)", a macro based on the
  keyword "for" that doesn't require changes to existing functions,
- clarify the commit messages of patches 3 and 5,
- convert the strbuf initialization tests as well.

  t0080: move expected output to a file
  unit-tests: add for_test
  t-ctype: use for_test
  t-reftable-basics: use for_test
  t-strvec: use for_test
  t-strbuf: use for_test

 .clang-format                    |   2 +
 t/helper/test-example-tap.c      |  33 +++
 t/t0080-unit-test-output.sh      |  48 +----
 t/t0080/expect                   |  76 +++++++
 t/unit-tests/t-ctype.c           |   4 +-
 t/unit-tests/t-reftable-basics.c | 228 +++++++++-----------
 t/unit-tests/t-strbuf.c          | 122 ++++++-----
 t/unit-tests/t-strvec.c          | 356 ++++++++++++++-----------------
 t/unit-tests/test-lib.h          |  19 ++
 9 files changed, 454 insertions(+), 434 deletions(-)
 create mode 100644 t/t0080/expect

Range-Diff gegen v1:
1:  6efe5a37f0 = 1:  5faabaea54 t0080: move expected output to a file
2:  8297c2b121 < -:  ---------- unit-tests: add TEST_RUN
-:  ---------- > 2:  d4f9fa0938 unit-tests: add for_test
3:  ec5599906d ! 3:  a7cd5a2a3a t-ctype: use TEST_RUN
    @@ Metadata
     Author: René Scharfe <l.s.r@web.de>

      ## Commit message ##
    -    t-ctype: use TEST_RUN
    +    t-ctype: use for_test

    -    Use the macro TEST_RUN instead of the internal functions
    -    test__run_begin() and test__run_end().
    +    Use the documented macro for_test instead of the internal functions
    +    test__run_begin() and test__run_end(), which are supposed to be private
    +    to the unit test framework.

      ## t/unit-tests/t-ctype.c ##
     @@
    @@ t/unit-tests/t-ctype.c
      		BUILD_ASSERT_OR_ZERO(sizeof(string[0]) == sizeof(char)); \
     -	int skip = test__run_begin(); \
     -	if (!skip) { \
    -+	if (TEST_RUN(#class " works")) { \
    ++	for_test (#class " works") { \
      		for (int i = 0; i < 256; i++) { \
      			if (!check_int(class(i), ==, !!memchr(string, i, len)))\
      				test_msg("      i: 0x%02x", i); \
4:  e589468f98 ! 4:  cc07910f88 t-reftable-basics: use TEST_RUN
    @@ Metadata
     Author: René Scharfe <l.s.r@web.de>

      ## Commit message ##
    -    t-reftable-basics: use TEST_RUN
    +    t-reftable-basics: use for_test

         The macro TEST takes a single expression.  If a test requires multiple
         statements then they need to be placed in a function that's called in
         the TEST expression.

         Remove the overhead of defining and calling single-use functions by
    -    using TEST_RUN instead.
    +    using for_test instead.

         Run the tests in the order of definition.  We can reorder them like that
         because they are independent.  Technically this changes the output, but
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -		struct integer_needle_lesseq_args args = {
     -			.haystack = haystack,
     -			.needle = testcases[i].needle,
    -+	if (TEST_RUN("binary search with binsearch works")) {
    ++	for_test ("binary search with binsearch works") {
     +		int haystack[] = { 2, 4, 6, 8, 10 };
     +		struct {
     +			int needle;
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -	const char *a[] = { "a", "b", "c", NULL };
     -	const char *b[] = { "a", "b", "d", NULL };
     -	const char *c[] = { "a", "b", NULL };
    -+	if (TEST_RUN("names_length retuns size of a NULL-terminated string array")) {
    ++	for_test ("names_length retuns size of a NULL-terminated string array") {
     +		const char *a[] = { "a", "b", NULL };
     +		check_int(names_length(a), ==, 2);
     +	}
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -	check(!names_equal(a, b));
     -	check(!names_equal(a, c));
     -}
    -+	if (TEST_RUN("names_equal compares NULL-terminated string arrays")) {
    ++	for_test ("names_equal compares NULL-terminated string arrays") {
     +		const char *a[] = { "a", "b", "c", NULL };
     +		const char *b[] = { "a", "b", "d", NULL };
     +		const char *c[] = { "a", "b", NULL };
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -	check(!out[2]);
     -	free_names(out);
     -}
    -+	if (TEST_RUN("parse_names works for basic input")) {
    ++	for_test ("parse_names works for basic input") {
     +		char in1[] = "line\n";
     +		char in2[] = "a\nb\nc";
     +		char **out = NULL;
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -		check_int(common_prefix_size(&a, &b), ==, cases[i].want);
     -		strbuf_reset(&a);
     -		strbuf_reset(&b);
    -+	if (TEST_RUN("parse_names drops empty string")) {
    ++	for_test ("parse_names drops empty string") {
     +		char in[] = "a\n\nb\n";
     +		char **out = NULL;
     +		parse_names(in, strlen(in), &out);
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -	out = get_be24(dest);
     -	check_int(in, ==, out);
     -}
    -+	if (TEST_RUN("common_prefix_size works")) {
    ++	for_test ("common_prefix_size works") {
     +		struct strbuf a = STRBUF_INIT;
     +		struct strbuf b = STRBUF_INIT;
     +		struct {
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -	TEST(test_names_equal(), "names_equal compares NULL-terminated string arrays");
     -	TEST(test_u24_roundtrip(), "put_be24 and get_be24 work");
     -	TEST(test_u16_roundtrip(), "put_be16 and get_be16 work");
    -+	if (TEST_RUN("put_be24 and get_be24 work")) {
    ++	for_test ("put_be24 and get_be24 work") {
     +		uint32_t in = 0x112233;
     +		uint8_t dest[3];
     +		uint32_t out;
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     +		check_int(in, ==, out);
     +	}
     +
    -+	if (TEST_RUN("put_be16 and get_be16 work")) {
    ++	for_test ("put_be16 and get_be16 work") {
     +		uint32_t in = 0xfef1;
     +		uint8_t dest[3];
     +		uint32_t out;
5:  5805a9cbd7 ! 5:  11c1675a13 t-strvec: use TEST_RUN
    @@ Metadata
     Author: René Scharfe <l.s.r@web.de>

      ## Commit message ##
    -    t-strvec: use TEST_RUN
    +    t-strvec: use for_test

         The macro TEST takes a single expression.  If a test requires multiple
         statements then they need to be placed in a function that's called in
         the TEST expression.

    -    Remove the overhead of defining and calling single-use functions by
    -    using TEST_RUN instead.
    +    Remove the cognitive overhead of defining and calling single-use
    +    functions by using for_test instead.

      ## t/unit-tests/t-strvec.c ##
     @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct strvec *vec, ...)
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_uint(vec.nr, ==, 0);
     -	check_uint(vec.alloc, ==, 0);
     -}
    -+	if (TEST_RUN("static initialization")) {
    ++	for_test ("static initialization") {
     +		struct strvec vec = STRVEC_INIT;
     +		check_pointer_eq(vec.v, empty_strvec);
     +		check_uint(vec.nr, ==, 0);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_uint(vec.nr, ==, 0);
     -	check_uint(vec.alloc, ==, 0);
     -}
    -+	if (TEST_RUN("dynamic initialization")) {
    ++	for_test ("dynamic initialization") {
     +		struct strvec vec;
     +		strvec_init(&vec);
     +		check_pointer_eq(vec.v, empty_strvec);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_uint(vec.nr, ==, 0);
     -	check_uint(vec.alloc, ==, 0);
     -}
    -+	if (TEST_RUN("clear")) {
    ++	for_test ("clear") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_push(&vec, "foo");
     +		strvec_clear(&vec);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -static void t_push(void)
     -{
     -	struct strvec vec = STRVEC_INIT;
    -+	if (TEST_RUN("push")) {
    ++	for_test ("push") {
     +		struct strvec vec = STRVEC_INIT;

     -	strvec_push(&vec, "foo");
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo: 1", NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("pushf")) {
    ++	for_test ("pushf") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushf(&vec, "foo: %d", 1);
     +		check_strvec(&vec, "foo: 1", NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "bar", "baz", NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("pushl")) {
    ++	for_test ("pushl") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		check_strvec(&vec, "foo", "bar", "baz", NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -		"foo", "bar", "baz", NULL,
     -	};
     -	struct strvec vec = STRVEC_INIT;
    -+	if (TEST_RUN("pushv")) {
    ++	for_test ("pushv") {
     +		const char *strings[] = {
     +			"foo", "bar", "baz", NULL,
     +		};
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "replaced", "bar", "baz", NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("replace at head")) {
    ++	for_test ("replace at head") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_replace(&vec, 0, "replaced");
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "bar", "replaced", NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("replace at tail")) {
    ++	for_test ("replace at tail") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_replace(&vec, 2, "replaced");
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "replaced", "baz", NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("replace in between")) {
    ++	for_test ("replace in between") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_replace(&vec, 1, "replaced");
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "oo", NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("replace with substring")) {
    ++	for_test ("replace with substring") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", NULL);
     +		strvec_replace(&vec, 0, vec.v[0] + 1);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "bar", "baz", NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("remove at head")) {
    ++	for_test ("remove at head") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_remove(&vec, 0);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "bar", NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("remove at tail")) {
    ++	for_test ("remove at tail") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_remove(&vec, 2);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "baz", NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("remove in between")) {
    ++	for_test ("remove in between") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_remove(&vec, 1);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("pop with empty array")) {
    ++	for_test ("pop with empty array") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pop(&vec);
     +		check_strvec(&vec, NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "bar", NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("pop with non-empty array")) {
    ++	for_test ("pop with non-empty array") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_pop(&vec);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("split empty string")) {
    ++	for_test ("split empty string") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_split(&vec, "");
     +		check_strvec(&vec, NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("split single item")) {
    ++	for_test ("split single item") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_split(&vec, "foo");
     +		check_strvec(&vec, "foo", NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "bar", "baz", NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("split multiple items")) {
    ++	for_test ("split multiple items") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_split(&vec, "foo bar baz");
     +		check_strvec(&vec, "foo", "bar", "baz", NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("split whitespace only")) {
    ++	for_test ("split whitespace only") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_split(&vec, " \t\n");
     +		check_strvec(&vec, NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "bar", NULL);
     -	strvec_clear(&vec);
     -}
    -+	if (TEST_RUN("split multiple consecutive whitespaces")) {
    ++	for_test ("split multiple consecutive whitespaces") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_split(&vec, "foo\n\t bar");
     +		check_strvec(&vec, "foo", "bar", NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -{
     -	struct strvec vec = STRVEC_INIT;
     -	const char **detached;
    -+	if (TEST_RUN("detach")) {
    ++	for_test ("detach") {
     +		struct strvec vec = STRVEC_INIT;
     +		const char **detached;

6:  188b31884b ! 6:  cd79132f95 t-strbuf: use TEST_RUN
    @@ Metadata
     Author: René Scharfe <l.s.r@web.de>

      ## Commit message ##
    -    t-strbuf: use TEST_RUN
    +    t-strbuf: use for_test

         The macro TEST takes a single expression.  If a test requires multiple
         statements then they need to be placed in a function that's called in
    @@ Commit message
         are used for that purpose and take another function as an argument,
         making the control flow hard to follow.

    -    Remove the overhead of these functions by using TEST_RUN instead.  Move
    +    Remove the overhead of these functions by using for_test instead.  Move
         their duplicate post-condition checks into a new helper, t_release(),
         and let t_addch() and t_addstr() accept properly typed input parameters
         instead of void pointers.
    @@ t/unit-tests/t-strbuf.c
      static int assert_sane_strbuf(struct strbuf *buf)
      {
      	/* Initialized strbufs should always have a non-NULL buffer */
    -@@ t/unit-tests/t-strbuf.c: static void t_dynamic_init(void)
    - 	strbuf_release(&buf);
    +@@ t/unit-tests/t-strbuf.c: static int assert_sane_strbuf(struct strbuf *buf)
    + 	return check_uint(buf->len, <, buf->alloc);
      }

    --static void t_addch(struct strbuf *buf, const void *data)
    +-static void t_static_init(void)
     +static void t_addch(struct strbuf *buf, int ch)
      {
    +-	struct strbuf buf = STRBUF_INIT;
    +-
    +-	check_uint(buf.len, ==, 0);
    +-	check_uint(buf.alloc, ==, 0);
    +-	check_char(buf.buf[0], ==, '\0');
    +-}
    +-
    +-static void t_dynamic_init(void)
    +-{
    +-	struct strbuf buf;
    +-
    +-	strbuf_init(&buf, 1024);
    +-	check(assert_sane_strbuf(&buf));
    +-	check_uint(buf.len, ==, 0);
    +-	check_uint(buf.alloc, >=, 1024);
    +-	check_char(buf.buf[0], ==, '\0');
    +-	strbuf_release(&buf);
    +-}
    +-
    +-static void t_addch(struct strbuf *buf, const void *data)
    +-{
     -	const char *p_ch = data;
     -	const char ch = *p_ch;
      	size_t orig_alloc = buf->alloc;
    @@ t/unit-tests/t-strbuf.c: static void t_addstr(struct strbuf *buf, const void *da
     +
      int cmd_main(int argc, const char **argv)
      {
    - 	if (!TEST(t_static_init(), "static initialization works"))
    - 		test_skip_all("STRBUF_INIT is broken");
    - 	TEST(t_dynamic_init(), "dynamic initialization works");
    +-	if (!TEST(t_static_init(), "static initialization works"))
    +-		test_skip_all("STRBUF_INIT is broken");
    +-	TEST(t_dynamic_init(), "dynamic initialization works");
     -	TEST(setup(t_addch, "a"), "strbuf_addch adds char");
     -	TEST(setup(t_addch, ""), "strbuf_addch adds NUL char");
     -	TEST(setup_populated(t_addch, "initial value", "a"),
    @@ t/unit-tests/t-strbuf.c: static void t_addstr(struct strbuf *buf, const void *da
     -	TEST(setup(t_addstr, "hello there"), "strbuf_addstr adds string");
     -	TEST(setup_populated(t_addstr, "initial value", "hello there"),
     -	     "strbuf_addstr appends string to initial value");
    ++	for_test ("static initialization works") {
    ++		struct strbuf buf = STRBUF_INIT;
    ++
    ++		if (!check_uint(buf.len, ==, 0) ||
    ++		    !check_uint(buf.alloc, ==, 0) ||
    ++		    !check_char(buf.buf[0], ==, '\0'))
    ++			test_skip_all("STRBUF_INIT is broken");
    ++	}
    ++
    ++	for_test ("dynamic initialization works") {
    ++		struct strbuf buf;
    ++
    ++		strbuf_init(&buf, 1024);
    ++		check(assert_sane_strbuf(&buf));
    ++		check_uint(buf.len, ==, 0);
    ++		check_uint(buf.alloc, >=, 1024);
    ++		check_char(buf.buf[0], ==, '\0');
    ++		strbuf_release(&buf);
    ++	}
     +
    -+	if (TEST_RUN("strbuf_addch adds char")) {
    ++	for_test ("strbuf_addch adds char") {
     +		struct strbuf sb = STRBUF_INIT;
     +		t_addch(&sb, 'a');
     +		t_release(&sb);
     +	}
     +
    -+	if (TEST_RUN("strbuf_addch adds NUL char")) {
    ++	for_test ("strbuf_addch adds NUL char") {
     +		struct strbuf sb = STRBUF_INIT;
     +		t_addch(&sb, '\0');
     +		t_release(&sb);
     +	}
     +
    -+	if (TEST_RUN("strbuf_addch appends to initial value")) {
    ++	for_test ("strbuf_addch appends to initial value") {
     +		struct strbuf sb = STRBUF_INIT;
     +		t_addstr(&sb, "initial value");
     +		t_addch(&sb, 'a');
     +		t_release(&sb);
     +	}
     +
    -+	if (TEST_RUN("strbuf_addstr adds string")) {
    ++	for_test ("strbuf_addstr adds string") {
     +		struct strbuf sb = STRBUF_INIT;
     +		t_addstr(&sb, "hello there");
     +		t_release(&sb);
     +	}
     +
    -+	if (TEST_RUN("strbuf_addstr appends string to initial value")) {
    ++	for_test ("strbuf_addstr appends string to initial value") {
     +		struct strbuf sb = STRBUF_INIT;
     +		t_addstr(&sb, "initial value");
     +		t_addstr(&sb, "hello there");
--
2.45.2

[PATCH v2 1/6] t0080: move expected output to a file

[René Scharfe]

Provide the expected output of "test-tool example-tap" verbatim instead
of as a here-doc, to avoid distractions due to quoting, variables
containing quotes and indentation.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/t0080-unit-test-output.sh | 48 +------------------------------------
 t/t0080/expect              | 43 +++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+), 47 deletions(-)
 create mode 100644 t/t0080/expect

diff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh
index 7bbb065d58..93f2defa19 100755
--- a/t/t0080-unit-test-output.sh
+++ b/t/t0080-unit-test-output.sh
@@ -6,54 +6,8 @@ TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh

 test_expect_success 'TAP output from unit tests' '
-	cat >expect <<-EOF &&
-	ok 1 - passing test
-	ok 2 - passing test and assertion return 1
-	# check "1 == 2" failed at t/helper/test-example-tap.c:77
-	#    left: 1
-	#   right: 2
-	not ok 3 - failing test
-	ok 4 - failing test and assertion return 0
-	not ok 5 - passing TEST_TODO() # TODO
-	ok 6 - passing TEST_TODO() returns 1
-	# todo check ${SQ}check(x)${SQ} succeeded at t/helper/test-example-tap.c:26
-	not ok 7 - failing TEST_TODO()
-	ok 8 - failing TEST_TODO() returns 0
-	# check "0" failed at t/helper/test-example-tap.c:31
-	# skipping test - missing prerequisite
-	# skipping check ${SQ}1${SQ} at t/helper/test-example-tap.c:33
-	ok 9 - test_skip() # SKIP
-	ok 10 - skipped test returns 1
-	# skipping test - missing prerequisite
-	ok 11 - test_skip() inside TEST_TODO() # SKIP
-	ok 12 - test_skip() inside TEST_TODO() returns 1
-	# check "0" failed at t/helper/test-example-tap.c:49
-	not ok 13 - TEST_TODO() after failing check
-	ok 14 - TEST_TODO() after failing check returns 0
-	# check "0" failed at t/helper/test-example-tap.c:57
-	not ok 15 - failing check after TEST_TODO()
-	ok 16 - failing check after TEST_TODO() returns 0
-	# check "!strcmp("\thello\\\\", "there\"\n")" failed at t/helper/test-example-tap.c:62
-	#    left: "\011hello\\\\"
-	#   right: "there\"\012"
-	# check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:63
-	#    left: "NULL"
-	#   right: NULL
-	# check "${SQ}a${SQ} == ${SQ}\n${SQ}" failed at t/helper/test-example-tap.c:64
-	#    left: ${SQ}a${SQ}
-	#   right: ${SQ}\012${SQ}
-	# check "${SQ}\\\\${SQ} == ${SQ}\\${SQ}${SQ}" failed at t/helper/test-example-tap.c:65
-	#    left: ${SQ}\\\\${SQ}
-	#   right: ${SQ}\\${SQ}${SQ}
-	not ok 17 - messages from failing string and char comparison
-	# BUG: test has no checks at t/helper/test-example-tap.c:92
-	not ok 18 - test with no checks
-	ok 19 - test with no checks returns 0
-	1..19
-	EOF
-
 	! test-tool example-tap >actual &&
-	test_cmp expect actual
+	test_cmp "$TEST_DIRECTORY"/t0080/expect actual
 '

 test_done
diff --git a/t/t0080/expect b/t/t0080/expect
new file mode 100644
index 0000000000..0cfa0dc6d8
--- /dev/null
+++ b/t/t0080/expect
@@ -0,0 +1,43 @@
+ok 1 - passing test
+ok 2 - passing test and assertion return 1
+# check "1 == 2" failed at t/helper/test-example-tap.c:77
+#    left: 1
+#   right: 2
+not ok 3 - failing test
+ok 4 - failing test and assertion return 0
+not ok 5 - passing TEST_TODO() # TODO
+ok 6 - passing TEST_TODO() returns 1
+# todo check 'check(x)' succeeded at t/helper/test-example-tap.c:26
+not ok 7 - failing TEST_TODO()
+ok 8 - failing TEST_TODO() returns 0
+# check "0" failed at t/helper/test-example-tap.c:31
+# skipping test - missing prerequisite
+# skipping check '1' at t/helper/test-example-tap.c:33
+ok 9 - test_skip() # SKIP
+ok 10 - skipped test returns 1
+# skipping test - missing prerequisite
+ok 11 - test_skip() inside TEST_TODO() # SKIP
+ok 12 - test_skip() inside TEST_TODO() returns 1
+# check "0" failed at t/helper/test-example-tap.c:49
+not ok 13 - TEST_TODO() after failing check
+ok 14 - TEST_TODO() after failing check returns 0
+# check "0" failed at t/helper/test-example-tap.c:57
+not ok 15 - failing check after TEST_TODO()
+ok 16 - failing check after TEST_TODO() returns 0
+# check "!strcmp("\thello\\", "there\"\n")" failed at t/helper/test-example-tap.c:62
+#    left: "\011hello\\"
+#   right: "there\"\012"
+# check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:63
+#    left: "NULL"
+#   right: NULL
+# check "'a' == '\n'" failed at t/helper/test-example-tap.c:64
+#    left: 'a'
+#   right: '\012'
+# check "'\\' == '\''" failed at t/helper/test-example-tap.c:65
+#    left: '\\'
+#   right: '\''
+not ok 17 - messages from failing string and char comparison
+# BUG: test has no checks at t/helper/test-example-tap.c:92
+not ok 18 - test with no checks
+ok 19 - test with no checks returns 0
+1..19
--
2.45.2

[PATCH v2 2/6] unit-tests: add for_test

[René Scharfe]

The macro TEST only allows defining a test that consists of a single
expression.  Add a new macro, for_test, which provides a way to define
unit tests that are made up of one or more statements.

for_test allows defining self-contained tests en bloc, a bit like
test_expect_success does for regular tests.  It acts like a for loop
that runs at most once; the test body is executed if test_skip_all()
had not been called before.

Helped-by: Phillip Wood <phillip.wood123@gmail.com>
Signed-off-by: René Scharfe <l.s.r@web.de>
---
 .clang-format               |  2 ++
 t/helper/test-example-tap.c | 33 +++++++++++++++++++++++++++++++++
 t/t0080/expect              | 35 ++++++++++++++++++++++++++++++++++-
 t/unit-tests/test-lib.h     | 19 +++++++++++++++++++
 4 files changed, 88 insertions(+), 1 deletion(-)

diff --git a/.clang-format b/.clang-format
index 6408251577..863dc87dfc 100644
--- a/.clang-format
+++ b/.clang-format
@@ -151,6 +151,7 @@ Cpp11BracedListStyle: false
 # function calls. Taken from:
 #   git grep -h '^#define [^[:space:]]*for_\?each[^[:space:]]*(' |
 #   sed "s/^#define /  - '/; s/(.*$/'/" | sort | uniq
+# Added for_test from t/unit-tests/test-lib.h manually as a special case.
 ForEachMacros:
   - 'for_each_builtin'
   - 'for_each_string_list_item'
@@ -168,6 +169,7 @@ ForEachMacros:
   - 'strintmap_for_each_entry'
   - 'strmap_for_each_entry'
   - 'strset_for_each_entry'
+  - 'for_test'

 # The maximum number of consecutive empty lines to keep.
 MaxEmptyLinesToKeep: 1
diff --git a/t/helper/test-example-tap.c b/t/helper/test-example-tap.c
index d072ad559f..51d5e6e75b 100644
--- a/t/helper/test-example-tap.c
+++ b/t/helper/test-example-tap.c
@@ -92,5 +92,38 @@ int cmd__example_tap(int argc, const char **argv)
 	test_res = TEST(t_empty(), "test with no checks");
 	TEST(check_int(test_res, ==, 0), "test with no checks returns 0");

+	for_test ("for_test passing test")
+		check_int(1, ==, 1);
+	for_test ("for_test failing test")
+		check_int(1, ==, 2);
+	for_test ("for_test passing TEST_TODO()")
+		TEST_TODO(check(0));
+	for_test ("for_test failing TEST_TODO()")
+		TEST_TODO(check(1));
+	for_test ("for_test test_skip()") {
+		check(0);
+		test_skip("missing prerequisite");
+		check(1);
+	}
+	for_test ("for_test test_skip() inside TEST_TODO()")
+		TEST_TODO((test_skip("missing prerequisite"), 1));
+	for_test ("for_test TEST_TODO() after failing check") {
+		check(0);
+		TEST_TODO(check(0));
+	}
+	for_test ("for_test failing check after TEST_TODO()") {
+		check(1);
+		TEST_TODO(check(0));
+		check(0);
+	}
+	for_test ("for_test messages from failing string and char comparison") {
+		check_str("\thello\\", "there\"\n");
+		check_str("NULL", NULL);
+		check_char('a', ==, '\n');
+		check_char('\\', ==, '\'');
+	}
+	for_test ("for_test test with no checks")
+		; /* nothing */
+
 	return test_done();
 }
diff --git a/t/t0080/expect b/t/t0080/expect
index 0cfa0dc6d8..583f41b8c9 100644
--- a/t/t0080/expect
+++ b/t/t0080/expect
@@ -40,4 +40,37 @@ not ok 17 - messages from failing string and char comparison
 # BUG: test has no checks at t/helper/test-example-tap.c:92
 not ok 18 - test with no checks
 ok 19 - test with no checks returns 0
-1..19
+ok 20 - for_test passing test
+# check "1 == 2" failed at t/helper/test-example-tap.c:98
+#    left: 1
+#   right: 2
+not ok 21 - for_test failing test
+not ok 22 - for_test passing TEST_TODO() # TODO
+# todo check 'check(1)' succeeded at t/helper/test-example-tap.c:102
+not ok 23 - for_test failing TEST_TODO()
+# check "0" failed at t/helper/test-example-tap.c:104
+# skipping test - missing prerequisite
+# skipping check '1' at t/helper/test-example-tap.c:106
+ok 24 - for_test test_skip() # SKIP
+# skipping test - missing prerequisite
+ok 25 - for_test test_skip() inside TEST_TODO() # SKIP
+# check "0" failed at t/helper/test-example-tap.c:111
+not ok 26 - for_test TEST_TODO() after failing check
+# check "0" failed at t/helper/test-example-tap.c:117
+not ok 27 - for_test failing check after TEST_TODO()
+# check "!strcmp("\thello\\", "there\"\n")" failed at t/helper/test-example-tap.c:120
+#    left: "\011hello\\"
+#   right: "there\"\012"
+# check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:121
+#    left: "NULL"
+#   right: NULL
+# check "'a' == '\n'" failed at t/helper/test-example-tap.c:122
+#    left: 'a'
+#   right: '\012'
+# check "'\\' == '\''" failed at t/helper/test-example-tap.c:123
+#    left: '\\'
+#   right: '\''
+not ok 28 - for_test messages from failing string and char comparison
+# BUG: test has no checks at t/helper/test-example-tap.c:125
+not ok 29 - for_test test with no checks
+1..29
diff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h
index 2de6d715d5..12afd47ac9 100644
--- a/t/unit-tests/test-lib.h
+++ b/t/unit-tests/test-lib.h
@@ -14,6 +14,25 @@
 	test__run_end(test__run_begin() ? 0 : (t, 1),	\
 		      TEST_LOCATION(),  __VA_ARGS__)

+/*
+ * Run a test unless test_skip_all() has been called.  Acts like a for
+ * loop that runs at most once, with the test description between the
+ * parentheses and the test body as a statement or block after them.
+ * The description for each test should be unique.  E.g.:
+ *
+ *  for_test ("something else %d %d", arg1, arg2) {
+ *          prepare();
+ *          test_something_else(arg1, arg2);
+ *          cleanup();
+ *  }
+ */
+#define for_test(...)							\
+	for (int for_test_running_ = test__run_begin() ?		\
+		(test__run_end(0, TEST_LOCATION(), __VA_ARGS__), 0) : 1;\
+	     for_test_running_;						\
+	     test__run_end(1, TEST_LOCATION(), __VA_ARGS__),		\
+		for_test_running_ = 0)
+
 /*
  * Print a test plan, should be called before any tests. If the number
  * of tests is not known in advance test_done() will automatically
--
2.45.2

[PATCH v2 3/6] t-ctype: use for_test

[René Scharfe]

Use the documented macro for_test instead of the internal functions
test__run_begin() and test__run_end(), which are supposed to be private
to the unit test framework.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-ctype.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/t/unit-tests/t-ctype.c b/t/unit-tests/t-ctype.c
index d6ac1fe678..92a05f02b3 100644
--- a/t/unit-tests/t-ctype.c
+++ b/t/unit-tests/t-ctype.c
@@ -4,15 +4,13 @@
 	size_t len = ARRAY_SIZE(string) - 1 + \
 		BUILD_ASSERT_OR_ZERO(ARRAY_SIZE(string) > 0) + \
 		BUILD_ASSERT_OR_ZERO(sizeof(string[0]) == sizeof(char)); \
-	int skip = test__run_begin(); \
-	if (!skip) { \
+	for_test (#class " works") { \
 		for (int i = 0; i < 256; i++) { \
 			if (!check_int(class(i), ==, !!memchr(string, i, len)))\
 				test_msg("      i: 0x%02x", i); \
 		} \
 		check(!class(EOF)); \
 	} \
-	test__run_end(!skip, TEST_LOCATION(), #class " works"); \
 } while (0)

 #define DIGIT "0123456789"
--
2.45.2

[PATCH v2 4/6] t-reftable-basics: use for_test

[René Scharfe]

The macro TEST takes a single expression.  If a test requires multiple
statements then they need to be placed in a function that's called in
the TEST expression.

Remove the overhead of defining and calling single-use functions by
using for_test instead.

Run the tests in the order of definition.  We can reorder them like that
because they are independent.  Technically this changes the output, but
retains the meaning of a full run and allows for easier review e.g. with
diff option --ignore-all-space.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-reftable-basics.c | 228 ++++++++++++++-----------------
 1 file changed, 106 insertions(+), 122 deletions(-)

diff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c
index 4e80bdf16d..eb7af9ade4 100644
--- a/t/unit-tests/t-reftable-basics.c
+++ b/t/unit-tests/t-reftable-basics.c
@@ -20,141 +20,125 @@ static int integer_needle_lesseq(size_t i, void *_args)
 	return args->needle <= args->haystack[i];
 }

-static void test_binsearch(void)
+int cmd_main(int argc, const char *argv[])
 {
-	int haystack[] = { 2, 4, 6, 8, 10 };
-	struct {
-		int needle;
-		size_t expected_idx;
-	} testcases[] = {
-		{-9000, 0},
-		{-1, 0},
-		{0, 0},
-		{2, 0},
-		{3, 1},
-		{4, 1},
-		{7, 3},
-		{9, 4},
-		{10, 4},
-		{11, 5},
-		{9000, 5},
-	};
-
-	for (size_t i = 0; i < ARRAY_SIZE(testcases); i++) {
-		struct integer_needle_lesseq_args args = {
-			.haystack = haystack,
-			.needle = testcases[i].needle,
+	for_test ("binary search with binsearch works") {
+		int haystack[] = { 2, 4, 6, 8, 10 };
+		struct {
+			int needle;
+			size_t expected_idx;
+		} testcases[] = {
+			{-9000, 0},
+			{-1, 0},
+			{0, 0},
+			{2, 0},
+			{3, 1},
+			{4, 1},
+			{7, 3},
+			{9, 4},
+			{10, 4},
+			{11, 5},
+			{9000, 5},
 		};
-		size_t idx;

-		idx = binsearch(ARRAY_SIZE(haystack), &integer_needle_lesseq, &args);
-		check_int(idx, ==, testcases[i].expected_idx);
+		for (size_t i = 0; i < ARRAY_SIZE(testcases); i++) {
+			struct integer_needle_lesseq_args args = {
+				.haystack = haystack,
+				.needle = testcases[i].needle,
+			};
+			size_t idx;
+
+			idx = binsearch(ARRAY_SIZE(haystack),
+					&integer_needle_lesseq, &args);
+			check_int(idx, ==, testcases[i].expected_idx);
+		}
 	}
-}

-static void test_names_length(void)
-{
-	const char *a[] = { "a", "b", NULL };
-	check_int(names_length(a), ==, 2);
-}
-
-static void test_names_equal(void)
-{
-	const char *a[] = { "a", "b", "c", NULL };
-	const char *b[] = { "a", "b", "d", NULL };
-	const char *c[] = { "a", "b", NULL };
+	for_test ("names_length retuns size of a NULL-terminated string array") {
+		const char *a[] = { "a", "b", NULL };
+		check_int(names_length(a), ==, 2);
+	}

-	check(names_equal(a, a));
-	check(!names_equal(a, b));
-	check(!names_equal(a, c));
-}
+	for_test ("names_equal compares NULL-terminated string arrays") {
+		const char *a[] = { "a", "b", "c", NULL };
+		const char *b[] = { "a", "b", "d", NULL };
+		const char *c[] = { "a", "b", NULL };

-static void test_parse_names_normal(void)
-{
-	char in1[] = "line\n";
-	char in2[] = "a\nb\nc";
-	char **out = NULL;
-	parse_names(in1, strlen(in1), &out);
-	check_str(out[0], "line");
-	check(!out[1]);
-	free_names(out);
-
-	parse_names(in2, strlen(in2), &out);
-	check_str(out[0], "a");
-	check_str(out[1], "b");
-	check_str(out[2], "c");
-	check(!out[3]);
-	free_names(out);
-}
+		check(names_equal(a, a));
+		check(!names_equal(a, b));
+		check(!names_equal(a, c));
+	}

-static void test_parse_names_drop_empty(void)
-{
-	char in[] = "a\n\nb\n";
-	char **out = NULL;
-	parse_names(in, strlen(in), &out);
-	check_str(out[0], "a");
-	/* simply '\n' should be dropped as empty string */
-	check_str(out[1], "b");
-	check(!out[2]);
-	free_names(out);
-}
+	for_test ("parse_names works for basic input") {
+		char in1[] = "line\n";
+		char in2[] = "a\nb\nc";
+		char **out = NULL;
+		parse_names(in1, strlen(in1), &out);
+		check_str(out[0], "line");
+		check(!out[1]);
+		free_names(out);
+
+		parse_names(in2, strlen(in2), &out);
+		check_str(out[0], "a");
+		check_str(out[1], "b");
+		check_str(out[2], "c");
+		check(!out[3]);
+		free_names(out);
+	}

-static void test_common_prefix(void)
-{
-	struct strbuf a = STRBUF_INIT;
-	struct strbuf b = STRBUF_INIT;
-	struct {
-		const char *a, *b;
-		int want;
-	} cases[] = {
-		{"abcdef", "abc", 3},
-		{ "abc", "ab", 2 },
-		{ "", "abc", 0 },
-		{ "abc", "abd", 2 },
-		{ "abc", "pqr", 0 },
-	};
-
-	for (size_t i = 0; i < ARRAY_SIZE(cases); i++) {
-		strbuf_addstr(&a, cases[i].a);
-		strbuf_addstr(&b, cases[i].b);
-		check_int(common_prefix_size(&a, &b), ==, cases[i].want);
-		strbuf_reset(&a);
-		strbuf_reset(&b);
+	for_test ("parse_names drops empty string") {
+		char in[] = "a\n\nb\n";
+		char **out = NULL;
+		parse_names(in, strlen(in), &out);
+		check_str(out[0], "a");
+		/* simply '\n' should be dropped as empty string */
+		check_str(out[1], "b");
+		check(!out[2]);
+		free_names(out);
 	}
-	strbuf_release(&a);
-	strbuf_release(&b);
-}

-static void test_u24_roundtrip(void)
-{
-	uint32_t in = 0x112233;
-	uint8_t dest[3];
-	uint32_t out;
-	put_be24(dest, in);
-	out = get_be24(dest);
-	check_int(in, ==, out);
-}
+	for_test ("common_prefix_size works") {
+		struct strbuf a = STRBUF_INIT;
+		struct strbuf b = STRBUF_INIT;
+		struct {
+			const char *a, *b;
+			int want;
+		} cases[] = {
+			{"abcdef", "abc", 3},
+			{ "abc", "ab", 2 },
+			{ "", "abc", 0 },
+			{ "abc", "abd", 2 },
+			{ "abc", "pqr", 0 },
+		};

-static void test_u16_roundtrip(void)
-{
-	uint32_t in = 0xfef1;
-	uint8_t dest[3];
-	uint32_t out;
-	put_be16(dest, in);
-	out = get_be16(dest);
-	check_int(in, ==, out);
-}
+		for (size_t i = 0; i < ARRAY_SIZE(cases); i++) {
+			strbuf_addstr(&a, cases[i].a);
+			strbuf_addstr(&b, cases[i].b);
+			check_int(common_prefix_size(&a, &b), ==, cases[i].want);
+			strbuf_reset(&a);
+			strbuf_reset(&b);
+		}
+		strbuf_release(&a);
+		strbuf_release(&b);
+	}

-int cmd_main(int argc, const char *argv[])
-{
-	TEST(test_common_prefix(), "common_prefix_size works");
-	TEST(test_parse_names_normal(), "parse_names works for basic input");
-	TEST(test_parse_names_drop_empty(), "parse_names drops empty string");
-	TEST(test_binsearch(), "binary search with binsearch works");
-	TEST(test_names_length(), "names_length retuns size of a NULL-terminated string array");
-	TEST(test_names_equal(), "names_equal compares NULL-terminated string arrays");
-	TEST(test_u24_roundtrip(), "put_be24 and get_be24 work");
-	TEST(test_u16_roundtrip(), "put_be16 and get_be16 work");
+	for_test ("put_be24 and get_be24 work") {
+		uint32_t in = 0x112233;
+		uint8_t dest[3];
+		uint32_t out;
+		put_be24(dest, in);
+		out = get_be24(dest);
+		check_int(in, ==, out);
+	}
+
+	for_test ("put_be16 and get_be16 work") {
+		uint32_t in = 0xfef1;
+		uint8_t dest[3];
+		uint32_t out;
+		put_be16(dest, in);
+		out = get_be16(dest);
+		check_int(in, ==, out);
+	}

 	return test_done();
 }
--
2.45.2

[PATCH v2 5/6] t-strvec: use for_test

[René Scharfe]

The macro TEST takes a single expression.  If a test requires multiple
statements then they need to be placed in a function that's called in
the TEST expression.

Remove the cognitive overhead of defining and calling single-use
functions by using for_test instead.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-strvec.c | 356 ++++++++++++++++++----------------------
 1 file changed, 156 insertions(+), 200 deletions(-)

diff --git a/t/unit-tests/t-strvec.c b/t/unit-tests/t-strvec.c
index d4615ab06d..4fa2c21afb 100644
--- a/t/unit-tests/t-strvec.c
+++ b/t/unit-tests/t-strvec.c
@@ -36,237 +36,193 @@ static void check_strvec_loc(const char *loc, struct strvec *vec, ...)
 	check_pointer_eq(vec->v[nr], NULL);
 }

-static void t_static_init(void)
+int cmd_main(int argc, const char **argv)
 {
-	struct strvec vec = STRVEC_INIT;
-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
-}
+	for_test ("static initialization") {
+		struct strvec vec = STRVEC_INIT;
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);
+	}

-static void t_dynamic_init(void)
-{
-	struct strvec vec;
-	strvec_init(&vec);
-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
-}
+	for_test ("dynamic initialization") {
+		struct strvec vec;
+		strvec_init(&vec);
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);
+	}

-static void t_clear(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_push(&vec, "foo");
-	strvec_clear(&vec);
-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
-}
+	for_test ("clear") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_push(&vec, "foo");
+		strvec_clear(&vec);
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);
+	}

-static void t_push(void)
-{
-	struct strvec vec = STRVEC_INIT;
+	for_test ("push") {
+		struct strvec vec = STRVEC_INIT;

-	strvec_push(&vec, "foo");
-	check_strvec(&vec, "foo", NULL);
+		strvec_push(&vec, "foo");
+		check_strvec(&vec, "foo", NULL);

-	strvec_push(&vec, "bar");
-	check_strvec(&vec, "foo", "bar", NULL);
+		strvec_push(&vec, "bar");
+		check_strvec(&vec, "foo", "bar", NULL);

-	strvec_clear(&vec);
-}
+		strvec_clear(&vec);
+	}

-static void t_pushf(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushf(&vec, "foo: %d", 1);
-	check_strvec(&vec, "foo: 1", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("pushf") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushf(&vec, "foo: %d", 1);
+		check_strvec(&vec, "foo: 1", NULL);
+		strvec_clear(&vec);
+	}

-static void t_pushl(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	check_strvec(&vec, "foo", "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("pushl") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		check_strvec(&vec, "foo", "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_pushv(void)
-{
-	const char *strings[] = {
-		"foo", "bar", "baz", NULL,
-	};
-	struct strvec vec = STRVEC_INIT;
+	for_test ("pushv") {
+		const char *strings[] = {
+			"foo", "bar", "baz", NULL,
+		};
+		struct strvec vec = STRVEC_INIT;

-	strvec_pushv(&vec, strings);
-	check_strvec(&vec, "foo", "bar", "baz", NULL);
+		strvec_pushv(&vec, strings);
+		check_strvec(&vec, "foo", "bar", "baz", NULL);

-	strvec_clear(&vec);
-}
+		strvec_clear(&vec);
+	}

-static void t_replace_at_head(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_replace(&vec, 0, "replaced");
-	check_strvec(&vec, "replaced", "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("replace at head") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_replace(&vec, 0, "replaced");
+		check_strvec(&vec, "replaced", "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_replace_at_tail(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_replace(&vec, 2, "replaced");
-	check_strvec(&vec, "foo", "bar", "replaced", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("replace at tail") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_replace(&vec, 2, "replaced");
+		check_strvec(&vec, "foo", "bar", "replaced", NULL);
+		strvec_clear(&vec);
+	}

-static void t_replace_in_between(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_replace(&vec, 1, "replaced");
-	check_strvec(&vec, "foo", "replaced", "baz", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("replace in between") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_replace(&vec, 1, "replaced");
+		check_strvec(&vec, "foo", "replaced", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_replace_with_substring(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", NULL);
-	strvec_replace(&vec, 0, vec.v[0] + 1);
-	check_strvec(&vec, "oo", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("replace with substring") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", NULL);
+		strvec_replace(&vec, 0, vec.v[0] + 1);
+		check_strvec(&vec, "oo", NULL);
+		strvec_clear(&vec);
+	}

-static void t_remove_at_head(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_remove(&vec, 0);
-	check_strvec(&vec, "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("remove at head") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_remove(&vec, 0);
+		check_strvec(&vec, "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_remove_at_tail(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_remove(&vec, 2);
-	check_strvec(&vec, "foo", "bar", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("remove at tail") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_remove(&vec, 2);
+		check_strvec(&vec, "foo", "bar", NULL);
+		strvec_clear(&vec);
+	}

-static void t_remove_in_between(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_remove(&vec, 1);
-	check_strvec(&vec, "foo", "baz", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("remove in between") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_remove(&vec, 1);
+		check_strvec(&vec, "foo", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_pop_empty_array(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pop(&vec);
-	check_strvec(&vec, NULL);
-	strvec_clear(&vec);
-}
+	for_test ("pop with empty array") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pop(&vec);
+		check_strvec(&vec, NULL);
+		strvec_clear(&vec);
+	}

-static void t_pop_non_empty_array(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_pop(&vec);
-	check_strvec(&vec, "foo", "bar", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("pop with non-empty array") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_pop(&vec);
+		check_strvec(&vec, "foo", "bar", NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_empty_string(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "");
-	check_strvec(&vec, NULL);
-	strvec_clear(&vec);
-}
+	for_test ("split empty string") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "");
+		check_strvec(&vec, NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_single_item(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "foo");
-	check_strvec(&vec, "foo", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("split single item") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "foo");
+		check_strvec(&vec, "foo", NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_multiple_items(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "foo bar baz");
-	check_strvec(&vec, "foo", "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("split multiple items") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "foo bar baz");
+		check_strvec(&vec, "foo", "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_whitespace_only(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, " \t\n");
-	check_strvec(&vec, NULL);
-	strvec_clear(&vec);
-}
+	for_test ("split whitespace only") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, " \t\n");
+		check_strvec(&vec, NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_multiple_consecutive_whitespaces(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "foo\n\t bar");
-	check_strvec(&vec, "foo", "bar", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("split multiple consecutive whitespaces") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "foo\n\t bar");
+		check_strvec(&vec, "foo", "bar", NULL);
+		strvec_clear(&vec);
+	}

-static void t_detach(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	const char **detached;
+	for_test ("detach") {
+		struct strvec vec = STRVEC_INIT;
+		const char **detached;

-	strvec_push(&vec, "foo");
+		strvec_push(&vec, "foo");

-	detached = strvec_detach(&vec);
-	check_str(detached[0], "foo");
-	check_pointer_eq(detached[1], NULL);
+		detached = strvec_detach(&vec);
+		check_str(detached[0], "foo");
+		check_pointer_eq(detached[1], NULL);

-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);

-	free((char *) detached[0]);
-	free(detached);
-}
+		free((char *) detached[0]);
+		free(detached);
+	}

-int cmd_main(int argc, const char **argv)
-{
-	TEST(t_static_init(), "static initialization");
-	TEST(t_dynamic_init(), "dynamic initialization");
-	TEST(t_clear(), "clear");
-	TEST(t_push(), "push");
-	TEST(t_pushf(), "pushf");
-	TEST(t_pushl(), "pushl");
-	TEST(t_pushv(), "pushv");
-	TEST(t_replace_at_head(), "replace at head");
-	TEST(t_replace_in_between(), "replace in between");
-	TEST(t_replace_at_tail(), "replace at tail");
-	TEST(t_replace_with_substring(), "replace with substring");
-	TEST(t_remove_at_head(), "remove at head");
-	TEST(t_remove_in_between(), "remove in between");
-	TEST(t_remove_at_tail(), "remove at tail");
-	TEST(t_pop_empty_array(), "pop with empty array");
-	TEST(t_pop_non_empty_array(), "pop with non-empty array");
-	TEST(t_split_empty_string(), "split empty string");
-	TEST(t_split_single_item(), "split single item");
-	TEST(t_split_multiple_items(), "split multiple items");
-	TEST(t_split_whitespace_only(), "split whitespace only");
-	TEST(t_split_multiple_consecutive_whitespaces(), "split multiple consecutive whitespaces");
-	TEST(t_detach(), "detach");
 	return test_done();
 }
--
2.45.2

[PATCH v2 6/6] t-strbuf: use for_test

[René Scharfe]

The macro TEST takes a single expression.  If a test requires multiple
statements then they need to be placed in a function that's called in
the TEST expression.  The functions setup() and setup_populated() here
are used for that purpose and take another function as an argument,
making the control flow hard to follow.

Remove the overhead of these functions by using for_test instead.  Move
their duplicate post-condition checks into a new helper, t_release(),
and let t_addch() and t_addstr() accept properly typed input parameters
instead of void pointers.

Use the fully checking t_addstr() for adding initial values instead of
only doing only a length comparison -- there's no need for skipping the
other checks.

This results in test cases that look much more like strbuf usage in
production code, only with checked strbuf functions replaced by checking
wrappers.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-strbuf.c | 122 ++++++++++++++++++++--------------------
 1 file changed, 60 insertions(+), 62 deletions(-)

diff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c
index 6027dafef7..82cae4cbe3 100644
--- a/t/unit-tests/t-strbuf.c
+++ b/t/unit-tests/t-strbuf.c
@@ -1,32 +1,6 @@
 #include "test-lib.h"
 #include "strbuf.h"

-/* wrapper that supplies tests with an empty, initialized strbuf */
-static void setup(void (*f)(struct strbuf*, const void*),
-		  const void *data)
-{
-	struct strbuf buf = STRBUF_INIT;
-
-	f(&buf, data);
-	strbuf_release(&buf);
-	check_uint(buf.len, ==, 0);
-	check_uint(buf.alloc, ==, 0);
-}
-
-/* wrapper that supplies tests with a populated, initialized strbuf */
-static void setup_populated(void (*f)(struct strbuf*, const void*),
-			    const char *init_str, const void *data)
-{
-	struct strbuf buf = STRBUF_INIT;
-
-	strbuf_addstr(&buf, init_str);
-	check_uint(buf.len, ==, strlen(init_str));
-	f(&buf, data);
-	strbuf_release(&buf);
-	check_uint(buf.len, ==, 0);
-	check_uint(buf.alloc, ==, 0);
-}
-
 static int assert_sane_strbuf(struct strbuf *buf)
 {
 	/* Initialized strbufs should always have a non-NULL buffer */
@@ -45,31 +19,8 @@ static int assert_sane_strbuf(struct strbuf *buf)
 	return check_uint(buf->len, <, buf->alloc);
 }

-static void t_static_init(void)
+static void t_addch(struct strbuf *buf, int ch)
 {
-	struct strbuf buf = STRBUF_INIT;
-
-	check_uint(buf.len, ==, 0);
-	check_uint(buf.alloc, ==, 0);
-	check_char(buf.buf[0], ==, '\0');
-}
-
-static void t_dynamic_init(void)
-{
-	struct strbuf buf;
-
-	strbuf_init(&buf, 1024);
-	check(assert_sane_strbuf(&buf));
-	check_uint(buf.len, ==, 0);
-	check_uint(buf.alloc, >=, 1024);
-	check_char(buf.buf[0], ==, '\0');
-	strbuf_release(&buf);
-}
-
-static void t_addch(struct strbuf *buf, const void *data)
-{
-	const char *p_ch = data;
-	const char ch = *p_ch;
 	size_t orig_alloc = buf->alloc;
 	size_t orig_len = buf->len;

@@ -85,9 +36,8 @@ static void t_addch(struct strbuf *buf, const void *data)
 	check_char(buf->buf[buf->len], ==, '\0');
 }

-static void t_addstr(struct strbuf *buf, const void *data)
+static void t_addstr(struct strbuf *buf, const char *text)
 {
-	const char *text = data;
 	size_t len = strlen(text);
 	size_t orig_alloc = buf->alloc;
 	size_t orig_len = buf->len;
@@ -105,18 +55,66 @@ static void t_addstr(struct strbuf *buf, const void *data)
 	check_str(buf->buf + orig_len, text);
 }

+static void t_release(struct strbuf *sb)
+{
+	strbuf_release(sb);
+	check_uint(sb->len, ==, 0);
+	check_uint(sb->alloc, ==, 0);
+}
+
 int cmd_main(int argc, const char **argv)
 {
-	if (!TEST(t_static_init(), "static initialization works"))
-		test_skip_all("STRBUF_INIT is broken");
-	TEST(t_dynamic_init(), "dynamic initialization works");
-	TEST(setup(t_addch, "a"), "strbuf_addch adds char");
-	TEST(setup(t_addch, ""), "strbuf_addch adds NUL char");
-	TEST(setup_populated(t_addch, "initial value", "a"),
-	     "strbuf_addch appends to initial value");
-	TEST(setup(t_addstr, "hello there"), "strbuf_addstr adds string");
-	TEST(setup_populated(t_addstr, "initial value", "hello there"),
-	     "strbuf_addstr appends string to initial value");
+	for_test ("static initialization works") {
+		struct strbuf buf = STRBUF_INIT;
+
+		if (!check_uint(buf.len, ==, 0) ||
+		    !check_uint(buf.alloc, ==, 0) ||
+		    !check_char(buf.buf[0], ==, '\0'))
+			test_skip_all("STRBUF_INIT is broken");
+	}
+
+	for_test ("dynamic initialization works") {
+		struct strbuf buf;
+
+		strbuf_init(&buf, 1024);
+		check(assert_sane_strbuf(&buf));
+		check_uint(buf.len, ==, 0);
+		check_uint(buf.alloc, >=, 1024);
+		check_char(buf.buf[0], ==, '\0');
+		strbuf_release(&buf);
+	}
+
+	for_test ("strbuf_addch adds char") {
+		struct strbuf sb = STRBUF_INIT;
+		t_addch(&sb, 'a');
+		t_release(&sb);
+	}
+
+	for_test ("strbuf_addch adds NUL char") {
+		struct strbuf sb = STRBUF_INIT;
+		t_addch(&sb, '\0');
+		t_release(&sb);
+	}
+
+	for_test ("strbuf_addch appends to initial value") {
+		struct strbuf sb = STRBUF_INIT;
+		t_addstr(&sb, "initial value");
+		t_addch(&sb, 'a');
+		t_release(&sb);
+	}
+
+	for_test ("strbuf_addstr adds string") {
+		struct strbuf sb = STRBUF_INIT;
+		t_addstr(&sb, "hello there");
+		t_release(&sb);
+	}
+
+	for_test ("strbuf_addstr appends string to initial value") {
+		struct strbuf sb = STRBUF_INIT;
+		t_addstr(&sb, "initial value");
+		t_addstr(&sb, "hello there");
+		t_release(&sb);
+	}

 	return test_done();
 }
--
2.45.2

Re: [PATCH v2 2/6] unit-tests: add for_test

[Kyle Lippincott]

On Sat, Jul 20, 2024 at 11:22 PM René Scharfe <l.s.r@web.de> wrote:
>
> The macro TEST only allows defining a test that consists of a single
> expression.  Add a new macro, for_test, which provides a way to define
> unit tests that are made up of one or more statements.
>
> for_test allows defining self-contained tests en bloc, a bit like
> test_expect_success does for regular tests.  It acts like a for loop
> that runs at most once; the test body is executed if test_skip_all()
> had not been called before.

I can see based on this description where the name came from, but
without this context, it's not clear when reading a test what it
actually does. The name comes from an implementation detail, and is
not describing what it _does_, just _how_ it does it.

Maybe a name like `small_test` or `quick_test` or something like that
would better express the intended usage?

>
> Helped-by: Phillip Wood <phillip.wood123@gmail.com>
> Signed-off-by: René Scharfe <l.s.r@web.de>
> ---
>  .clang-format               |  2 ++
>  t/helper/test-example-tap.c | 33 +++++++++++++++++++++++++++++++++
>  t/t0080/expect              | 35 ++++++++++++++++++++++++++++++++++-
>  t/unit-tests/test-lib.h     | 19 +++++++++++++++++++
>  4 files changed, 88 insertions(+), 1 deletion(-)
>
> diff --git a/.clang-format b/.clang-format
> index 6408251577..863dc87dfc 100644
> --- a/.clang-format
> +++ b/.clang-format
> @@ -151,6 +151,7 @@ Cpp11BracedListStyle: false
>  # function calls. Taken from:
>  #   git grep -h '^#define [^[:space:]]*for_\?each[^[:space:]]*(' |
>  #   sed "s/^#define /  - '/; s/(.*$/'/" | sort | uniq
> +# Added for_test from t/unit-tests/test-lib.h manually as a special case.
>  ForEachMacros:
>    - 'for_each_builtin'
>    - 'for_each_string_list_item'
> @@ -168,6 +169,7 @@ ForEachMacros:
>    - 'strintmap_for_each_entry'
>    - 'strmap_for_each_entry'
>    - 'strset_for_each_entry'
> +  - 'for_test'
>
>  # The maximum number of consecutive empty lines to keep.
>  MaxEmptyLinesToKeep: 1
> diff --git a/t/helper/test-example-tap.c b/t/helper/test-example-tap.c
> index d072ad559f..51d5e6e75b 100644
> --- a/t/helper/test-example-tap.c
> +++ b/t/helper/test-example-tap.c
> @@ -92,5 +92,38 @@ int cmd__example_tap(int argc, const char **argv)
>         test_res = TEST(t_empty(), "test with no checks");
>         TEST(check_int(test_res, ==, 0), "test with no checks returns 0");
>
> +       for_test ("for_test passing test")
> +               check_int(1, ==, 1);

I'm concerned that users will write this like:
+       for_test ("for_test passing test");
+               check_int(1, ==, 1);

And the issue won't be caught. Maybe that's fine; the only issue here
is that it's not going to be as descriptive if it fails, right?

> +       for_test ("for_test failing test")
> +               check_int(1, ==, 2);
> +       for_test ("for_test passing TEST_TODO()")
> +               TEST_TODO(check(0));
> +       for_test ("for_test failing TEST_TODO()")
> +               TEST_TODO(check(1));
> +       for_test ("for_test test_skip()") {
> +               check(0);
> +               test_skip("missing prerequisite");
> +               check(1);
> +       }
> +       for_test ("for_test test_skip() inside TEST_TODO()")
> +               TEST_TODO((test_skip("missing prerequisite"), 1));
> +       for_test ("for_test TEST_TODO() after failing check") {
> +               check(0);
> +               TEST_TODO(check(0));
> +       }
> +       for_test ("for_test failing check after TEST_TODO()") {
> +               check(1);
> +               TEST_TODO(check(0));
> +               check(0);
> +       }
> +       for_test ("for_test messages from failing string and char comparison") {
> +               check_str("\thello\\", "there\"\n");
> +               check_str("NULL", NULL);
> +               check_char('a', ==, '\n');
> +               check_char('\\', ==, '\'');
> +       }
> +       for_test ("for_test test with no checks")
> +               ; /* nothing */
> +
>         return test_done();
>  }
> diff --git a/t/t0080/expect b/t/t0080/expect
> index 0cfa0dc6d8..583f41b8c9 100644
> --- a/t/t0080/expect
> +++ b/t/t0080/expect
> @@ -40,4 +40,37 @@ not ok 17 - messages from failing string and char comparison
>  # BUG: test has no checks at t/helper/test-example-tap.c:92
>  not ok 18 - test with no checks
>  ok 19 - test with no checks returns 0
> -1..19
> +ok 20 - for_test passing test
> +# check "1 == 2" failed at t/helper/test-example-tap.c:98
> +#    left: 1
> +#   right: 2
> +not ok 21 - for_test failing test
> +not ok 22 - for_test passing TEST_TODO() # TODO
> +# todo check 'check(1)' succeeded at t/helper/test-example-tap.c:102
> +not ok 23 - for_test failing TEST_TODO()
> +# check "0" failed at t/helper/test-example-tap.c:104
> +# skipping test - missing prerequisite
> +# skipping check '1' at t/helper/test-example-tap.c:106
> +ok 24 - for_test test_skip() # SKIP
> +# skipping test - missing prerequisite
> +ok 25 - for_test test_skip() inside TEST_TODO() # SKIP
> +# check "0" failed at t/helper/test-example-tap.c:111
> +not ok 26 - for_test TEST_TODO() after failing check
> +# check "0" failed at t/helper/test-example-tap.c:117
> +not ok 27 - for_test failing check after TEST_TODO()
> +# check "!strcmp("\thello\\", "there\"\n")" failed at t/helper/test-example-tap.c:120
> +#    left: "\011hello\\"
> +#   right: "there\"\012"
> +# check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:121
> +#    left: "NULL"
> +#   right: NULL
> +# check "'a' == '\n'" failed at t/helper/test-example-tap.c:122
> +#    left: 'a'
> +#   right: '\012'
> +# check "'\\' == '\''" failed at t/helper/test-example-tap.c:123
> +#    left: '\\'
> +#   right: '\''
> +not ok 28 - for_test messages from failing string and char comparison
> +# BUG: test has no checks at t/helper/test-example-tap.c:125
> +not ok 29 - for_test test with no checks
> +1..29
> diff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h
> index 2de6d715d5..12afd47ac9 100644
> --- a/t/unit-tests/test-lib.h
> +++ b/t/unit-tests/test-lib.h
> @@ -14,6 +14,25 @@
>         test__run_end(test__run_begin() ? 0 : (t, 1),   \
>                       TEST_LOCATION(),  __VA_ARGS__)
>
> +/*
> + * Run a test unless test_skip_all() has been called.  Acts like a for
> + * loop that runs at most once, with the test description between the
> + * parentheses and the test body as a statement or block after them.
> + * The description for each test should be unique.  E.g.:
> + *
> + *  for_test ("something else %d %d", arg1, arg2) {
> + *          prepare();
> + *          test_something_else(arg1, arg2);
> + *          cleanup();
> + *  }
> + */
> +#define for_test(...)                                                  \
> +       for (int for_test_running_ = test__run_begin() ?                \
> +               (test__run_end(0, TEST_LOCATION(), __VA_ARGS__), 0) : 1;\
> +            for_test_running_;                                         \
> +            test__run_end(1, TEST_LOCATION(), __VA_ARGS__),            \
> +               for_test_running_ = 0)

IMHO: this is borderline "too much magic" for my tastes. I think
having multiple test functions is generally easier to understand, and
the overhead isn't really relevant. It's not _as_ compact in the
source file, and requires that we have both the TEST statement and the
function (and forgetting the TEST statement means that we won't invoke
the function). If that is the main issue we're facing here, I wonder
if there's some other way of resolving that (such as unused function
detection via some compiler flags; even if it's not detected on all
platforms, getting at least one of the CI platforms should be
sufficient to prevent the issue [but we should target as many as
possible, so it's caught earlier than "I tried to send it to the
list"])

If others agree that this is a good simplification for the people
reading the test code (and hopefully for the test author), I'm fine
with this going in (with a different name). I'm not trying to veto the
concept.


> +
>  /*
>   * Print a test plan, should be called before any tests. If the number
>   * of tests is not known in advance test_done() will automatically
> --
> 2.45.2
>

Re: [PATCH v2 2/6] unit-tests: add for_test

[Junio C Hamano]

Kyle Lippincott <spectral@google.com> writes:

> I can see based on this description where the name came from, but
> without this context, it's not clear when reading a test what it
> actually does. The name comes from an implementation detail, and is
> not describing what it _does_, just _how_ it does it.
>
> Maybe a name like `small_test` or `quick_test` or something like that
> would better express the intended usage?

Names that explicitly have C keyword for control structures, e.g.
"if_somecondition_test()", "while_foo_test()" or "for_test()" is
vastly preferrable than these, in order to make it clear that we are
introducing a macro that defines control structure.

>> +       for_test ("for_test passing test")
>> +               check_int(1, ==, 1);
>
> I'm concerned that users will write this like:
> +       for_test ("for_test passing test");
> +               check_int(1, ==, 1);

And that is exactly why we want the macro name to include C keyword 
for control structures.

> And the issue won't be caught.

You are right.  Making an empty body somehow catchable by the
compiler would be a vast improvement.

>> +#define for_test(...)                                                  \
>> +       for (int for_test_running_ = test__run_begin() ?                \
>> +               (test__run_end(0, TEST_LOCATION(), __VA_ARGS__), 0) : 1;\
>> +            for_test_running_;                                         \
>> +            test__run_end(1, TEST_LOCATION(), __VA_ARGS__),            \
>> +               for_test_running_ = 0)
>
> IMHO: this is borderline "too much magic" for my tastes. I think
> having multiple test functions is generally easier to understand, and
> the overhead isn't really relevant. It's not _as_ compact in the
> source file, and requires that we have both the TEST statement and the
> function (and forgetting the TEST statement means that we won't invoke
> the function). If that is the main issue we're facing here, I wonder
> if there's some other way of resolving that (such as unused function
> detection via some compiler flags; even if it's not detected on all
> platforms, getting at least one of the CI platforms should be
> sufficient to prevent the issue [but we should target as many as
> possible, so it's caught earlier than "I tried to send it to the
> list"])

Interesting.

> If others agree that this is a good simplification for the people
> reading the test code (and hopefully for the test author), I'm fine
> with this going in (with a different name). I'm not trying to veto the
> concept.

OK.  But what you suggested in the previous paragraph has merit.
Are there other things that could be improved in the existing unit
test helpers, that would help those who do not use this new for_test()
thing?  Let's see how the patches to improve them would look like.

Thanks.

Re: [PATCH v2 2/6] unit-tests: add for_test

[René Scharfe]

Am 22.07.24 um 21:36 schrieb Junio C Hamano:
> Kyle Lippincott <spectral@google.com> writes:
>
>>> +       for_test ("for_test passing test")
>>> +               check_int(1, ==, 1);
>>
>> I'm concerned that users will write this like:
>> +       for_test ("for_test passing test");
>> +               check_int(1, ==, 1);
>
> And that is exactly why we want the macro name to include C keyword
> for control structures.
>
>> And the issue won't be caught.
>
> You are right.  Making an empty body somehow catchable by the
> compiler would be a vast improvement.

That would be nice, but I have no idea how to do that without compiler
changes.  In the meantime the existing runtime checks will catch both
the empty test in the first line and the out-of-test check in the second
one and report them like this:

 # BUG: test has no checks at t/helper/test-example-tap.c:75
 not ok 1 - for_test passing test
 Assertion failed: (ctx.running), function test_assert, file test-lib.c, line 267.

File name and line in the second one are not as helpful as they could
be.  Here's a patch to improve that output.

--- >8 ---
Subject: [PATCH] unit-tests: show location of checks outside of tests

Checks outside of tests are caught at runtime and reported like this:

 Assertion failed: (ctx.running), function test_assert, file test-lib.c, line 267.

The assert() call aborts the unit test and doesn't reveal the location
or even the type of the offending check, as test_assert() is called by
all of them.

Handle it like the opposite case, a test without any checks: Don't
abort, but report the location of the actual check, along with a message
explaining the situation.  The output for example above becomes:

 # BUG: check outside of test at t/helper/test-example-tap.c:75

... and the unit test program continues.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/test-lib.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c
index 3c513ce59a..9977c81739 100644
--- a/t/unit-tests/test-lib.c
+++ b/t/unit-tests/test-lib.c
@@ -264,7 +264,11 @@ static void test_todo(void)

 int test_assert(const char *location, const char *check, int ok)
 {
-	assert(ctx.running);
+	if (!ctx.running) {
+		test_msg("BUG: check outside of test at %s",
+			 make_relative(location));
+		return 0;
+	}

 	if (ctx.result == RESULT_SKIP) {
 		test_msg("skipping check '%s' at %s", check,
--
2.45.2

Re: [PATCH v2 2/6] unit-tests: add for_test

[Junio C Hamano]

René Scharfe <l.s.r@web.de> writes:

>>> And the issue won't be caught.
>>
>> You are right.  Making an empty body somehow catchable by the
>> compiler would be a vast improvement.
>
> That would be nice, but I have no idea how to do that without compiler
> changes.

Me neither.  I was trying to nerd-snipe Kyle into coming up with a
solution ;-)

> In the meantime the existing runtime checks will catch both
> the empty test in the first line and the out-of-test check in the second
> one and report them like this:
>
>  # BUG: test has no checks at t/helper/test-example-tap.c:75
>  not ok 1 - for_test passing test
>  Assertion failed: (ctx.running), function test_assert, file test-lib.c, line 267.

Nice improvement, I would say.

> diff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c
> index 3c513ce59a..9977c81739 100644
> --- a/t/unit-tests/test-lib.c
> +++ b/t/unit-tests/test-lib.c
> @@ -264,7 +264,11 @@ static void test_todo(void)
>
>  int test_assert(const char *location, const char *check, int ok)
>  {
> -	assert(ctx.running);
> +	if (!ctx.running) {
> +		test_msg("BUG: check outside of test at %s",
> +			 make_relative(location));
> +		return 0;
> +	}
>
>  	if (ctx.result == RESULT_SKIP) {
>  		test_msg("skipping check '%s' at %s", check,
> --
> 2.45.2

Re: [PATCH v2 2/6] unit-tests: add for_test

[Kyle Lippincott]

On Mon, Jul 22, 2024 at 12:37 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> Kyle Lippincott <spectral@google.com> writes:
>
> > I can see based on this description where the name came from, but
> > without this context, it's not clear when reading a test what it
> > actually does. The name comes from an implementation detail, and is
> > not describing what it _does_, just _how_ it does it.
> >
> > Maybe a name like `small_test` or `quick_test` or something like that
> > would better express the intended usage?
>
> Names that explicitly have C keyword for control structures, e.g.
> "if_somecondition_test()", "while_foo_test()" or "for_test()" is
> vastly preferrable than these, in order to make it clear that we are
> introducing a macro that defines control structure.

Perhaps expression_test or something, then? It's testing an expression
(I think blocks are a type of expression? I never remember which is
'larger': expressions or statements, and that might only be a C++
thing, C might not have the same terminology).

I was going to suggest a lint check that checks to ensure that we
don't have a semicolon immediately after the description, or a lint
check that enforced that even single-statement tests are wrapped in {}
(inverting the style guide requirements), but realistically neither
are actually needed: `test__run_begin` already asserts that a test
isn't currently running. `check_int` and friends already assert that a
test _is_ running. So this is already defended against:

for_test ("test description");
    check_int(1, ==, 1);   /* `assert`s: not in the middle of a test */

What we don't have is defense against a completely empty test, or a
test without any actual conditions:

for_test ("test one");
for_test ("test two") {
    printf("this test doesn't actually _test_ anything\n");
}

Adding that is doable, and improves the first case: the `for_test
("test description");` line fails because the test didn't do anything,
not the `check_int` line.

>
> >> +       for_test ("for_test passing test")
> >> +               check_int(1, ==, 1);
> >
> > I'm concerned that users will write this like:
> > +       for_test ("for_test passing test");
> > +               check_int(1, ==, 1);
>
> And that is exactly why we want the macro name to include C keyword
> for control structures.
>
> > And the issue won't be caught.
>
> You are right.  Making an empty body somehow catchable by the
> compiler would be a vast improvement.
>
> >> +#define for_test(...)                                                  \
> >> +       for (int for_test_running_ = test__run_begin() ?                \
> >> +               (test__run_end(0, TEST_LOCATION(), __VA_ARGS__), 0) : 1;\
> >> +            for_test_running_;                                         \
> >> +            test__run_end(1, TEST_LOCATION(), __VA_ARGS__),            \
> >> +               for_test_running_ = 0)
> >
> > IMHO: this is borderline "too much magic" for my tastes. I think
> > having multiple test functions is generally easier to understand, and
> > the overhead isn't really relevant. It's not _as_ compact in the
> > source file, and requires that we have both the TEST statement and the
> > function (and forgetting the TEST statement means that we won't invoke
> > the function). If that is the main issue we're facing here, I wonder
> > if there's some other way of resolving that (such as unused function
> > detection via some compiler flags; even if it's not detected on all
> > platforms, getting at least one of the CI platforms should be
> > sufficient to prevent the issue [but we should target as many as
> > possible, so it's caught earlier than "I tried to send it to the
> > list"])
>
> Interesting.
>
> > If others agree that this is a good simplification for the people
> > reading the test code (and hopefully for the test author), I'm fine
> > with this going in (with a different name). I'm not trying to veto the
> > concept.
>
> OK.  But what you suggested in the previous paragraph has merit.
> Are there other things that could be improved in the existing unit
> test helpers, that would help those who do not use this new for_test()
> thing?  Let's see how the patches to improve them would look like.
>
> Thanks.
>

Re: [PATCH v2 2/6] unit-tests: add for_test

[Kyle Lippincott]

On Mon, Jul 22, 2024 at 1:31 PM René Scharfe <l.s.r@web.de> wrote:
>
> Am 22.07.24 um 21:36 schrieb Junio C Hamano:
> > Kyle Lippincott <spectral@google.com> writes:
> >
> >>> +       for_test ("for_test passing test")
> >>> +               check_int(1, ==, 1);
> >>
> >> I'm concerned that users will write this like:
> >> +       for_test ("for_test passing test");
> >> +               check_int(1, ==, 1);
> >
> > And that is exactly why we want the macro name to include C keyword
> > for control structures.
> >
> >> And the issue won't be caught.
> >
> > You are right.  Making an empty body somehow catchable by the
> > compiler would be a vast improvement.
>
> That would be nice, but I have no idea how to do that without compiler
> changes.  In the meantime the existing runtime checks will catch both
> the empty test in the first line and the out-of-test check in the second
> one and report them like this:
>
>  # BUG: test has no checks at t/helper/test-example-tap.c:75
>  not ok 1 - for_test passing test
>  Assertion failed: (ctx.running), function test_assert, file test-lib.c, line 267.
>
> File name and line in the second one are not as helpful as they could
> be.  Here's a patch to improve that output.
>
> --- >8 ---
> Subject: [PATCH] unit-tests: show location of checks outside of tests
>
> Checks outside of tests are caught at runtime and reported like this:
>
>  Assertion failed: (ctx.running), function test_assert, file test-lib.c, line 267.
>
> The assert() call aborts the unit test and doesn't reveal the location
> or even the type of the offending check, as test_assert() is called by
> all of them.
>
> Handle it like the opposite case, a test without any checks: Don't
> abort, but report the location of the actual check, along with a message
> explaining the situation.  The output for example above becomes:

Oh, I referenced "detect tests without checks" as a possible
improvement in my previous message, and didn't actually check whether
it was there. This statement made me go and check, and you're right,
it is there already. So I think we're well defended against the empty
check case, especially with the improvement here. Thanks!

If we have a good set of protections against misuse, does this mean
we're free to rename it? :) The concern raised for why `for` had to be
in the name was because it was using a control statement (a
at-most-1-execution for loop) to achieve its magic, and the control
statement puts certain restrictions and obligations on how to use it
correctly. If the misuse is detected reliably, we can choose a name
that's more descriptive about what it's doing.

>
>  # BUG: check outside of test at t/helper/test-example-tap.c:75
>
> ... and the unit test program continues.
>
> Signed-off-by: René Scharfe <l.s.r@web.de>
> ---
>  t/unit-tests/test-lib.c | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c
> index 3c513ce59a..9977c81739 100644
> --- a/t/unit-tests/test-lib.c
> +++ b/t/unit-tests/test-lib.c
> @@ -264,7 +264,11 @@ static void test_todo(void)
>
>  int test_assert(const char *location, const char *check, int ok)
>  {
> -       assert(ctx.running);
> +       if (!ctx.running) {
> +               test_msg("BUG: check outside of test at %s",
> +                        make_relative(location));
> +               return 0;
> +       }
>
>         if (ctx.result == RESULT_SKIP) {
>                 test_msg("skipping check '%s' at %s", check,
> --
> 2.45.2

[PATCH v2] unit-tests: show location of checks outside of tests

[René Scharfe]

Checks outside of tests are caught at runtime and reported like this:

 Assertion failed: (ctx.running), function test_assert, file test-lib.c, line 267.

The assert() call aborts the unit test and doesn't reveal the location
or even the type of the offending check, as test_assert() is called by
all of them.

Handle it like the opposite case, a test without any checks: Don't
abort, but report the location of the actual check, along with a message
explaining the situation.  The output for example above becomes:

 # BUG: check outside of test at t/helper/test-example-tap.c:75

... and the unit test program continues and indicates the error in its
exit code at the end.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
Changes since v1:
- Set ctx.failed to report the mistake via exit code as well.

 t/unit-tests/test-lib.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c
index 3c513ce59a..989dc758e6 100644
--- a/t/unit-tests/test-lib.c
+++ b/t/unit-tests/test-lib.c
@@ -264,7 +264,12 @@ static void test_todo(void)

 int test_assert(const char *location, const char *check, int ok)
 {
-	assert(ctx.running);
+	if (!ctx.running) {
+		test_msg("BUG: check outside of test at %s",
+			 make_relative(location));
+		ctx.failed = 1;
+		return 0;
+	}

 	if (ctx.result == RESULT_SKIP) {
 		test_msg("skipping check '%s' at %s", check,
--
2.45.2

Re: [PATCH v2 2/6] unit-tests: add for_test

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 4437 bytes --]

On Mon, Jul 22, 2024 at 12:36:57PM -0700, Junio C Hamano wrote:
> Kyle Lippincott <spectral@google.com> writes:
> >> +#define for_test(...)                                                  \
> >> +       for (int for_test_running_ = test__run_begin() ?                \
> >> +               (test__run_end(0, TEST_LOCATION(), __VA_ARGS__), 0) : 1;\
> >> +            for_test_running_;                                         \
> >> +            test__run_end(1, TEST_LOCATION(), __VA_ARGS__),            \
> >> +               for_test_running_ = 0)
> >
> > IMHO: this is borderline "too much magic" for my tastes. I think
> > having multiple test functions is generally easier to understand, and
> > the overhead isn't really relevant. It's not _as_ compact in the
> > source file, and requires that we have both the TEST statement and the
> > function (and forgetting the TEST statement means that we won't invoke
> > the function). If that is the main issue we're facing here, I wonder
> > if there's some other way of resolving that (such as unused function
> > detection via some compiler flags; even if it's not detected on all
> > platforms, getting at least one of the CI platforms should be
> > sufficient to prevent the issue [but we should target as many as
> > possible, so it's caught earlier than "I tried to send it to the
> > list"])
> 
> Interesting.
> 
> > If others agree that this is a good simplification for the people
> > reading the test code (and hopefully for the test author), I'm fine
> > with this going in (with a different name). I'm not trying to veto the
> > concept.
> 
> OK.  But what you suggested in the previous paragraph has merit.
> Are there other things that could be improved in the existing unit
> test helpers, that would help those who do not use this new for_test()
> thing?  Let's see how the patches to improve them would look like.

Honestly I'm also not too sure how I feel about these new macros, and
I'm somewhat in the same boat that it starts to feels "magicky".

Taking a step back: what is it that is bugging folks about the current
way of writing one function per test? The boilerplate cannot really be
it, as tests should be self-contained anyway and thus we should have the
same boilerplate regardless of whether we use separate functions or the
new macros. I also doubt that it's the function declaration itself, as
that isn't all that involved. So I guess what people are annoyed with is
that every function needs to be defined, but then also wired up via
`TEST_RUN()`, right? And that I totally get, as it is quite annoying.

So... can we solve this in a different way? I quite liked the system
that we had in libgit2: every function must conform to a specific name
`test_foo_bar`. We then have a Python script that reads all test files,
extracts all files that have the `test_` prefix, and writes those into
an array. Optionally, the `test_foo` test suite can also have a setup
and teardown function that gets called for every test, namely
`test_foo_setup()` and `test_foo_teardown()`.

Altogether, the output would look somewhat like this:


```test.c
static test_foo_setup(void)
{
    ... setup global state ...
}

static test_foo_teardown(void)
{
    ... tear down global state ...
}

static test_foo_one(void)
{
    ... first test ...
}

static test_foo_two(void)
{
    ... second test ...
}
```

```generated.c
static const struct test_func _test_foo_functions[] = {
    {
        name: "foo::one",
        test: test_foo_one,
        setup: test_foo_setup,
        teardown: test_foo_teardown,
    },
    {
        name: "foo::two",
        test: test_foo_two,
        setup: test_foo_setup,
        teardown: test_foo_teardown,
    },
};

int main(int argc, const char *argv[])
{
    ... boilerplate to execute all functions ...
}
```

The setup and teardown functions are optional -- if not defined for a
specific test unit, then we simply won't invoke them.

There is of course some magic involved with how we generate the file.
But I think that would be quite manageable, and ultimately all that the
developer would need to care about is writing a `test_foo_something()`
function. Everything else would be handled by our infra.

I'd be happy to implement something along these lines if folks think
that this is a good direction to go into.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 2/6] unit-tests: add for_test

[René Scharfe]

Am 23.07.24 um 00:41 schrieb Kyle Lippincott:
> On Mon, Jul 22, 2024 at 12:37 PM Junio C Hamano <gitster@pobox.com> wrote:
>>
>> Kyle Lippincott <spectral@google.com> writes:
>>
>>> I can see based on this description where the name came from, but
>>> without this context, it's not clear when reading a test what it
>>> actually does. The name comes from an implementation detail, and is
>>> not describing what it _does_, just _how_ it does it.
>>>
>>> Maybe a name like `small_test` or `quick_test` or something like that
>>> would better express the intended usage?
>>
>> Names that explicitly have C keyword for control structures, e.g.
>> "if_somecondition_test()", "while_foo_test()" or "for_test()" is
>> vastly preferrable than these, in order to make it clear that we are
>> introducing a macro that defines control structure.
>
> Perhaps expression_test or something, then? It's testing an expression
> (I think blocks are a type of expression? I never remember which is
> 'larger': expressions or statements, and that might only be a C++
> thing, C might not have the same terminology).

If in doubt, feel free to look it up.  We're not taking a test here
(bad pun intended).

https://en.cppreference.com/w/c/language/expressions
https://en.cppreference.com/w/c/language/statements

TEST requires test bodies to be expressions, for_test allows them to
be statements.  Statements can include arbitrary expressions, but not
the other way around.

The "for" in for_test indicates that it's a for-like macro.  This
already implies that it is followed by a statement.  It works for the
existing for_each macros.

René

Re: [PATCH v2 2/6] unit-tests: add for_test

[René Scharfe]

Am 23.07.24 um 08:36 schrieb Patrick Steinhardt:
> On Mon, Jul 22, 2024 at 12:36:57PM -0700, Junio C Hamano wrote:
>> Kyle Lippincott <spectral@google.com> writes:
>>>> +#define for_test(...)                                                  \
>>>> +       for (int for_test_running_ = test__run_begin() ?                \
>>>> +               (test__run_end(0, TEST_LOCATION(), __VA_ARGS__), 0) : 1;\
>>>> +            for_test_running_;                                         \
>>>> +            test__run_end(1, TEST_LOCATION(), __VA_ARGS__),            \
>>>> +               for_test_running_ = 0)
>>>
>>> IMHO: this is borderline "too much magic" for my tastes. I think
>>> having multiple test functions is generally easier to understand, and
>>> the overhead isn't really relevant. It's not _as_ compact in the
>>> source file, and requires that we have both the TEST statement and the
>>> function (and forgetting the TEST statement means that we won't invoke
>>> the function). If that is the main issue we're facing here, I wonder
>>> if there's some other way of resolving that (such as unused function
>>> detection via some compiler flags; even if it's not detected on all
>>> platforms, getting at least one of the CI platforms should be
>>> sufficient to prevent the issue [but we should target as many as
>>> possible, so it's caught earlier than "I tried to send it to the
>>> list"])
>>
>> Interesting.
>>
>>> If others agree that this is a good simplification for the people
>>> reading the test code (and hopefully for the test author), I'm fine
>>> with this going in (with a different name). I'm not trying to veto the
>>> concept.
>>
>> OK.  But what you suggested in the previous paragraph has merit.
>> Are there other things that could be improved in the existing unit
>> test helpers, that would help those who do not use this new for_test()
>> thing?  Let's see how the patches to improve them would look like.
>
> Honestly I'm also not too sure how I feel about these new macros, and
> I'm somewhat in the same boat that it starts to feels "magicky".
>
> Taking a step back: what is it that is bugging folks about the current
> way of writing one function per test?

My goal is to be able to define a test without repeating its
description even partly, like test_expect_success allows for shell-based
tests.

> I quite liked the system
> that we had in libgit2: every function must conform to a specific name
> `test_foo_bar`. We then have a Python script that reads all test files,
> extracts all files that have the `test_` prefix, and writes those into
> an array. Optionally, the `test_foo` test suite can also have a setup
> and teardown function that gets called for every test, namely
> `test_foo_setup()` and `test_foo_teardown()`.
>
> Altogether, the output would look somewhat like this:
>
>
> ```test.c
> static test_foo_setup(void)
> {
>     ... setup global state ...
> }
>
> static test_foo_teardown(void)
> {
>     ... tear down global state ...
> }
>
> static test_foo_one(void)
> {
>     ... first test ...
> }
>
> static test_foo_two(void)
> {
>     ... second test ...
> }
> ```
>
> ```generated.c
> static const struct test_func _test_foo_functions[] = {
>     {
>         name: "foo::one",
>         test: test_foo_one,
>         setup: test_foo_setup,
>         teardown: test_foo_teardown,
>     },
>     {
>         name: "foo::two",
>         test: test_foo_two,
>         setup: test_foo_setup,
>         teardown: test_foo_teardown,
>     },
> };
>
> int main(int argc, const char *argv[])
> {
>     ... boilerplate to execute all functions ...
> }
> ```
>
> The setup and teardown functions are optional -- if not defined for a
> specific test unit, then we simply won't invoke them.

So the name of a test is generated automatically based on the function
name and there is no way to add a description beyond that.  This would
achieve my goal above, but prevent developers from using space and
punctuation in test descriptions.  Fair enough.

> There is of course some magic involved with how we generate the file.

It requires magic function names and generates code using a different
language, while for_test is a just single new control flow keyword,
like the dozen or so we already have.  So the magic employed by the
libgit2 system is both broader and deeper.

> But I think that would be quite manageable, and ultimately all that the
> developer would need to care about is writing a `test_foo_something()`
> function. Everything else would be handled by our infra.

With for_test all the developer has to do is write a test with a
description, no extra infrastructure beyond the existing unit test
framework needed.

> I'd be happy to implement something along these lines if folks think
> that this is a good direction to go into.

FWIW I don't see the appeal.  It uses more code and more dependencies
to almost allow what for_test permits.

René

Re: [PATCH v2 2/6] unit-tests: add for_test

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

On Tue, Jul 23, 2024 at 11:25:29AM +0200, René Scharfe wrote:
> Am 23.07.24 um 08:36 schrieb Patrick Steinhardt:
> > There is of course some magic involved with how we generate the file.
> 
> It requires magic function names and generates code using a different
> language, while for_test is a just single new control flow keyword,
> like the dozen or so we already have.  So the magic employed by the
> libgit2 system is both broader and deeper.

It is broader, that's certainly true. But it feels more self-contained,
less fragile and easier to read to me compared to macros.

> > But I think that would be quite manageable, and ultimately all that the
> > developer would need to care about is writing a `test_foo_something()`
> > function. Everything else would be handled by our infra.
> 
> With for_test all the developer has to do is write a test with a
> description, no extra infrastructure beyond the existing unit test
> framework needed.

True, but it feels like it is an invitation for writing unidiomatic
code to me. Unidiomatic in this context to me mostly means code that is
not self-contained and thus cannot be run standalone.

These are quite obviously subjective opinions though.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 2/6] unit-tests: add for_test

[René Scharfe]

Am 23.07.24 um 11:53 schrieb Patrick Steinhardt:
> On Tue, Jul 23, 2024 at 11:25:29AM +0200, René Scharfe wrote:
>> Am 23.07.24 um 08:36 schrieb Patrick Steinhardt:
>>> There is of course some magic involved with how we generate the file.
>>
>> It requires magic function names and generates code using a different
>> language, while for_test is a just single new control flow keyword,
>> like the dozen or so we already have.  So the magic employed by the
>> libgit2 system is both broader and deeper.
>
> It is broader, that's certainly true. But it feels more self-contained,
> less fragile and easier to read to me compared to macros.

In which ways can for_test break?

>>> But I think that would be quite manageable, and ultimately all that the
>>> developer would need to care about is writing a `test_foo_something()`
>>> function. Everything else would be handled by our infra.
>>
>> With for_test all the developer has to do is write a test with a
>> description, no extra infrastructure beyond the existing unit test
>> framework needed.
>
> True, but it feels like it is an invitation for writing unidiomatic
> code to me. Unidiomatic in this context to me mostly means code that is
> not self-contained and thus cannot be run standalone.

Partly this is intentional -- I want to do something that the current
idiom (the TEST macro) doesn't allow.

It's possible that reducing friction will cause more sloppy tests to
be created.  I hope that the bad parts will stand out more when
there's less boilerplate code.

René

Re: [PATCH v2 2/6] unit-tests: add for_test

[René Scharfe]

Am 23.07.24 um 00:47 schrieb Kyle Lippincott:
>
> If we have a good set of protections against misuse, does this mean
> we're free to rename it? :) The concern raised for why `for` had to be
> in the name was because it was using a control statement (a
> at-most-1-execution for loop) to achieve its magic, and the control
> statement puts certain restrictions and obligations on how to use it
> correctly. If the misuse is detected reliably, we can choose a name
> that's more descriptive about what it's doing.

Well, I called it "test" initially because I'm lazy and it describes
its purpose.

But the fact remains that this is an iteration statement, even though
it kinda looks like a function, and it should be used as such.  We have
to explicitly tell clang-format, and having "for" in the name reminds
developers as well.

There is a slight dissonance between it executing at most once and it
being an iteration.  In C we commonly use conditionals ("if") for that.
Calling it "if_test" would be misleading with "for" under the hood,
though, as you cannot have an "else" branch -- which we don't need
anyway.

That said, iterating zero or one times is possible and understandable
not too much of a stretch IMHO.  I can understand that one needs a
minute to get used to that new keyword, though.

René

Re: [PATCH v2 2/6] unit-tests: add for_test

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

On Tue, Jul 23, 2024 at 02:37:35PM +0200, René Scharfe wrote:
> Am 23.07.24 um 11:53 schrieb Patrick Steinhardt:
> > On Tue, Jul 23, 2024 at 11:25:29AM +0200, René Scharfe wrote:
> >> Am 23.07.24 um 08:36 schrieb Patrick Steinhardt:
> >>> There is of course some magic involved with how we generate the file.
> >>
> >> It requires magic function names and generates code using a different
> >> language, while for_test is a just single new control flow keyword,
> >> like the dozen or so we already have.  So the magic employed by the
> >> libgit2 system is both broader and deeper.
> >
> > It is broader, that's certainly true. But it feels more self-contained,
> > less fragile and easier to read to me compared to macros.
> 
> In which ways can for_test break?

I was mostly referring to the potential for empty-by-accident test
bodies that were mentioned in other parts of this thread.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 2/6] unit-tests: add for_test

[Phillip Wood]

On 23/07/2024 13:37, René Scharfe wrote:
> Am 23.07.24 um 11:53 schrieb Patrick Steinhardt:
>> On Tue, Jul 23, 2024 at 11:25:29AM +0200, René Scharfe wrote:
>>> Am 23.07.24 um 08:36 schrieb Patrick Steinhardt:
>>>> There is of course some magic involved with how we generate the file.
>>>
>>> It requires magic function names and generates code using a different
>>> language, while for_test is a just single new control flow keyword,
>>> like the dozen or so we already have.  So the magic employed by the
>>> libgit2 system is both broader and deeper.
>>
>> It is broader, that's certainly true. But it feels more self-contained,
>> less fragile and easier to read to me compared to macros.
> 
> In which ways can for_test break?

Using a "break" statement to exit the test early will exit the loop 
without calling test__run_end()

Best Wishes

Phillip

> 
>>>> But I think that would be quite manageable, and ultimately all that the
>>>> developer would need to care about is writing a `test_foo_something()`
>>>> function. Everything else would be handled by our infra.
>>>
>>> With for_test all the developer has to do is write a test with a
>>> description, no extra infrastructure beyond the existing unit test
>>> framework needed.
>>
>> True, but it feels like it is an invitation for writing unidiomatic
>> code to me. Unidiomatic in this context to me mostly means code that is
>> not self-contained and thus cannot be run standalone.
> 
> Partly this is intentional -- I want to do something that the current
> idiom (the TEST macro) doesn't allow.
> 
> It's possible that reducing friction will cause more sloppy tests to
> be created.  I hope that the bad parts will stand out more when
> there's less boilerplate code.
> 
> René
> 

Re: [PATCH v2 6/6] t-strbuf: use for_test

[Phillip Wood]

Hi René

On 21/07/2024 07:26, René Scharfe wrote:

I'm still ambivalent about patches 3-6 but if we do this can we visually 
separate the initialization and cleanup from the main test body please. 
For example

> +	for_test ("strbuf_addch adds char") {
> +		struct strbuf sb = STRBUF_INIT;
+
> +		t_addch(&sb, 'a');
+
> +		t_release(&sb);
> +	}

Best Wishes

Phillip

Re: [PATCH v2 2/6] unit-tests: add for_test

[Phillip Wood]

On 22/07/2024 20:13, Kyle Lippincott wrote:
> On Sat, Jul 20, 2024 at 11:22 PM René Scharfe <l.s.r@web.de> wrote:
>>
>> The macro TEST only allows defining a test that consists of a single
>> expression.  Add a new macro, for_test, which provides a way to define
>> unit tests that are made up of one or more statements.
>>
>> for_test allows defining self-contained tests en bloc, a bit like
>> test_expect_success does for regular tests.  It acts like a for loop
>> that runs at most once; the test body is executed if test_skip_all()
>> had not been called before.
> 
> I can see based on this description where the name came from, but
> without this context, it's not clear when reading a test what it
> actually does. The name comes from an implementation detail, and is
> not describing what it _does_, just _how_ it does it.

That's my feeling too. In fact I think the name "for_test" actually 
works against us by implicitly suggesting that "break" can be used to 
exit the test early when in fact that will not work because we'll skip 
the call to test__run_end()

>> + * Run a test unless test_skip_all() has been called.  Acts like a for
>> + * loop that runs at most once, with the test description between the
>> + * parentheses and the test body as a statement or block after them.
>> + * The description for each test should be unique.  E.g.:
>> + *
>> + *  for_test ("something else %d %d", arg1, arg2) {
>> + *          prepare();
>> + *          test_something_else(arg1, arg2);
>> + *          cleanup();
>> + *  }
>> + */
>> +#define for_test(...)                                                  \
>> +       for (int for_test_running_ = test__run_begin() ?                \
>> +               (test__run_end(0, TEST_LOCATION(), __VA_ARGS__), 0) : 1;\
>> +            for_test_running_;                                         \
>> +            test__run_end(1, TEST_LOCATION(), __VA_ARGS__),            \
>> +               for_test_running_ = 0)
> 
> IMHO: this is borderline "too much magic" for my tastes.

Yes, compared to TEST_RUN() in v1 the magic level has jumped from Muggle 
to Dumbledore. Using a for loop is clever as it ensures test__run_end() 
is called at the end of the test without the need for separate 
TEST_END()[1] macro. The alternative is something like 
test_if_enabled()[2] which was discussed in the last round.

[1] 
https://lore.kernel.org/git/4f41f509-1d44-4476-92b0-9bb643f64576@gmail.com
[2] https://lore.kernel.org/git/xmqqa5iot01s.fsf@gitster.g

> I think
> having multiple test functions is generally easier to understand, and
> the overhead isn't really relevant. It's not _as_ compact in the
> source file, and requires that we have both the TEST statement and the
> function (and forgetting the TEST statement means that we won't invoke
> the function). If that is the main issue we're facing here, I wonder
> if there's some other way of resolving that (such as unused function
> detection via some compiler flags; even if it's not detected on all
> platforms, getting at least one of the CI platforms should be
> sufficient to prevent the issue [but we should target as many as
> possible, so it's caught earlier than "I tried to send it to the
> list"])

I also have a preference for function based tests but I do think 
something like for_test() can be useful in certain situations. For 
example in test-ctype.c where testing the ctype macros leads to a lot of 
repetition or a giant macro with a function based approach. If isalpha() 
and friends were functions instead we could write a single helper 
function which is passed the function under test together with the input 
data and expect result. Because they are macros that approach does not work.

Another example is where we are using a helper function with several 
inputs and we would prefer to write

for_test("test 1") {
	int input[] = ...
	int expect[] = ...

	test_helper(input, expect);

...

for_test("test 10") {
	int input[] = ...
	int expect[] = ...

	test_helper(input, expect);
}

rather then declaring all the input and expect variables up front with

	int input_1 = ...
	int input_2 = ...
	...
	int expect_1 = ...
	int expect_2 = ...

	TEST(test_helper(input_1, expect_1), "test 1");
	...
	TEST(test_helper(input_10, expect_10), "test 10");

> If others agree that this is a good simplification for the people
> reading the test code (and hopefully for the test author), I'm fine
> with this going in (with a different name). I'm not trying to veto the
> concept.

That's my feeling too.

Best Wishes

Phillip

> 
>> +
>>   /*
>>    * Print a test plan, should be called before any tests. If the number
>>    * of tests is not known in advance test_done() will automatically
>> --
>> 2.45.2
>>
> 

Re: [PATCH v2] unit-tests: show location of checks outside of tests

[Phillip Wood]

On 23/07/2024 07:02, René Scharfe wrote:
> Checks outside of tests are caught at runtime and reported like this:
> 
>   Assertion failed: (ctx.running), function test_assert, file test-lib.c, line 267.
> 
> The assert() call aborts the unit test and doesn't reveal the location
> or even the type of the offending check, as test_assert() is called by
> all of them.
> 
> Handle it like the opposite case, a test without any checks: Don't
> abort, but report the location of the actual check, along with a message
> explaining the situation.  The output for example above becomes:
> 
>   # BUG: check outside of test at t/helper/test-example-tap.c:75
> 
> ... and the unit test program continues and indicates the error in its
> exit code at the end.

This is a nice idea and looks good

Thanks

Phillip

> Signed-off-by: René Scharfe <l.s.r@web.de>
> ---
> Changes since v1:
> - Set ctx.failed to report the mistake via exit code as well.
> 
>   t/unit-tests/test-lib.c | 7 ++++++-
>   1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c
> index 3c513ce59a..989dc758e6 100644
> --- a/t/unit-tests/test-lib.c
> +++ b/t/unit-tests/test-lib.c
> @@ -264,7 +264,12 @@ static void test_todo(void)
> 
>   int test_assert(const char *location, const char *check, int ok)
>   {
> -	assert(ctx.running);
> +	if (!ctx.running) {
> +		test_msg("BUG: check outside of test at %s",
> +			 make_relative(location));
> +		ctx.failed = 1;
> +		return 0;
> +	}
> 
>   	if (ctx.result == RESULT_SKIP) {
>   		test_msg("skipping check '%s' at %s", check,
> --
> 2.45.2
> 

Re: [PATCH v2 2/6] unit-tests: add for_test

[René Scharfe]

Am 23.07.24 um 15:23 schrieb Phillip Wood:
> On 23/07/2024 13:37, René Scharfe wrote:
>>
>> In which ways can for_test break?
>
> Using a "break" statement to exit the test early will exit the loop
> without calling test__run_end()
Good point!  That will only be caught at runtime by the asserts at the
start of test__run_begin() or test_done() depending on whether it was
the last test:

 Assertion failed: (!ctx.running), function test_done, file test-lib.c, line 128.

 Assertion failed: (!ctx.running), function test__run_begin, file test-lib.c, line 172.

These messages could be more helpful.  And the use of the unsupported
break with a first-class for-like keyword would yield a compiler error.
But at least it won't go unnoticed even with this limited look-alike.
Deserves a comment in test-lib.h.

René

Re: [PATCH v2 1/6] t0080: move expected output to a file

[Jeff King]

On Sun, Jul 21, 2024 at 08:15:48AM +0200, René Scharfe wrote:

> Provide the expected output of "test-tool example-tap" verbatim instead
> of as a here-doc, to avoid distractions due to quoting, variables
> containing quotes and indentation.

Just to bring our earlier conversation full circle, you can do now do
this with:

  test_expect_success 'TAP output from unit tests' - <<\EOT
	cat >expect <<-\EOF
	...
	EOF
  EOT

since those patches are in master. But I'm OK if you don't want to
bother with it.

-Peff

[PATCH v3 0/7] add and use for_test to simplify tests

[René Scharfe]

Changes since v2:
- use nested here-docs in t0080 to reduce quoting and while keeping it
  self-contained,
- include spin-off patch 4e1041dc39 (unit-tests: show location of checks
  outside of tests) from seen, but with a test,
- document incompatibility of for_test and break,
- add blank line between variable declarations and rest of strbuf test
  code to match the original style.

  t0080: use here-doc test body
  unit-tests: show location of checks outside of tests
  unit-tests: add for_test
  t-ctype: use for_test
  t-reftable-basics: use for_test
  t-strvec: use for_test
  t-strbuf: use for_test

 .clang-format                    |   2 +
 t/helper/test-example-tap.c      |  35 +++
 t/t0080-unit-test-output.sh      |  60 ++++--
 t/unit-tests/t-ctype.c           |   4 +-
 t/unit-tests/t-reftable-basics.c | 228 +++++++++-----------
 t/unit-tests/t-strbuf.c          | 127 +++++------
 t/unit-tests/t-strvec.c          | 356 ++++++++++++++-----------------
 t/unit-tests/test-lib.c          |   7 +-
 t/unit-tests/test-lib.h          |  20 ++
 9 files changed, 438 insertions(+), 401 deletions(-)

Range-Diff gegen v2:
1:  5faabaea54 < -:  ---------- t0080: move expected output to a file
-:  ---------- > 1:  9b919853df t0080: use here-doc test body
-:  ---------- > 2:  9cea2b43b9 unit-tests: show location of checks outside of tests
2:  d4f9fa0938 ! 3:  5ea7472d8a unit-tests: add for_test
    @@ t/helper/test-example-tap.c: int cmd__example_tap(int argc, const char **argv)
      	return test_done();
      }

    - ## t/t0080/expect ##
    -@@ t/t0080/expect: not ok 17 - messages from failing string and char comparison
    - # BUG: test has no checks at t/helper/test-example-tap.c:92
    - not ok 18 - test with no checks
    - ok 19 - test with no checks returns 0
    --1..19
    -+ok 20 - for_test passing test
    -+# check "1 == 2" failed at t/helper/test-example-tap.c:98
    -+#    left: 1
    -+#   right: 2
    -+not ok 21 - for_test failing test
    -+not ok 22 - for_test passing TEST_TODO() # TODO
    -+# todo check 'check(1)' succeeded at t/helper/test-example-tap.c:102
    -+not ok 23 - for_test failing TEST_TODO()
    -+# check "0" failed at t/helper/test-example-tap.c:104
    -+# skipping test - missing prerequisite
    -+# skipping check '1' at t/helper/test-example-tap.c:106
    -+ok 24 - for_test test_skip() # SKIP
    -+# skipping test - missing prerequisite
    -+ok 25 - for_test test_skip() inside TEST_TODO() # SKIP
    -+# check "0" failed at t/helper/test-example-tap.c:111
    -+not ok 26 - for_test TEST_TODO() after failing check
    -+# check "0" failed at t/helper/test-example-tap.c:117
    -+not ok 27 - for_test failing check after TEST_TODO()
    -+# check "!strcmp("\thello\\", "there\"\n")" failed at t/helper/test-example-tap.c:120
    -+#    left: "\011hello\\"
    -+#   right: "there\"\012"
    -+# check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:121
    -+#    left: "NULL"
    -+#   right: NULL
    -+# check "'a' == '\n'" failed at t/helper/test-example-tap.c:122
    -+#    left: 'a'
    -+#   right: '\012'
    -+# check "'\\' == '\''" failed at t/helper/test-example-tap.c:123
    -+#    left: '\\'
    -+#   right: '\''
    -+not ok 28 - for_test messages from failing string and char comparison
    -+# BUG: test has no checks at t/helper/test-example-tap.c:125
    -+not ok 29 - for_test test with no checks
    -+1..29
    + ## t/t0080-unit-test-output.sh ##
    +@@ t/t0080-unit-test-output.sh: test_expect_success 'TAP output from unit tests' - <<\EOT
    + 	# BUG: test has no checks at t/helper/test-example-tap.c:94
    + 	not ok 18 - test with no checks
    + 	ok 19 - test with no checks returns 0
    +-	1..19
    ++	ok 20 - for_test passing test
    ++	# check "1 == 2" failed at t/helper/test-example-tap.c:100
    ++	#    left: 1
    ++	#   right: 2
    ++	not ok 21 - for_test failing test
    ++	not ok 22 - for_test passing TEST_TODO() # TODO
    ++	# todo check 'check(1)' succeeded at t/helper/test-example-tap.c:104
    ++	not ok 23 - for_test failing TEST_TODO()
    ++	# check "0" failed at t/helper/test-example-tap.c:106
    ++	# skipping test - missing prerequisite
    ++	# skipping check '1' at t/helper/test-example-tap.c:108
    ++	ok 24 - for_test test_skip() # SKIP
    ++	# skipping test - missing prerequisite
    ++	ok 25 - for_test test_skip() inside TEST_TODO() # SKIP
    ++	# check "0" failed at t/helper/test-example-tap.c:113
    ++	not ok 26 - for_test TEST_TODO() after failing check
    ++	# check "0" failed at t/helper/test-example-tap.c:119
    ++	not ok 27 - for_test failing check after TEST_TODO()
    ++	# check "!strcmp("\thello\\\\", "there\"\n")" failed at t/helper/test-example-tap.c:122
    ++	#    left: "\011hello\\\\"
    ++	#   right: "there\"\012"
    ++	# check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:123
    ++	#    left: "NULL"
    ++	#   right: NULL
    ++	# check "'a' == '\n'" failed at t/helper/test-example-tap.c:124
    ++	#    left: 'a'
    ++	#   right: '\012'
    ++	# check "'\\\\' == '\\''" failed at t/helper/test-example-tap.c:125
    ++	#    left: '\\\\'
    ++	#   right: '\\''
    ++	not ok 28 - for_test messages from failing string and char comparison
    ++	# BUG: test has no checks at t/helper/test-example-tap.c:127
    ++	not ok 29 - for_test test with no checks
    ++	1..29
    + 	EOF
    +
    + 	! test-tool example-tap >actual &&

      ## t/unit-tests/test-lib.h ##
     @@
    @@ t/unit-tests/test-lib.h
     + * Run a test unless test_skip_all() has been called.  Acts like a for
     + * loop that runs at most once, with the test description between the
     + * parentheses and the test body as a statement or block after them.
    -+ * The description for each test should be unique.  E.g.:
    ++ * Supports continue to end the test early, but not break. The
    ++ * description for each test should be unique.  E.g.:
     + *
     + *  for_test ("something else %d %d", arg1, arg2) {
     + *          prepare();
3:  a7cd5a2a3a = 4:  cf2fa74e9c t-ctype: use for_test
4:  cc07910f88 = 5:  f2e6271124 t-reftable-basics: use for_test
5:  11c1675a13 = 6:  c87a17189c t-strvec: use for_test
6:  cd79132f95 ! 7:  beaf8194fd t-strbuf: use for_test
    @@ t/unit-tests/t-strbuf.c: static void t_addstr(struct strbuf *buf, const void *da
     +
     +	for_test ("strbuf_addch adds char") {
     +		struct strbuf sb = STRBUF_INIT;
    ++
     +		t_addch(&sb, 'a');
     +		t_release(&sb);
     +	}
     +
     +	for_test ("strbuf_addch adds NUL char") {
     +		struct strbuf sb = STRBUF_INIT;
    ++
     +		t_addch(&sb, '\0');
     +		t_release(&sb);
     +	}
     +
     +	for_test ("strbuf_addch appends to initial value") {
     +		struct strbuf sb = STRBUF_INIT;
    ++
     +		t_addstr(&sb, "initial value");
     +		t_addch(&sb, 'a');
     +		t_release(&sb);
    @@ t/unit-tests/t-strbuf.c: static void t_addstr(struct strbuf *buf, const void *da
     +
     +	for_test ("strbuf_addstr adds string") {
     +		struct strbuf sb = STRBUF_INIT;
    ++
     +		t_addstr(&sb, "hello there");
     +		t_release(&sb);
     +	}
     +
     +	for_test ("strbuf_addstr appends string to initial value") {
     +		struct strbuf sb = STRBUF_INIT;
    ++
     +		t_addstr(&sb, "initial value");
     +		t_addstr(&sb, "hello there");
     +		t_release(&sb);
7:  83dafb009d < -:  ---------- unit-tests: show location of checks outside of tests
--
2.45.2

[PATCH v3 1/7] t0080: use here-doc test body

[René Scharfe]

Improve the readability of the expected output by using a here-doc for
the test body and replacing the unwieldy ${SQ} references with literal
single quotes.

Suggested-by: Jeff King <peff@peff.net>
Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/t0080-unit-test-output.sh | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh
index 7bbb065d58..9ec47b7360 100755
--- a/t/t0080-unit-test-output.sh
+++ b/t/t0080-unit-test-output.sh
@@ -5,7 +5,7 @@ test_description='Test the output of the unit test framework'
 TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh

-test_expect_success 'TAP output from unit tests' '
+test_expect_success 'TAP output from unit tests' - <<\EOT
 	cat >expect <<-EOF &&
 	ok 1 - passing test
 	ok 2 - passing test and assertion return 1
@@ -16,12 +16,12 @@ test_expect_success 'TAP output from unit tests' '
 	ok 4 - failing test and assertion return 0
 	not ok 5 - passing TEST_TODO() # TODO
 	ok 6 - passing TEST_TODO() returns 1
-	# todo check ${SQ}check(x)${SQ} succeeded at t/helper/test-example-tap.c:26
+	# todo check 'check(x)' succeeded at t/helper/test-example-tap.c:26
 	not ok 7 - failing TEST_TODO()
 	ok 8 - failing TEST_TODO() returns 0
 	# check "0" failed at t/helper/test-example-tap.c:31
 	# skipping test - missing prerequisite
-	# skipping check ${SQ}1${SQ} at t/helper/test-example-tap.c:33
+	# skipping check '1' at t/helper/test-example-tap.c:33
 	ok 9 - test_skip() # SKIP
 	ok 10 - skipped test returns 1
 	# skipping test - missing prerequisite
@@ -39,12 +39,12 @@ test_expect_success 'TAP output from unit tests' '
 	# check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:63
 	#    left: "NULL"
 	#   right: NULL
-	# check "${SQ}a${SQ} == ${SQ}\n${SQ}" failed at t/helper/test-example-tap.c:64
-	#    left: ${SQ}a${SQ}
-	#   right: ${SQ}\012${SQ}
-	# check "${SQ}\\\\${SQ} == ${SQ}\\${SQ}${SQ}" failed at t/helper/test-example-tap.c:65
-	#    left: ${SQ}\\\\${SQ}
-	#   right: ${SQ}\\${SQ}${SQ}
+	# check "'a' == '\n'" failed at t/helper/test-example-tap.c:64
+	#    left: 'a'
+	#   right: '\012'
+	# check "'\\\\' == '\\''" failed at t/helper/test-example-tap.c:65
+	#    left: '\\\\'
+	#   right: '\\''
 	not ok 17 - messages from failing string and char comparison
 	# BUG: test has no checks at t/helper/test-example-tap.c:92
 	not ok 18 - test with no checks
@@ -54,6 +54,6 @@ test_expect_success 'TAP output from unit tests' '

 	! test-tool example-tap >actual &&
 	test_cmp expect actual
-'
+EOT

 test_done
--
2.45.2

[PATCH v3 2/7] unit-tests: show location of checks outside of tests

[René Scharfe]

Checks outside of tests are caught at runtime and reported like this:

 Assertion failed: (ctx.running), function test_assert, file test-lib.c, line 267.

The assert() call aborts the unit test and doesn't reveal the location
or even the type of the offending check, as test_assert() is called by
all of them.

Handle it like the opposite case, a test without any checks: Don't
abort, but report the location of the actual check, along with a message
explaining the situation.  The output for example above becomes:

 # BUG: check outside of test at t/helper/test-example-tap.c:75

... and the unit test program continues and indicates the error in its
exit code at the end.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
Supersedes 9cea2b43b9 (unit-tests: show location of checks outside of
tests, 2024-07-22) in seen.  Changes: Got a test.

 t/helper/test-example-tap.c | 2 ++
 t/t0080-unit-test-output.sh | 5 +++--
 t/unit-tests/test-lib.c     | 7 ++++++-
 3 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/t/helper/test-example-tap.c b/t/helper/test-example-tap.c
index d072ad559f..79c12b01cd 100644
--- a/t/helper/test-example-tap.c
+++ b/t/helper/test-example-tap.c
@@ -72,6 +72,8 @@ static void t_empty(void)

 int cmd__example_tap(int argc, const char **argv)
 {
+	check(1);
+
 	test_res = TEST(check_res = check_int(1, ==, 1), "passing test");
 	TEST(t_res(1), "passing test and assertion return 1");
 	test_res = TEST(check_res = check_int(1, ==, 2), "failing test");
diff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh
index 9ec47b7360..fe221f3bdb 100755
--- a/t/t0080-unit-test-output.sh
+++ b/t/t0080-unit-test-output.sh
@@ -7,9 +7,10 @@ TEST_PASSES_SANITIZE_LEAK=true

 test_expect_success 'TAP output from unit tests' - <<\EOT
 	cat >expect <<-EOF &&
+	# BUG: check outside of test at t/helper/test-example-tap.c:75
 	ok 1 - passing test
 	ok 2 - passing test and assertion return 1
-	# check "1 == 2" failed at t/helper/test-example-tap.c:77
+	# check "1 == 2" failed at t/helper/test-example-tap.c:79
 	#    left: 1
 	#   right: 2
 	not ok 3 - failing test
@@ -46,7 +47,7 @@ test_expect_success 'TAP output from unit tests' - <<\EOT
 	#    left: '\\\\'
 	#   right: '\\''
 	not ok 17 - messages from failing string and char comparison
-	# BUG: test has no checks at t/helper/test-example-tap.c:92
+	# BUG: test has no checks at t/helper/test-example-tap.c:94
 	not ok 18 - test with no checks
 	ok 19 - test with no checks returns 0
 	1..19
diff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c
index 3c513ce59a..989dc758e6 100644
--- a/t/unit-tests/test-lib.c
+++ b/t/unit-tests/test-lib.c
@@ -264,7 +264,12 @@ static void test_todo(void)

 int test_assert(const char *location, const char *check, int ok)
 {
-	assert(ctx.running);
+	if (!ctx.running) {
+		test_msg("BUG: check outside of test at %s",
+			 make_relative(location));
+		ctx.failed = 1;
+		return 0;
+	}

 	if (ctx.result == RESULT_SKIP) {
 		test_msg("skipping check '%s' at %s", check,
--
2.45.2

[PATCH v3 3/7] unit-tests: add for_test

[René Scharfe]

The macro TEST only allows defining a test that consists of a single
expression.  Add a new macro, for_test, which provides a way to define
unit tests that are made up of one or more statements.

for_test allows defining self-contained tests en bloc, a bit like
test_expect_success does for regular tests.  It acts like a for loop
that runs at most once; the test body is executed if test_skip_all()
had not been called before.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 .clang-format               |  2 ++
 t/helper/test-example-tap.c | 33 +++++++++++++++++++++++++++++++++
 t/t0080-unit-test-output.sh | 35 ++++++++++++++++++++++++++++++++++-
 t/unit-tests/test-lib.h     | 20 ++++++++++++++++++++
 4 files changed, 89 insertions(+), 1 deletion(-)

diff --git a/.clang-format b/.clang-format
index 6408251577..863dc87dfc 100644
--- a/.clang-format
+++ b/.clang-format
@@ -151,6 +151,7 @@ Cpp11BracedListStyle: false
 # function calls. Taken from:
 #   git grep -h '^#define [^[:space:]]*for_\?each[^[:space:]]*(' |
 #   sed "s/^#define /  - '/; s/(.*$/'/" | sort | uniq
+# Added for_test from t/unit-tests/test-lib.h manually as a special case.
 ForEachMacros:
   - 'for_each_builtin'
   - 'for_each_string_list_item'
@@ -168,6 +169,7 @@ ForEachMacros:
   - 'strintmap_for_each_entry'
   - 'strmap_for_each_entry'
   - 'strset_for_each_entry'
+  - 'for_test'

 # The maximum number of consecutive empty lines to keep.
 MaxEmptyLinesToKeep: 1
diff --git a/t/helper/test-example-tap.c b/t/helper/test-example-tap.c
index 79c12b01cd..5e49fb1e7e 100644
--- a/t/helper/test-example-tap.c
+++ b/t/helper/test-example-tap.c
@@ -94,5 +94,38 @@ int cmd__example_tap(int argc, const char **argv)
 	test_res = TEST(t_empty(), "test with no checks");
 	TEST(check_int(test_res, ==, 0), "test with no checks returns 0");

+	for_test ("for_test passing test")
+		check_int(1, ==, 1);
+	for_test ("for_test failing test")
+		check_int(1, ==, 2);
+	for_test ("for_test passing TEST_TODO()")
+		TEST_TODO(check(0));
+	for_test ("for_test failing TEST_TODO()")
+		TEST_TODO(check(1));
+	for_test ("for_test test_skip()") {
+		check(0);
+		test_skip("missing prerequisite");
+		check(1);
+	}
+	for_test ("for_test test_skip() inside TEST_TODO()")
+		TEST_TODO((test_skip("missing prerequisite"), 1));
+	for_test ("for_test TEST_TODO() after failing check") {
+		check(0);
+		TEST_TODO(check(0));
+	}
+	for_test ("for_test failing check after TEST_TODO()") {
+		check(1);
+		TEST_TODO(check(0));
+		check(0);
+	}
+	for_test ("for_test messages from failing string and char comparison") {
+		check_str("\thello\\", "there\"\n");
+		check_str("NULL", NULL);
+		check_char('a', ==, '\n');
+		check_char('\\', ==, '\'');
+	}
+	for_test ("for_test test with no checks")
+		; /* nothing */
+
 	return test_done();
 }
diff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh
index fe221f3bdb..5185154414 100755
--- a/t/t0080-unit-test-output.sh
+++ b/t/t0080-unit-test-output.sh
@@ -50,7 +50,40 @@ test_expect_success 'TAP output from unit tests' - <<\EOT
 	# BUG: test has no checks at t/helper/test-example-tap.c:94
 	not ok 18 - test with no checks
 	ok 19 - test with no checks returns 0
-	1..19
+	ok 20 - for_test passing test
+	# check "1 == 2" failed at t/helper/test-example-tap.c:100
+	#    left: 1
+	#   right: 2
+	not ok 21 - for_test failing test
+	not ok 22 - for_test passing TEST_TODO() # TODO
+	# todo check 'check(1)' succeeded at t/helper/test-example-tap.c:104
+	not ok 23 - for_test failing TEST_TODO()
+	# check "0" failed at t/helper/test-example-tap.c:106
+	# skipping test - missing prerequisite
+	# skipping check '1' at t/helper/test-example-tap.c:108
+	ok 24 - for_test test_skip() # SKIP
+	# skipping test - missing prerequisite
+	ok 25 - for_test test_skip() inside TEST_TODO() # SKIP
+	# check "0" failed at t/helper/test-example-tap.c:113
+	not ok 26 - for_test TEST_TODO() after failing check
+	# check "0" failed at t/helper/test-example-tap.c:119
+	not ok 27 - for_test failing check after TEST_TODO()
+	# check "!strcmp("\thello\\\\", "there\"\n")" failed at t/helper/test-example-tap.c:122
+	#    left: "\011hello\\\\"
+	#   right: "there\"\012"
+	# check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:123
+	#    left: "NULL"
+	#   right: NULL
+	# check "'a' == '\n'" failed at t/helper/test-example-tap.c:124
+	#    left: 'a'
+	#   right: '\012'
+	# check "'\\\\' == '\\''" failed at t/helper/test-example-tap.c:125
+	#    left: '\\\\'
+	#   right: '\\''
+	not ok 28 - for_test messages from failing string and char comparison
+	# BUG: test has no checks at t/helper/test-example-tap.c:127
+	not ok 29 - for_test test with no checks
+	1..29
 	EOF

 	! test-tool example-tap >actual &&
diff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h
index 2de6d715d5..598c6ff9f3 100644
--- a/t/unit-tests/test-lib.h
+++ b/t/unit-tests/test-lib.h
@@ -14,6 +14,26 @@
 	test__run_end(test__run_begin() ? 0 : (t, 1),	\
 		      TEST_LOCATION(),  __VA_ARGS__)

+/*
+ * Run a test unless test_skip_all() has been called.  Acts like a for
+ * loop that runs at most once, with the test description between the
+ * parentheses and the test body as a statement or block after them.
+ * Supports continue to end the test early, but not break. The
+ * description for each test should be unique.  E.g.:
+ *
+ *  for_test ("something else %d %d", arg1, arg2) {
+ *          prepare();
+ *          test_something_else(arg1, arg2);
+ *          cleanup();
+ *  }
+ */
+#define for_test(...)							\
+	for (int for_test_running_ = test__run_begin() ?		\
+		(test__run_end(0, TEST_LOCATION(), __VA_ARGS__), 0) : 1;\
+	     for_test_running_;						\
+	     test__run_end(1, TEST_LOCATION(), __VA_ARGS__),		\
+		for_test_running_ = 0)
+
 /*
  * Print a test plan, should be called before any tests. If the number
  * of tests is not known in advance test_done() will automatically
--
2.45.2

[PATCH v3 4/7] t-ctype: use for_test

[René Scharfe]

Use the documented macro for_test instead of the internal functions
test__run_begin() and test__run_end(), which are supposed to be private
to the unit test framework.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-ctype.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/t/unit-tests/t-ctype.c b/t/unit-tests/t-ctype.c
index d6ac1fe678..92a05f02b3 100644
--- a/t/unit-tests/t-ctype.c
+++ b/t/unit-tests/t-ctype.c
@@ -4,15 +4,13 @@
 	size_t len = ARRAY_SIZE(string) - 1 + \
 		BUILD_ASSERT_OR_ZERO(ARRAY_SIZE(string) > 0) + \
 		BUILD_ASSERT_OR_ZERO(sizeof(string[0]) == sizeof(char)); \
-	int skip = test__run_begin(); \
-	if (!skip) { \
+	for_test (#class " works") { \
 		for (int i = 0; i < 256; i++) { \
 			if (!check_int(class(i), ==, !!memchr(string, i, len)))\
 				test_msg("      i: 0x%02x", i); \
 		} \
 		check(!class(EOF)); \
 	} \
-	test__run_end(!skip, TEST_LOCATION(), #class " works"); \
 } while (0)

 #define DIGIT "0123456789"
--
2.45.2

[PATCH v3 5/7] t-reftable-basics: use for_test

[René Scharfe]

The macro TEST takes a single expression.  If a test requires multiple
statements then they need to be placed in a function that's called in
the TEST expression.

Remove the overhead of defining and calling single-use functions by
using for_test instead.

Run the tests in the order of definition.  We can reorder them like that
because they are independent.  Technically this changes the output, but
retains the meaning of a full run and allows for easier review e.g. with
diff option --ignore-all-space.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-reftable-basics.c | 228 ++++++++++++++-----------------
 1 file changed, 106 insertions(+), 122 deletions(-)

diff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c
index 4e80bdf16d..eb7af9ade4 100644
--- a/t/unit-tests/t-reftable-basics.c
+++ b/t/unit-tests/t-reftable-basics.c
@@ -20,141 +20,125 @@ static int integer_needle_lesseq(size_t i, void *_args)
 	return args->needle <= args->haystack[i];
 }

-static void test_binsearch(void)
+int cmd_main(int argc, const char *argv[])
 {
-	int haystack[] = { 2, 4, 6, 8, 10 };
-	struct {
-		int needle;
-		size_t expected_idx;
-	} testcases[] = {
-		{-9000, 0},
-		{-1, 0},
-		{0, 0},
-		{2, 0},
-		{3, 1},
-		{4, 1},
-		{7, 3},
-		{9, 4},
-		{10, 4},
-		{11, 5},
-		{9000, 5},
-	};
-
-	for (size_t i = 0; i < ARRAY_SIZE(testcases); i++) {
-		struct integer_needle_lesseq_args args = {
-			.haystack = haystack,
-			.needle = testcases[i].needle,
+	for_test ("binary search with binsearch works") {
+		int haystack[] = { 2, 4, 6, 8, 10 };
+		struct {
+			int needle;
+			size_t expected_idx;
+		} testcases[] = {
+			{-9000, 0},
+			{-1, 0},
+			{0, 0},
+			{2, 0},
+			{3, 1},
+			{4, 1},
+			{7, 3},
+			{9, 4},
+			{10, 4},
+			{11, 5},
+			{9000, 5},
 		};
-		size_t idx;

-		idx = binsearch(ARRAY_SIZE(haystack), &integer_needle_lesseq, &args);
-		check_int(idx, ==, testcases[i].expected_idx);
+		for (size_t i = 0; i < ARRAY_SIZE(testcases); i++) {
+			struct integer_needle_lesseq_args args = {
+				.haystack = haystack,
+				.needle = testcases[i].needle,
+			};
+			size_t idx;
+
+			idx = binsearch(ARRAY_SIZE(haystack),
+					&integer_needle_lesseq, &args);
+			check_int(idx, ==, testcases[i].expected_idx);
+		}
 	}
-}

-static void test_names_length(void)
-{
-	const char *a[] = { "a", "b", NULL };
-	check_int(names_length(a), ==, 2);
-}
-
-static void test_names_equal(void)
-{
-	const char *a[] = { "a", "b", "c", NULL };
-	const char *b[] = { "a", "b", "d", NULL };
-	const char *c[] = { "a", "b", NULL };
+	for_test ("names_length retuns size of a NULL-terminated string array") {
+		const char *a[] = { "a", "b", NULL };
+		check_int(names_length(a), ==, 2);
+	}

-	check(names_equal(a, a));
-	check(!names_equal(a, b));
-	check(!names_equal(a, c));
-}
+	for_test ("names_equal compares NULL-terminated string arrays") {
+		const char *a[] = { "a", "b", "c", NULL };
+		const char *b[] = { "a", "b", "d", NULL };
+		const char *c[] = { "a", "b", NULL };

-static void test_parse_names_normal(void)
-{
-	char in1[] = "line\n";
-	char in2[] = "a\nb\nc";
-	char **out = NULL;
-	parse_names(in1, strlen(in1), &out);
-	check_str(out[0], "line");
-	check(!out[1]);
-	free_names(out);
-
-	parse_names(in2, strlen(in2), &out);
-	check_str(out[0], "a");
-	check_str(out[1], "b");
-	check_str(out[2], "c");
-	check(!out[3]);
-	free_names(out);
-}
+		check(names_equal(a, a));
+		check(!names_equal(a, b));
+		check(!names_equal(a, c));
+	}

-static void test_parse_names_drop_empty(void)
-{
-	char in[] = "a\n\nb\n";
-	char **out = NULL;
-	parse_names(in, strlen(in), &out);
-	check_str(out[0], "a");
-	/* simply '\n' should be dropped as empty string */
-	check_str(out[1], "b");
-	check(!out[2]);
-	free_names(out);
-}
+	for_test ("parse_names works for basic input") {
+		char in1[] = "line\n";
+		char in2[] = "a\nb\nc";
+		char **out = NULL;
+		parse_names(in1, strlen(in1), &out);
+		check_str(out[0], "line");
+		check(!out[1]);
+		free_names(out);
+
+		parse_names(in2, strlen(in2), &out);
+		check_str(out[0], "a");
+		check_str(out[1], "b");
+		check_str(out[2], "c");
+		check(!out[3]);
+		free_names(out);
+	}

-static void test_common_prefix(void)
-{
-	struct strbuf a = STRBUF_INIT;
-	struct strbuf b = STRBUF_INIT;
-	struct {
-		const char *a, *b;
-		int want;
-	} cases[] = {
-		{"abcdef", "abc", 3},
-		{ "abc", "ab", 2 },
-		{ "", "abc", 0 },
-		{ "abc", "abd", 2 },
-		{ "abc", "pqr", 0 },
-	};
-
-	for (size_t i = 0; i < ARRAY_SIZE(cases); i++) {
-		strbuf_addstr(&a, cases[i].a);
-		strbuf_addstr(&b, cases[i].b);
-		check_int(common_prefix_size(&a, &b), ==, cases[i].want);
-		strbuf_reset(&a);
-		strbuf_reset(&b);
+	for_test ("parse_names drops empty string") {
+		char in[] = "a\n\nb\n";
+		char **out = NULL;
+		parse_names(in, strlen(in), &out);
+		check_str(out[0], "a");
+		/* simply '\n' should be dropped as empty string */
+		check_str(out[1], "b");
+		check(!out[2]);
+		free_names(out);
 	}
-	strbuf_release(&a);
-	strbuf_release(&b);
-}

-static void test_u24_roundtrip(void)
-{
-	uint32_t in = 0x112233;
-	uint8_t dest[3];
-	uint32_t out;
-	put_be24(dest, in);
-	out = get_be24(dest);
-	check_int(in, ==, out);
-}
+	for_test ("common_prefix_size works") {
+		struct strbuf a = STRBUF_INIT;
+		struct strbuf b = STRBUF_INIT;
+		struct {
+			const char *a, *b;
+			int want;
+		} cases[] = {
+			{"abcdef", "abc", 3},
+			{ "abc", "ab", 2 },
+			{ "", "abc", 0 },
+			{ "abc", "abd", 2 },
+			{ "abc", "pqr", 0 },
+		};

-static void test_u16_roundtrip(void)
-{
-	uint32_t in = 0xfef1;
-	uint8_t dest[3];
-	uint32_t out;
-	put_be16(dest, in);
-	out = get_be16(dest);
-	check_int(in, ==, out);
-}
+		for (size_t i = 0; i < ARRAY_SIZE(cases); i++) {
+			strbuf_addstr(&a, cases[i].a);
+			strbuf_addstr(&b, cases[i].b);
+			check_int(common_prefix_size(&a, &b), ==, cases[i].want);
+			strbuf_reset(&a);
+			strbuf_reset(&b);
+		}
+		strbuf_release(&a);
+		strbuf_release(&b);
+	}

-int cmd_main(int argc, const char *argv[])
-{
-	TEST(test_common_prefix(), "common_prefix_size works");
-	TEST(test_parse_names_normal(), "parse_names works for basic input");
-	TEST(test_parse_names_drop_empty(), "parse_names drops empty string");
-	TEST(test_binsearch(), "binary search with binsearch works");
-	TEST(test_names_length(), "names_length retuns size of a NULL-terminated string array");
-	TEST(test_names_equal(), "names_equal compares NULL-terminated string arrays");
-	TEST(test_u24_roundtrip(), "put_be24 and get_be24 work");
-	TEST(test_u16_roundtrip(), "put_be16 and get_be16 work");
+	for_test ("put_be24 and get_be24 work") {
+		uint32_t in = 0x112233;
+		uint8_t dest[3];
+		uint32_t out;
+		put_be24(dest, in);
+		out = get_be24(dest);
+		check_int(in, ==, out);
+	}
+
+	for_test ("put_be16 and get_be16 work") {
+		uint32_t in = 0xfef1;
+		uint8_t dest[3];
+		uint32_t out;
+		put_be16(dest, in);
+		out = get_be16(dest);
+		check_int(in, ==, out);
+	}

 	return test_done();
 }
--
2.45.2

[PATCH v3 6/7] t-strvec: use for_test

[René Scharfe]

The macro TEST takes a single expression.  If a test requires multiple
statements then they need to be placed in a function that's called in
the TEST expression.

Remove the cognitive overhead of defining and calling single-use
functions by using for_test instead.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-strvec.c | 356 ++++++++++++++++++----------------------
 1 file changed, 156 insertions(+), 200 deletions(-)

diff --git a/t/unit-tests/t-strvec.c b/t/unit-tests/t-strvec.c
index d4615ab06d..4fa2c21afb 100644
--- a/t/unit-tests/t-strvec.c
+++ b/t/unit-tests/t-strvec.c
@@ -36,237 +36,193 @@ static void check_strvec_loc(const char *loc, struct strvec *vec, ...)
 	check_pointer_eq(vec->v[nr], NULL);
 }

-static void t_static_init(void)
+int cmd_main(int argc, const char **argv)
 {
-	struct strvec vec = STRVEC_INIT;
-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
-}
+	for_test ("static initialization") {
+		struct strvec vec = STRVEC_INIT;
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);
+	}

-static void t_dynamic_init(void)
-{
-	struct strvec vec;
-	strvec_init(&vec);
-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
-}
+	for_test ("dynamic initialization") {
+		struct strvec vec;
+		strvec_init(&vec);
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);
+	}

-static void t_clear(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_push(&vec, "foo");
-	strvec_clear(&vec);
-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
-}
+	for_test ("clear") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_push(&vec, "foo");
+		strvec_clear(&vec);
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);
+	}

-static void t_push(void)
-{
-	struct strvec vec = STRVEC_INIT;
+	for_test ("push") {
+		struct strvec vec = STRVEC_INIT;

-	strvec_push(&vec, "foo");
-	check_strvec(&vec, "foo", NULL);
+		strvec_push(&vec, "foo");
+		check_strvec(&vec, "foo", NULL);

-	strvec_push(&vec, "bar");
-	check_strvec(&vec, "foo", "bar", NULL);
+		strvec_push(&vec, "bar");
+		check_strvec(&vec, "foo", "bar", NULL);

-	strvec_clear(&vec);
-}
+		strvec_clear(&vec);
+	}

-static void t_pushf(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushf(&vec, "foo: %d", 1);
-	check_strvec(&vec, "foo: 1", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("pushf") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushf(&vec, "foo: %d", 1);
+		check_strvec(&vec, "foo: 1", NULL);
+		strvec_clear(&vec);
+	}

-static void t_pushl(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	check_strvec(&vec, "foo", "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("pushl") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		check_strvec(&vec, "foo", "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_pushv(void)
-{
-	const char *strings[] = {
-		"foo", "bar", "baz", NULL,
-	};
-	struct strvec vec = STRVEC_INIT;
+	for_test ("pushv") {
+		const char *strings[] = {
+			"foo", "bar", "baz", NULL,
+		};
+		struct strvec vec = STRVEC_INIT;

-	strvec_pushv(&vec, strings);
-	check_strvec(&vec, "foo", "bar", "baz", NULL);
+		strvec_pushv(&vec, strings);
+		check_strvec(&vec, "foo", "bar", "baz", NULL);

-	strvec_clear(&vec);
-}
+		strvec_clear(&vec);
+	}

-static void t_replace_at_head(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_replace(&vec, 0, "replaced");
-	check_strvec(&vec, "replaced", "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("replace at head") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_replace(&vec, 0, "replaced");
+		check_strvec(&vec, "replaced", "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_replace_at_tail(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_replace(&vec, 2, "replaced");
-	check_strvec(&vec, "foo", "bar", "replaced", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("replace at tail") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_replace(&vec, 2, "replaced");
+		check_strvec(&vec, "foo", "bar", "replaced", NULL);
+		strvec_clear(&vec);
+	}

-static void t_replace_in_between(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_replace(&vec, 1, "replaced");
-	check_strvec(&vec, "foo", "replaced", "baz", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("replace in between") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_replace(&vec, 1, "replaced");
+		check_strvec(&vec, "foo", "replaced", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_replace_with_substring(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", NULL);
-	strvec_replace(&vec, 0, vec.v[0] + 1);
-	check_strvec(&vec, "oo", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("replace with substring") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", NULL);
+		strvec_replace(&vec, 0, vec.v[0] + 1);
+		check_strvec(&vec, "oo", NULL);
+		strvec_clear(&vec);
+	}

-static void t_remove_at_head(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_remove(&vec, 0);
-	check_strvec(&vec, "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("remove at head") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_remove(&vec, 0);
+		check_strvec(&vec, "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_remove_at_tail(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_remove(&vec, 2);
-	check_strvec(&vec, "foo", "bar", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("remove at tail") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_remove(&vec, 2);
+		check_strvec(&vec, "foo", "bar", NULL);
+		strvec_clear(&vec);
+	}

-static void t_remove_in_between(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_remove(&vec, 1);
-	check_strvec(&vec, "foo", "baz", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("remove in between") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_remove(&vec, 1);
+		check_strvec(&vec, "foo", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_pop_empty_array(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pop(&vec);
-	check_strvec(&vec, NULL);
-	strvec_clear(&vec);
-}
+	for_test ("pop with empty array") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pop(&vec);
+		check_strvec(&vec, NULL);
+		strvec_clear(&vec);
+	}

-static void t_pop_non_empty_array(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_pop(&vec);
-	check_strvec(&vec, "foo", "bar", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("pop with non-empty array") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_pop(&vec);
+		check_strvec(&vec, "foo", "bar", NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_empty_string(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "");
-	check_strvec(&vec, NULL);
-	strvec_clear(&vec);
-}
+	for_test ("split empty string") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "");
+		check_strvec(&vec, NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_single_item(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "foo");
-	check_strvec(&vec, "foo", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("split single item") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "foo");
+		check_strvec(&vec, "foo", NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_multiple_items(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "foo bar baz");
-	check_strvec(&vec, "foo", "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("split multiple items") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "foo bar baz");
+		check_strvec(&vec, "foo", "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_whitespace_only(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, " \t\n");
-	check_strvec(&vec, NULL);
-	strvec_clear(&vec);
-}
+	for_test ("split whitespace only") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, " \t\n");
+		check_strvec(&vec, NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_multiple_consecutive_whitespaces(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "foo\n\t bar");
-	check_strvec(&vec, "foo", "bar", NULL);
-	strvec_clear(&vec);
-}
+	for_test ("split multiple consecutive whitespaces") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "foo\n\t bar");
+		check_strvec(&vec, "foo", "bar", NULL);
+		strvec_clear(&vec);
+	}

-static void t_detach(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	const char **detached;
+	for_test ("detach") {
+		struct strvec vec = STRVEC_INIT;
+		const char **detached;

-	strvec_push(&vec, "foo");
+		strvec_push(&vec, "foo");

-	detached = strvec_detach(&vec);
-	check_str(detached[0], "foo");
-	check_pointer_eq(detached[1], NULL);
+		detached = strvec_detach(&vec);
+		check_str(detached[0], "foo");
+		check_pointer_eq(detached[1], NULL);

-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);

-	free((char *) detached[0]);
-	free(detached);
-}
+		free((char *) detached[0]);
+		free(detached);
+	}

-int cmd_main(int argc, const char **argv)
-{
-	TEST(t_static_init(), "static initialization");
-	TEST(t_dynamic_init(), "dynamic initialization");
-	TEST(t_clear(), "clear");
-	TEST(t_push(), "push");
-	TEST(t_pushf(), "pushf");
-	TEST(t_pushl(), "pushl");
-	TEST(t_pushv(), "pushv");
-	TEST(t_replace_at_head(), "replace at head");
-	TEST(t_replace_in_between(), "replace in between");
-	TEST(t_replace_at_tail(), "replace at tail");
-	TEST(t_replace_with_substring(), "replace with substring");
-	TEST(t_remove_at_head(), "remove at head");
-	TEST(t_remove_in_between(), "remove in between");
-	TEST(t_remove_at_tail(), "remove at tail");
-	TEST(t_pop_empty_array(), "pop with empty array");
-	TEST(t_pop_non_empty_array(), "pop with non-empty array");
-	TEST(t_split_empty_string(), "split empty string");
-	TEST(t_split_single_item(), "split single item");
-	TEST(t_split_multiple_items(), "split multiple items");
-	TEST(t_split_whitespace_only(), "split whitespace only");
-	TEST(t_split_multiple_consecutive_whitespaces(), "split multiple consecutive whitespaces");
-	TEST(t_detach(), "detach");
 	return test_done();
 }
--
2.45.2

[PATCH v3 7/7] t-strbuf: use for_test

[René Scharfe]

The macro TEST takes a single expression.  If a test requires multiple
statements then they need to be placed in a function that's called in
the TEST expression.  The functions setup() and setup_populated() here
are used for that purpose and take another function as an argument,
making the control flow hard to follow.

Remove the overhead of these functions by using for_test instead.  Move
their duplicate post-condition checks into a new helper, t_release(),
and let t_addch() and t_addstr() accept properly typed input parameters
instead of void pointers.

Use the fully checking t_addstr() for adding initial values instead of
only doing only a length comparison -- there's no need for skipping the
other checks.

This results in test cases that look much more like strbuf usage in
production code, only with checked strbuf functions replaced by checking
wrappers.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-strbuf.c | 127 ++++++++++++++++++++--------------------
 1 file changed, 65 insertions(+), 62 deletions(-)

diff --git a/t/unit-tests/t-strbuf.c b/t/unit-tests/t-strbuf.c
index 6027dafef7..1e4a7885c9 100644
--- a/t/unit-tests/t-strbuf.c
+++ b/t/unit-tests/t-strbuf.c
@@ -1,32 +1,6 @@
 #include "test-lib.h"
 #include "strbuf.h"

-/* wrapper that supplies tests with an empty, initialized strbuf */
-static void setup(void (*f)(struct strbuf*, const void*),
-		  const void *data)
-{
-	struct strbuf buf = STRBUF_INIT;
-
-	f(&buf, data);
-	strbuf_release(&buf);
-	check_uint(buf.len, ==, 0);
-	check_uint(buf.alloc, ==, 0);
-}
-
-/* wrapper that supplies tests with a populated, initialized strbuf */
-static void setup_populated(void (*f)(struct strbuf*, const void*),
-			    const char *init_str, const void *data)
-{
-	struct strbuf buf = STRBUF_INIT;
-
-	strbuf_addstr(&buf, init_str);
-	check_uint(buf.len, ==, strlen(init_str));
-	f(&buf, data);
-	strbuf_release(&buf);
-	check_uint(buf.len, ==, 0);
-	check_uint(buf.alloc, ==, 0);
-}
-
 static int assert_sane_strbuf(struct strbuf *buf)
 {
 	/* Initialized strbufs should always have a non-NULL buffer */
@@ -45,31 +19,8 @@ static int assert_sane_strbuf(struct strbuf *buf)
 	return check_uint(buf->len, <, buf->alloc);
 }

-static void t_static_init(void)
+static void t_addch(struct strbuf *buf, int ch)
 {
-	struct strbuf buf = STRBUF_INIT;
-
-	check_uint(buf.len, ==, 0);
-	check_uint(buf.alloc, ==, 0);
-	check_char(buf.buf[0], ==, '\0');
-}
-
-static void t_dynamic_init(void)
-{
-	struct strbuf buf;
-
-	strbuf_init(&buf, 1024);
-	check(assert_sane_strbuf(&buf));
-	check_uint(buf.len, ==, 0);
-	check_uint(buf.alloc, >=, 1024);
-	check_char(buf.buf[0], ==, '\0');
-	strbuf_release(&buf);
-}
-
-static void t_addch(struct strbuf *buf, const void *data)
-{
-	const char *p_ch = data;
-	const char ch = *p_ch;
 	size_t orig_alloc = buf->alloc;
 	size_t orig_len = buf->len;

@@ -85,9 +36,8 @@ static void t_addch(struct strbuf *buf, const void *data)
 	check_char(buf->buf[buf->len], ==, '\0');
 }

-static void t_addstr(struct strbuf *buf, const void *data)
+static void t_addstr(struct strbuf *buf, const char *text)
 {
-	const char *text = data;
 	size_t len = strlen(text);
 	size_t orig_alloc = buf->alloc;
 	size_t orig_len = buf->len;
@@ -105,18 +55,71 @@ static void t_addstr(struct strbuf *buf, const void *data)
 	check_str(buf->buf + orig_len, text);
 }

+static void t_release(struct strbuf *sb)
+{
+	strbuf_release(sb);
+	check_uint(sb->len, ==, 0);
+	check_uint(sb->alloc, ==, 0);
+}
+
 int cmd_main(int argc, const char **argv)
 {
-	if (!TEST(t_static_init(), "static initialization works"))
-		test_skip_all("STRBUF_INIT is broken");
-	TEST(t_dynamic_init(), "dynamic initialization works");
-	TEST(setup(t_addch, "a"), "strbuf_addch adds char");
-	TEST(setup(t_addch, ""), "strbuf_addch adds NUL char");
-	TEST(setup_populated(t_addch, "initial value", "a"),
-	     "strbuf_addch appends to initial value");
-	TEST(setup(t_addstr, "hello there"), "strbuf_addstr adds string");
-	TEST(setup_populated(t_addstr, "initial value", "hello there"),
-	     "strbuf_addstr appends string to initial value");
+	for_test ("static initialization works") {
+		struct strbuf buf = STRBUF_INIT;
+
+		if (!check_uint(buf.len, ==, 0) ||
+		    !check_uint(buf.alloc, ==, 0) ||
+		    !check_char(buf.buf[0], ==, '\0'))
+			test_skip_all("STRBUF_INIT is broken");
+	}
+
+	for_test ("dynamic initialization works") {
+		struct strbuf buf;
+
+		strbuf_init(&buf, 1024);
+		check(assert_sane_strbuf(&buf));
+		check_uint(buf.len, ==, 0);
+		check_uint(buf.alloc, >=, 1024);
+		check_char(buf.buf[0], ==, '\0');
+		strbuf_release(&buf);
+	}
+
+	for_test ("strbuf_addch adds char") {
+		struct strbuf sb = STRBUF_INIT;
+
+		t_addch(&sb, 'a');
+		t_release(&sb);
+	}
+
+	for_test ("strbuf_addch adds NUL char") {
+		struct strbuf sb = STRBUF_INIT;
+
+		t_addch(&sb, '\0');
+		t_release(&sb);
+	}
+
+	for_test ("strbuf_addch appends to initial value") {
+		struct strbuf sb = STRBUF_INIT;
+
+		t_addstr(&sb, "initial value");
+		t_addch(&sb, 'a');
+		t_release(&sb);
+	}
+
+	for_test ("strbuf_addstr adds string") {
+		struct strbuf sb = STRBUF_INIT;
+
+		t_addstr(&sb, "hello there");
+		t_release(&sb);
+	}
+
+	for_test ("strbuf_addstr appends string to initial value") {
+		struct strbuf sb = STRBUF_INIT;
+
+		t_addstr(&sb, "initial value");
+		t_addstr(&sb, "hello there");
+		t_release(&sb);
+	}

 	return test_done();
 }
--
2.45.2

Re: [PATCH v3 3/7] unit-tests: add for_test

[Kyle Lippincott]

On Wed, Jul 24, 2024 at 7:53 AM René Scharfe <l.s.r@web.de> wrote:
>
> The macro TEST only allows defining a test that consists of a single
> expression.  Add a new macro, for_test, which provides a way to define
> unit tests that are made up of one or more statements.

Perhaps obvious to you and others, but I'm at a loss; can you
elaborate on why this is a significant problem? Why is having the
tests in a separate function a significant downside, and what are the
benefits of this new macro that counteract the "magic" here? I'm not
seeing it.

Macros like this require additional cognitive overhead to understand;
I can't read `for_test ("for_test passing test") check_int(1, ==, 1);`
and understand what's going on without going and reading the comment
above `for_test`. Unlike `TEST(<func>, "<description>")`, I can't even
_guess_ what it's doing. This macro is project-specific, and this
knowledge has to be attained by everyone wishing to even read this
code.

I personally consider the patches that use this to be regressions in
my ability to read and understand the tests. As an example, one of the
diff hunks in the strbuf patch (patch #6 in the series) does this:
-       TEST(setup_populated(t_addch, "initial value", "a"),
-            "strbuf_addch appends to initial value");
+       for_test ("strbuf_addch appends to initial value") {
+               struct strbuf sb = STRBUF_INIT;
+               t_addstr(&sb, "initial value");
+               t_addch(&sb, 'a');
+               t_release(&sb);
+       }

But the main simplification in that patch (not using
`setup_populated`) can be achieved without `for_test`:

+ static void t_addch_appends(void)
+ {
+     struct strbuf sb = STRBUF_INIT;
+     t_addstr(&sb, "initial value");
+     t_addch(&sb, 'a');
+     t_release(&sb);
+ }

-       TEST(setup_populated(t_addch, "initial value", "a"),
-            "strbuf_addch appends to initial value");
+       TEST(t_addch_appends(), "strbuf_addch appends to initial value");

It seems to me that all `for_test` is getting us there is an inlining
of the test logic, which seems like it's optimizing for vertical space
in the file. Maybe it's because I'm coming from a C++ environment at
$JOB that's using Google's gunit and gmock frameworks, where every
test is in its own function and we usually don't even write the main
function ourselves, but I have a preference for the separate
functions.

Maybe I'm in the minority, so only consider this at somewhere like a
-0.5 on this series (fine with deferring to a reviewer who is
significantly in favor of it, but if there aren't any, I'd lean
towards not landing this).

Re: [PATCH v2 2/6] unit-tests: add for_test

[Phillip Wood]

On 23/07/2024 14:24, Phillip Wood wrote:
> On 22/07/2024 20:13, Kyle Lippincott wrote:
>> On Sat, Jul 20, 2024 at 11:22 PM René Scharfe <l.s.r@web.de> wrote:
>>>
>> I think
>> having multiple test functions is generally easier to understand, and
>> the overhead isn't really relevant. It's not _as_ compact in the
>> source file, and requires that we have both the TEST statement and the
>> function (and forgetting the TEST statement means that we won't invoke
>> the function). If that is the main issue we're facing here, I wonder
>> if there's some other way of resolving that (such as unused function
>> detection via some compiler flags; even if it's not detected on all
>> platforms, getting at least one of the CI platforms should be
>> sufficient to prevent the issue [but we should target as many as
>> possible, so it's caught earlier than "I tried to send it to the
>> list"])
> 
> I also have a preference for function based tests but I do think 
> something like for_test() can be useful in certain situations.

I'm no-longer convinced that this is really the case

> For 
> example in test-ctype.c where testing the ctype macros leads to a lot of 
> repetition or a giant macro with a function based approach.

Having re-read the history of t/unit-tests/t-ctype.c I don't think the 
repetition in the original version was really that bad. The objection to 
it seems to have been that one had to write the class name (for example 
isalpha()) twice - once when defining the test function and again when 
calling it. I'm not sure that is really worth worrying about.

> Another example is where we are using a helper function with several 
> inputs and we would prefer to write
> 
> for_test("test 1") {
>      int input[] = ...
>      int expect[] = ...
> 
>      test_helper(input, expect);
> 
> ...
> 
> for_test("test 10") {
>      int input[] = ...
>      int expect[] = ...
> 
>      test_helper(input, expect);
> }

One can use blocks to achieve the same outcome with the TEST() macro

{
       int input[] = ...
       int expect[] = ...

       TEST(test_helper(input, expect), "test 1");
}

So overall I'm less convinced that adding something like for_test() is 
necessary and I'm very convinced that calling it for_test() and using 
"continue" to exit a test early is going to confuse contributors.

Best Wishes

Phillip

> rather then declaring all the input and expect variables up front with
> 
>      int input_1 = ...
>      int input_2 = ...
>      ...
>      int expect_1 = ...
>      int expect_2 = ...
> 
>      TEST(test_helper(input_1, expect_1), "test 1");
>      ...
>      TEST(test_helper(input_10, expect_10), "test 10");

>> If others agree that this is a good simplification for the people
>> reading the test code (and hopefully for the test author), I'm fine
>> with this going in (with a different name). I'm not trying to veto the
>> concept.
> 
> That's my feeling too.
> 
> Best Wishes
> 
> Phillip
> 
>>
>>> +
>>>   /*
>>>    * Print a test plan, should be called before any tests. If the number
>>>    * of tests is not known in advance test_done() will automatically
>>> -- 
>>> 2.45.2
>>>
>>
> 

Re: [PATCH v3 3/7] unit-tests: add for_test

[Phillip Wood]

On 24/07/2024 20:24, Kyle Lippincott wrote:
> On Wed, Jul 24, 2024 at 7:53 AM René Scharfe <l.s.r@web.de> wrote:
>>
> Macros like this require additional cognitive overhead to understand;
> I can't read `for_test ("for_test passing test") check_int(1, ==, 1);`
> and understand what's going on without going and reading the comment
> above `for_test`. Unlike `TEST(<func>, "<description>")`, I can't even
> _guess_ what it's doing. This macro is project-specific, and this
> knowledge has to be attained by everyone wishing to even read this
> code.

As well as the obscure name and oddness of using "continue" to exit a 
test early I think having two ways of defining tests also adds to the 
cognitive overhead of writing tests as one now has to decide which to use.

As you say below it is not clear to me that the changes in the last 
three patches that start using for_test() in our existing tests are an 
improvement.

This will be the last comment from me for a while as I'm about to go off 
the list for a couple of weeks

Best Wishes

Phillip

> I personally consider the patches that use this to be regressions in
> my ability to read and understand the tests. As an example, one of the
> diff hunks in the strbuf patch (patch #6 in the series) does this:
> -       TEST(setup_populated(t_addch, "initial value", "a"),
> -            "strbuf_addch appends to initial value");
> +       for_test ("strbuf_addch appends to initial value") {
> +               struct strbuf sb = STRBUF_INIT;
> +               t_addstr(&sb, "initial value");
> +               t_addch(&sb, 'a');
> +               t_release(&sb);
> +       }
> 
> But the main simplification in that patch (not using
> `setup_populated`) can be achieved without `for_test`:
> 
> + static void t_addch_appends(void)
> + {
> +     struct strbuf sb = STRBUF_INIT;
> +     t_addstr(&sb, "initial value");
> +     t_addch(&sb, 'a');
> +     t_release(&sb);
> + }
> 
> -       TEST(setup_populated(t_addch, "initial value", "a"),
> -            "strbuf_addch appends to initial value");
> +       TEST(t_addch_appends(), "strbuf_addch appends to initial value");
> 
> It seems to me that all `for_test` is getting us there is an inlining
> of the test logic, which seems like it's optimizing for vertical space
> in the file. Maybe it's because I'm coming from a C++ environment at
> $JOB that's using Google's gunit and gmock frameworks, where every
> test is in its own function and we usually don't even write the main
> function ourselves, but I have a preference for the separate
> functions.
> 
> Maybe I'm in the minority, so only consider this at somewhere like a
> -0.5 on this series (fine with deferring to a reviewer who is
> significantly in favor of it, but if there aren't any, I'd lean
> towards not landing this).
> 

Re: [PATCH v3 3/7] unit-tests: add for_test

[Junio C Hamano]

Kyle Lippincott <spectral@google.com> writes:

> But the main simplification in that patch (not using
> `setup_populated`) can be achieved without `for_test`:
>
> + static void t_addch_appends(void)
> + {
> +     struct strbuf sb = STRBUF_INIT;
> +     t_addstr(&sb, "initial value");
> +     t_addch(&sb, 'a');
> +     t_release(&sb);
> + }
>
> -       TEST(setup_populated(t_addch, "initial value", "a"),
> -            "strbuf_addch appends to initial value");
> +       TEST(t_addch_appends(), "strbuf_addch appends to initial value");

Yup.  I like that better when we are using TEST(), whether the other
thing exists or not.

> It seems to me that all `for_test` is getting us there is an inlining
> of the test logic, which seems like it's optimizing for vertical space
> in the file.

I would consider that it optimizes for human readers who have to
scan the file.  Having to jump around from caller to callee to
understand the whole thing (rather, the description being away from
the callee, which is where the most of the logic is anyway) is one
gripe I have with the approach taken by the TEST() thing.

> Maybe it's because I'm coming from a C++ environment at
> $JOB that's using Google's gunit and gmock frameworks, where every
> test is in its own function and we usually don't even write the main
> function ourselves, but I have a preference for the separate
> functions.

If we do not have to write the main at all, then it would make the
separate function that implements a single test a lot more palatable,
and we do not even have to implement and call TEST() macro ;-).

Thanks.

Re: [PATCH v3 3/7] unit-tests: add for_test

[Kyle Lippincott]

On Thu, Jul 25, 2024 at 9:02 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Kyle Lippincott <spectral@google.com> writes:
>
> > Maybe it's because I'm coming from a C++ environment at
> > $JOB that's using Google's gunit and gmock frameworks, where every
> > test is in its own function and we usually don't even write the main
> > function ourselves, but I have a preference for the separate
> > functions.
>
> If we do not have to write the main at all, then it would make the
> separate function that implements a single test a lot more palatable,
> and we do not even have to implement and call TEST() macro ;-).

I feel like you're trying to bait me into coming up with a way of
avoiding the main function ;) But I don't think it can really be done
in portable/vanilla C, unfortunately. I tried to think of a way to do
this, and they all involved some other system coming along and
identifying the tests and code-generating a main function, which also
seems like too much magic to me.

Re: [PATCH v3 3/7] unit-tests: add for_test

[Junio C Hamano]

Kyle Lippincott <spectral@google.com> writes:

>> > Maybe it's because I'm coming from a C++ environment at
>> > $JOB that's using Google's gunit and gmock frameworks, where every
>> > test is in its own function and we usually don't even write the main
>> > function ourselves, but I have a preference for the separate
>> > functions.
>>
>> If we do not have to write the main at all, then it would make the
>> separate function that implements a single test a lot more palatable,
>> and we do not even have to implement and call TEST() macro ;-).
> ...
> I tried to think of a way to do
> this, and they all involved some other system coming along and
> identifying the tests and code-generating a main function, which also
> seems like too much magic to me.

I thought that automatically generating the boilerplates from the
visible list of test functions and piecing them together with a
synthetic main was what was brought up as how libgit2 project does
this?  It does not sound all that involved and I do not find it a
rocket science.

In any case, a well written framework soon becomes "magic" anyway.
Our scripted test suite, back when Pasky and I invented it, was
unknown to many people, would have looked very far away from any
day-to-day shell scripts, and are full of magic.  Modern versions
gained even more magic than the what we had in the olden days.  But
I hear these days other projects imitate the pattern, so it may no
longer so much "magic" to many audiences.

If we use a well-written framework for C unit tests, I suspect that
the same thing would happen eventually.  An important point is the
"well-written" part, which needs to hide the ugly implementation
details well without revealing it in corners.  As Phillip said, a
loop construct similar to for() that you are not allowed to break or
continue may not qualify exactly for that reason.

Re: [PATCH v3 3/7] unit-tests: add for_test

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1565 bytes --]

On Thu, Jul 25, 2024 at 07:41:19PM -0700, Junio C Hamano wrote:
> Kyle Lippincott <spectral@google.com> writes:
> 
> >> > Maybe it's because I'm coming from a C++ environment at
> >> > $JOB that's using Google's gunit and gmock frameworks, where every
> >> > test is in its own function and we usually don't even write the main
> >> > function ourselves, but I have a preference for the separate
> >> > functions.
> >>
> >> If we do not have to write the main at all, then it would make the
> >> separate function that implements a single test a lot more palatable,
> >> and we do not even have to implement and call TEST() macro ;-).
> > ...
> > I tried to think of a way to do
> > this, and they all involved some other system coming along and
> > identifying the tests and code-generating a main function, which also
> > seems like too much magic to me.
> 
> I thought that automatically generating the boilerplates from the
> visible list of test functions and piecing them together with a
> synthetic main was what was brought up as how libgit2 project does
> this?  It does not sound all that involved and I do not find it a
> rocket science.

As I said when mentioning how libgit2 does it, I'd be happy to present a
working prototype for this if others think it was potentially useful.
Until now I didn't hear any positive feedback thoguh, you're the first
one saying that this might be something that is worth doing.

I just want to avoid wasting time on something that has no chance of
landing in the first place.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3 3/7] unit-tests: add for_test

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> As I said when mentioning how libgit2 does it, I'd be happy to present a
> working prototype for this if others think it was potentially useful.
> Until now I didn't hear any positive feedback thoguh, you're the first
> one saying that this might be something that is worth doing.
>
> I just want to avoid wasting time on something that has no chance of
> landing in the first place.

It takes such a sizeable effort to make it pleasant to use the
TEST() thing, it becomes somewhat questionable that it is worth the
effort.  What I was hoping to see was hinted in the part you did not
quote ;-)

If there is an existing well-written framework for C unit tests we
can apply to our project, it may not have to be so much "magic".  If
you think theirs is not so good and worth copying, then let's not.

Thanks.

Re: [PATCH v3 3/7] unit-tests: add for_test

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 774 bytes --]

On Fri, Jul 26, 2024 at 08:59:00AM -0700, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> If there is an existing well-written framework for C unit tests we
> can apply to our project, it may not have to be so much "magic".  If
> you think theirs is not so good and worth copying, then let's not.

With "theirs" being the libgit2 one? I always enjoyed using it, and it
is standalone nowadays and called "clar" [1]. The biggest downside of it
is that it depends on Python to auto-generate the test "main" function,
which is not really a good fit for the Git project. So ultimately, we'd
have to adapt the heart of clar anyway. It's not all that involved, but
something to keep in mind if we wanted to use it.

Patrick

[1]: https://github.com/clar-test/clar

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3 3/7] unit-tests: add for_test

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> is standalone nowadays and called "clar" [1]. The biggest downside of it
> is that it depends on Python to auto-generate the test "main" function,
> which is not really a good fit for the Git project.

Is that because Python is optional (like, we only use it for
optional Perforce thing and import-zip in contrib), or are there
other concerns?

Unlike these components, unit tests are not even for the end-user
consumers, so if it is Python that you find it a blocker, I do not
see a reason to reject it.  The thing looked like a simple script
that does not use any tricky language construct or modules.

Re: [PATCH v3 3/7] unit-tests: add for_test

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

On Mon, Jul 29, 2024 at 11:55:52AM -0700, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > is standalone nowadays and called "clar" [1]. The biggest downside of it
> > is that it depends on Python to auto-generate the test "main" function,
> > which is not really a good fit for the Git project.
> 
> Is that because Python is optional (like, we only use it for
> optional Perforce thing and import-zip in contrib), or are there
> other concerns?
> 
> Unlike these components, unit tests are not even for the end-user
> consumers, so if it is Python that you find it a blocker, I do not
> see a reason to reject it.  The thing looked like a simple script
> that does not use any tricky language construct or modules.

No concerns other than adding another mandatory dependency for devs. We
already depend on Perl and Shell, so adding Python to the stack feels
suboptimal to me.

As you say though, the script isn't all that involved. So it would also
be possible to port it to Perl if we want to do that, I guess.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 2/6] unit-tests: add for_test

[René Scharfe]

Am 25.07.24 um 11:45 schrieb Phillip Wood:
> On 23/07/2024 14:24, Phillip Wood wrote:
>
>> For example in test-ctype.c where testing the ctype macros leads to
>> a lot of repetition or a giant macro with a function based
>> approach.
>
> Having re-read the history of t/unit-tests/t-ctype.c I don't think
> the repetition in the original version was really that bad. The
> objection to it seems to have been that one had to write the class
> name (for example isalpha()) twice - once when defining the test
> function and again when calling it. I'm not sure that is really worth
> worrying about.

I can't see the purpose of requiring repetition like that.  It's an
unnecessary obstacle, small as it may be.  In the case of t-ctype it was
a regression to the original test helper code.  It's easy to avoid with
existing functions, except that those are off-limits to allow for future
changes.  So it's just a matter of packaging them nicely.

> So overall I'm less convinced that adding something like for_test()
> is necessary and I'm very convinced that calling it for_test() and
> using "continue" to exit a test early is going to confuse
> contributors.

I didn't expect anybody would want to use continue or break in a test,
but that's probably naive.  I got carried away there by the prospect
of a trivial macro-only solution, but alas, it's too clunky.  Two steps
back..

René

Re: [PATCH v3 3/7] unit-tests: add for_test

[René Scharfe]

Am 30.07.24 um 06:49 schrieb Patrick Steinhardt:
> On Mon, Jul 29, 2024 at 11:55:52AM -0700, Junio C Hamano wrote:
>> Patrick Steinhardt <ps@pks.im> writes:
>>
>>> is standalone nowadays and called "clar" [1]. The biggest downside of it
>>> is that it depends on Python to auto-generate the test "main" function,
>>> which is not really a good fit for the Git project.
>>
>> Is that because Python is optional (like, we only use it for
>> optional Perforce thing and import-zip in contrib), or are there
>> other concerns?
>>
>> Unlike these components, unit tests are not even for the end-user
>> consumers, so if it is Python that you find it a blocker, I do not
>> see a reason to reject it.  The thing looked like a simple script
>> that does not use any tricky language construct or modules.
>
> No concerns other than adding another mandatory dependency for devs. We
> already depend on Perl and Shell, so adding Python to the stack feels
> suboptimal to me.
>
> As you say though, the script isn't all that involved. So it would also
> be possible to port it to Perl if we want to do that, I guess.

From the point of view of a "minimal C unit testing framework" surely
the implementation language with the best dependency story would be C.
Perhaps it could then also test itself.  On the other hand, writing a
string handling program in 2024 in C is probably not the smartest idea.

Then again, generate.py uses a non-greedy regex to remove single-line
comments, which seems wrong, and doesn't seem to support preprocessor
directives like #if and #ifdef, so whole tests can't be disabled this
way.  And it uses pickle to cache results; does that mean it would be
slow without it?  Anyway, point being that parsing C isn't so easy
using Python either.

René

[PATCH v4 0/6] add and use if_test to simplify tests

[René Scharfe]

Changes since v3:
- Replace for_test with if_test, a more robust conditional-like macro.
  It stores description and location like v1, but without using strbuf.
- Drop patch 7 for t-strbuf to avoid hindering Kyle's cleanup idea.

Side note: Created with --creation-factor=60, as the default of 999 in
v2.46.0 pairs dropped patch 7 with new patch 4 and reports old patch 4
as dropped, which is confusing.

  t0080: use here-doc test body
  unit-tests: show location of checks outside of tests
  unit-tests: add if_test
  t-ctype: use if_test
  t-reftable-basics: use if_test
  t-strvec: use if_test

 .clang-format                    |   5 +
 t/helper/test-example-tap.c      |  35 +++
 t/t0080-unit-test-output.sh      |  60 ++++--
 t/unit-tests/t-ctype.c           |   4 +-
 t/unit-tests/t-reftable-basics.c | 228 +++++++++-----------
 t/unit-tests/t-strvec.c          | 356 ++++++++++++++-----------------
 t/unit-tests/test-lib.c          |  36 +++-
 t/unit-tests/test-lib.h          |  20 ++
 8 files changed, 405 insertions(+), 339 deletions(-)

Range-Diff gegen v3:
1:  497002df9e = 1:  497002df9e t0080: use here-doc test body
2:  0c1503fb5a = 2:  f85d4f9455 unit-tests: show location of checks outside of tests
3:  27f1f18b3d < -:  ---------- unit-tests: add for_test
-:  ---------- > 3:  77c7dfa1ad unit-tests: add if_test
4:  98a1e7abdf ! 4:  63fba50876 t-ctype: use for_test
    @@ Metadata
     Author: René Scharfe <l.s.r@web.de>

      ## Commit message ##
    -    t-ctype: use for_test
    +    t-ctype: use if_test

    -    Use the documented macro for_test instead of the internal functions
    +    Use the documented macro if_test instead of the internal functions
         test__run_begin() and test__run_end(), which are supposed to be private
         to the unit test framework.

    @@ t/unit-tests/t-ctype.c
      		BUILD_ASSERT_OR_ZERO(sizeof(string[0]) == sizeof(char)); \
     -	int skip = test__run_begin(); \
     -	if (!skip) { \
    -+	for_test (#class " works") { \
    ++	if_test (#class " works") { \
      		for (int i = 0; i < 256; i++) { \
      			if (!check_int(class(i), ==, !!memchr(string, i, len)))\
      				test_msg("      i: 0x%02x", i); \
5:  7c954f0864 ! 5:  ab86673484 t-reftable-basics: use for_test
    @@ Metadata
     Author: René Scharfe <l.s.r@web.de>

      ## Commit message ##
    -    t-reftable-basics: use for_test
    +    t-reftable-basics: use if_test

         The macro TEST takes a single expression.  If a test requires multiple
         statements then they need to be placed in a function that's called in
         the TEST expression.

         Remove the overhead of defining and calling single-use functions by
    -    using for_test instead.
    +    using if_test instead.

         Run the tests in the order of definition.  We can reorder them like that
         because they are independent.  Technically this changes the output, but
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -		struct integer_needle_lesseq_args args = {
     -			.haystack = haystack,
     -			.needle = testcases[i].needle,
    -+	for_test ("binary search with binsearch works") {
    ++	if_test ("binary search with binsearch works") {
     +		int haystack[] = { 2, 4, 6, 8, 10 };
     +		struct {
     +			int needle;
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -	const char *a[] = { "a", "b", "c", NULL };
     -	const char *b[] = { "a", "b", "d", NULL };
     -	const char *c[] = { "a", "b", NULL };
    -+	for_test ("names_length retuns size of a NULL-terminated string array") {
    ++	if_test ("names_length retuns size of a NULL-terminated string array") {
     +		const char *a[] = { "a", "b", NULL };
     +		check_int(names_length(a), ==, 2);
     +	}
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -	check(!names_equal(a, b));
     -	check(!names_equal(a, c));
     -}
    -+	for_test ("names_equal compares NULL-terminated string arrays") {
    ++	if_test ("names_equal compares NULL-terminated string arrays") {
     +		const char *a[] = { "a", "b", "c", NULL };
     +		const char *b[] = { "a", "b", "d", NULL };
     +		const char *c[] = { "a", "b", NULL };
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -	check(!out[2]);
     -	free_names(out);
     -}
    -+	for_test ("parse_names works for basic input") {
    ++	if_test ("parse_names works for basic input") {
     +		char in1[] = "line\n";
     +		char in2[] = "a\nb\nc";
     +		char **out = NULL;
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -		check_int(common_prefix_size(&a, &b), ==, cases[i].want);
     -		strbuf_reset(&a);
     -		strbuf_reset(&b);
    -+	for_test ("parse_names drops empty string") {
    ++	if_test ("parse_names drops empty string") {
     +		char in[] = "a\n\nb\n";
     +		char **out = NULL;
     +		parse_names(in, strlen(in), &out);
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -	out = get_be24(dest);
     -	check_int(in, ==, out);
     -}
    -+	for_test ("common_prefix_size works") {
    ++	if_test ("common_prefix_size works") {
     +		struct strbuf a = STRBUF_INIT;
     +		struct strbuf b = STRBUF_INIT;
     +		struct {
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     -	TEST(test_names_equal(), "names_equal compares NULL-terminated string arrays");
     -	TEST(test_u24_roundtrip(), "put_be24 and get_be24 work");
     -	TEST(test_u16_roundtrip(), "put_be16 and get_be16 work");
    -+	for_test ("put_be24 and get_be24 work") {
    ++	if_test ("put_be24 and get_be24 work") {
     +		uint32_t in = 0x112233;
     +		uint8_t dest[3];
     +		uint32_t out;
    @@ t/unit-tests/t-reftable-basics.c: static int integer_needle_lesseq(size_t i, voi
     +		check_int(in, ==, out);
     +	}
     +
    -+	for_test ("put_be16 and get_be16 work") {
    ++	if_test ("put_be16 and get_be16 work") {
     +		uint32_t in = 0xfef1;
     +		uint8_t dest[3];
     +		uint32_t out;
6:  d619a756d7 ! 6:  64bb731ba6 t-strvec: use for_test
    @@ Metadata
     Author: René Scharfe <l.s.r@web.de>

      ## Commit message ##
    -    t-strvec: use for_test
    +    t-strvec: use if_test

         The macro TEST takes a single expression.  If a test requires multiple
         statements then they need to be placed in a function that's called in
         the TEST expression.

         Remove the cognitive overhead of defining and calling single-use
    -    functions by using for_test instead.
    +    functions by using if_test instead.
    +
    +    Signed-off-by: René Scharfe <l.s.r@web.de>

      ## t/unit-tests/t-strvec.c ##
     @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct strvec *vec, ...)
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_uint(vec.nr, ==, 0);
     -	check_uint(vec.alloc, ==, 0);
     -}
    -+	for_test ("static initialization") {
    ++	if_test ("static initialization") {
     +		struct strvec vec = STRVEC_INIT;
     +		check_pointer_eq(vec.v, empty_strvec);
     +		check_uint(vec.nr, ==, 0);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_uint(vec.nr, ==, 0);
     -	check_uint(vec.alloc, ==, 0);
     -}
    -+	for_test ("dynamic initialization") {
    ++	if_test ("dynamic initialization") {
     +		struct strvec vec;
     +		strvec_init(&vec);
     +		check_pointer_eq(vec.v, empty_strvec);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_uint(vec.nr, ==, 0);
     -	check_uint(vec.alloc, ==, 0);
     -}
    -+	for_test ("clear") {
    ++	if_test ("clear") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_push(&vec, "foo");
     +		strvec_clear(&vec);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -static void t_push(void)
     -{
     -	struct strvec vec = STRVEC_INIT;
    -+	for_test ("push") {
    ++	if_test ("push") {
     +		struct strvec vec = STRVEC_INIT;

     -	strvec_push(&vec, "foo");
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo: 1", NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("pushf") {
    ++	if_test ("pushf") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushf(&vec, "foo: %d", 1);
     +		check_strvec(&vec, "foo: 1", NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "bar", "baz", NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("pushl") {
    ++	if_test ("pushl") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		check_strvec(&vec, "foo", "bar", "baz", NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -		"foo", "bar", "baz", NULL,
     -	};
     -	struct strvec vec = STRVEC_INIT;
    -+	for_test ("pushv") {
    ++	if_test ("pushv") {
     +		const char *strings[] = {
     +			"foo", "bar", "baz", NULL,
     +		};
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "replaced", "bar", "baz", NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("replace at head") {
    ++	if_test ("replace at head") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_replace(&vec, 0, "replaced");
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "bar", "replaced", NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("replace at tail") {
    ++	if_test ("replace at tail") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_replace(&vec, 2, "replaced");
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "replaced", "baz", NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("replace in between") {
    ++	if_test ("replace in between") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_replace(&vec, 1, "replaced");
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "oo", NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("replace with substring") {
    ++	if_test ("replace with substring") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", NULL);
     +		strvec_replace(&vec, 0, vec.v[0] + 1);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "bar", "baz", NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("remove at head") {
    ++	if_test ("remove at head") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_remove(&vec, 0);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "bar", NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("remove at tail") {
    ++	if_test ("remove at tail") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_remove(&vec, 2);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "baz", NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("remove in between") {
    ++	if_test ("remove in between") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_remove(&vec, 1);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("pop with empty array") {
    ++	if_test ("pop with empty array") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pop(&vec);
     +		check_strvec(&vec, NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "bar", NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("pop with non-empty array") {
    ++	if_test ("pop with non-empty array") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
     +		strvec_pop(&vec);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("split empty string") {
    ++	if_test ("split empty string") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_split(&vec, "");
     +		check_strvec(&vec, NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("split single item") {
    ++	if_test ("split single item") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_split(&vec, "foo");
     +		check_strvec(&vec, "foo", NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "bar", "baz", NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("split multiple items") {
    ++	if_test ("split multiple items") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_split(&vec, "foo bar baz");
     +		check_strvec(&vec, "foo", "bar", "baz", NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("split whitespace only") {
    ++	if_test ("split whitespace only") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_split(&vec, " \t\n");
     +		check_strvec(&vec, NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -	check_strvec(&vec, "foo", "bar", NULL);
     -	strvec_clear(&vec);
     -}
    -+	for_test ("split multiple consecutive whitespaces") {
    ++	if_test ("split multiple consecutive whitespaces") {
     +		struct strvec vec = STRVEC_INIT;
     +		strvec_split(&vec, "foo\n\t bar");
     +		check_strvec(&vec, "foo", "bar", NULL);
    @@ t/unit-tests/t-strvec.c: static void check_strvec_loc(const char *loc, struct st
     -{
     -	struct strvec vec = STRVEC_INIT;
     -	const char **detached;
    -+	for_test ("detach") {
    ++	if_test ("detach") {
     +		struct strvec vec = STRVEC_INIT;
     +		const char **detached;

7:  ea088728ad < -:  ---------- t-strbuf: use for_test
--
2.46.0

[PATCH v4 1/6] t0080: use here-doc test body

[René Scharfe]

Improve the readability of the expected output by using a here-doc for
the test body and replacing the unwieldy ${SQ} references with literal
single quotes.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/t0080-unit-test-output.sh | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh
index 7bbb065d58..9ec47b7360 100755
--- a/t/t0080-unit-test-output.sh
+++ b/t/t0080-unit-test-output.sh
@@ -5,7 +5,7 @@ test_description='Test the output of the unit test framework'
 TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh

-test_expect_success 'TAP output from unit tests' '
+test_expect_success 'TAP output from unit tests' - <<\EOT
 	cat >expect <<-EOF &&
 	ok 1 - passing test
 	ok 2 - passing test and assertion return 1
@@ -16,12 +16,12 @@ test_expect_success 'TAP output from unit tests' '
 	ok 4 - failing test and assertion return 0
 	not ok 5 - passing TEST_TODO() # TODO
 	ok 6 - passing TEST_TODO() returns 1
-	# todo check ${SQ}check(x)${SQ} succeeded at t/helper/test-example-tap.c:26
+	# todo check 'check(x)' succeeded at t/helper/test-example-tap.c:26
 	not ok 7 - failing TEST_TODO()
 	ok 8 - failing TEST_TODO() returns 0
 	# check "0" failed at t/helper/test-example-tap.c:31
 	# skipping test - missing prerequisite
-	# skipping check ${SQ}1${SQ} at t/helper/test-example-tap.c:33
+	# skipping check '1' at t/helper/test-example-tap.c:33
 	ok 9 - test_skip() # SKIP
 	ok 10 - skipped test returns 1
 	# skipping test - missing prerequisite
@@ -39,12 +39,12 @@ test_expect_success 'TAP output from unit tests' '
 	# check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:63
 	#    left: "NULL"
 	#   right: NULL
-	# check "${SQ}a${SQ} == ${SQ}\n${SQ}" failed at t/helper/test-example-tap.c:64
-	#    left: ${SQ}a${SQ}
-	#   right: ${SQ}\012${SQ}
-	# check "${SQ}\\\\${SQ} == ${SQ}\\${SQ}${SQ}" failed at t/helper/test-example-tap.c:65
-	#    left: ${SQ}\\\\${SQ}
-	#   right: ${SQ}\\${SQ}${SQ}
+	# check "'a' == '\n'" failed at t/helper/test-example-tap.c:64
+	#    left: 'a'
+	#   right: '\012'
+	# check "'\\\\' == '\\''" failed at t/helper/test-example-tap.c:65
+	#    left: '\\\\'
+	#   right: '\\''
 	not ok 17 - messages from failing string and char comparison
 	# BUG: test has no checks at t/helper/test-example-tap.c:92
 	not ok 18 - test with no checks
@@ -54,6 +54,6 @@ test_expect_success 'TAP output from unit tests' '

 	! test-tool example-tap >actual &&
 	test_cmp expect actual
-'
+EOT

 test_done
--
2.46.0

[PATCH v4 2/6] unit-tests: show location of checks outside of tests

[René Scharfe]

Checks outside of tests are caught at runtime and reported like this:

 Assertion failed: (ctx.running), function test_assert, file test-lib.c, line 267.

The assert() call aborts the unit test and doesn't reveal the location
or even the type of the offending check, as test_assert() is called by
all of them.

Handle it like the opposite case, a test without any checks: Don't
abort, but report the location of the actual check, along with a message
explaining the situation.  The output for example above becomes:

 # BUG: check outside of test at t/helper/test-example-tap.c:75

... and the unit test program continues and indicates the error in its
exit code at the end.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/helper/test-example-tap.c | 2 ++
 t/t0080-unit-test-output.sh | 5 +++--
 t/unit-tests/test-lib.c     | 7 ++++++-
 3 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/t/helper/test-example-tap.c b/t/helper/test-example-tap.c
index d072ad559f..79c12b01cd 100644
--- a/t/helper/test-example-tap.c
+++ b/t/helper/test-example-tap.c
@@ -72,6 +72,8 @@ static void t_empty(void)

 int cmd__example_tap(int argc, const char **argv)
 {
+	check(1);
+
 	test_res = TEST(check_res = check_int(1, ==, 1), "passing test");
 	TEST(t_res(1), "passing test and assertion return 1");
 	test_res = TEST(check_res = check_int(1, ==, 2), "failing test");
diff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh
index 9ec47b7360..fe221f3bdb 100755
--- a/t/t0080-unit-test-output.sh
+++ b/t/t0080-unit-test-output.sh
@@ -7,9 +7,10 @@ TEST_PASSES_SANITIZE_LEAK=true

 test_expect_success 'TAP output from unit tests' - <<\EOT
 	cat >expect <<-EOF &&
+	# BUG: check outside of test at t/helper/test-example-tap.c:75
 	ok 1 - passing test
 	ok 2 - passing test and assertion return 1
-	# check "1 == 2" failed at t/helper/test-example-tap.c:77
+	# check "1 == 2" failed at t/helper/test-example-tap.c:79
 	#    left: 1
 	#   right: 2
 	not ok 3 - failing test
@@ -46,7 +47,7 @@ test_expect_success 'TAP output from unit tests' - <<\EOT
 	#    left: '\\\\'
 	#   right: '\\''
 	not ok 17 - messages from failing string and char comparison
-	# BUG: test has no checks at t/helper/test-example-tap.c:92
+	# BUG: test has no checks at t/helper/test-example-tap.c:94
 	not ok 18 - test with no checks
 	ok 19 - test with no checks returns 0
 	1..19
diff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c
index 3c513ce59a..989dc758e6 100644
--- a/t/unit-tests/test-lib.c
+++ b/t/unit-tests/test-lib.c
@@ -264,7 +264,12 @@ static void test_todo(void)

 int test_assert(const char *location, const char *check, int ok)
 {
-	assert(ctx.running);
+	if (!ctx.running) {
+		test_msg("BUG: check outside of test at %s",
+			 make_relative(location));
+		ctx.failed = 1;
+		return 0;
+	}

 	if (ctx.result == RESULT_SKIP) {
 		test_msg("skipping check '%s' at %s", check,
--
2.46.0

[PATCH v4 3/6] unit-tests: add if_test

[René Scharfe]

The macro TEST only allows defining a test that consists of a single
expression.  Add a new macro, if_test, which provides a way to define
unit tests that are made up of one or more statements.

if_test allows defining self-contained tests en bloc, a bit like
test_expect_success does for regular tests.  It acts like a conditional;
the test body is executed if test_skip_all() had not been called before.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 .clang-format               |  5 +++++
 t/helper/test-example-tap.c | 33 +++++++++++++++++++++++++++++++++
 t/t0080-unit-test-output.sh | 35 ++++++++++++++++++++++++++++++++++-
 t/unit-tests/test-lib.c     | 29 +++++++++++++++++++++++++++++
 t/unit-tests/test-lib.h     | 20 ++++++++++++++++++++
 5 files changed, 121 insertions(+), 1 deletion(-)

diff --git a/.clang-format b/.clang-format
index 6408251577..4c6d317508 100644
--- a/.clang-format
+++ b/.clang-format
@@ -169,6 +169,11 @@ ForEachMacros:
   - 'strmap_for_each_entry'
   - 'strset_for_each_entry'

+# A list of macros that should be interpreted as conditionals instead of as
+# function calls.
+IfMacros:
+  - 'if_test'
+
 # The maximum number of consecutive empty lines to keep.
 MaxEmptyLinesToKeep: 1

diff --git a/t/helper/test-example-tap.c b/t/helper/test-example-tap.c
index 79c12b01cd..914af88e0a 100644
--- a/t/helper/test-example-tap.c
+++ b/t/helper/test-example-tap.c
@@ -94,5 +94,38 @@ int cmd__example_tap(int argc, const char **argv)
 	test_res = TEST(t_empty(), "test with no checks");
 	TEST(check_int(test_res, ==, 0), "test with no checks returns 0");

+	if_test ("if_test passing test")
+		check_int(1, ==, 1);
+	if_test ("if_test failing test")
+		check_int(1, ==, 2);
+	if_test ("if_test passing TEST_TODO()")
+		TEST_TODO(check(0));
+	if_test ("if_test failing TEST_TODO()")
+		TEST_TODO(check(1));
+	if_test ("if_test test_skip()") {
+		check(0);
+		test_skip("missing prerequisite");
+		check(1);
+	}
+	if_test ("if_test test_skip() inside TEST_TODO()")
+		TEST_TODO((test_skip("missing prerequisite"), 1));
+	if_test ("if_test TEST_TODO() after failing check") {
+		check(0);
+		TEST_TODO(check(0));
+	}
+	if_test ("if_test failing check after TEST_TODO()") {
+		check(1);
+		TEST_TODO(check(0));
+		check(0);
+	}
+	if_test ("if_test messages from failing string and char comparison") {
+		check_str("\thello\\", "there\"\n");
+		check_str("NULL", NULL);
+		check_char('a', ==, '\n');
+		check_char('\\', ==, '\'');
+	}
+	if_test ("if_test test with no checks")
+		; /* nothing */
+
 	return test_done();
 }
diff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh
index fe221f3bdb..3c369c88e2 100755
--- a/t/t0080-unit-test-output.sh
+++ b/t/t0080-unit-test-output.sh
@@ -50,7 +50,40 @@ test_expect_success 'TAP output from unit tests' - <<\EOT
 	# BUG: test has no checks at t/helper/test-example-tap.c:94
 	not ok 18 - test with no checks
 	ok 19 - test with no checks returns 0
-	1..19
+	ok 20 - if_test passing test
+	# check "1 == 2" failed at t/helper/test-example-tap.c:100
+	#    left: 1
+	#   right: 2
+	not ok 21 - if_test failing test
+	not ok 22 - if_test passing TEST_TODO() # TODO
+	# todo check 'check(1)' succeeded at t/helper/test-example-tap.c:104
+	not ok 23 - if_test failing TEST_TODO()
+	# check "0" failed at t/helper/test-example-tap.c:106
+	# skipping test - missing prerequisite
+	# skipping check '1' at t/helper/test-example-tap.c:108
+	ok 24 - if_test test_skip() # SKIP
+	# skipping test - missing prerequisite
+	ok 25 - if_test test_skip() inside TEST_TODO() # SKIP
+	# check "0" failed at t/helper/test-example-tap.c:113
+	not ok 26 - if_test TEST_TODO() after failing check
+	# check "0" failed at t/helper/test-example-tap.c:119
+	not ok 27 - if_test failing check after TEST_TODO()
+	# check "!strcmp("\thello\\\\", "there\"\n")" failed at t/helper/test-example-tap.c:122
+	#    left: "\011hello\\\\"
+	#   right: "there\"\012"
+	# check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:123
+	#    left: "NULL"
+	#   right: NULL
+	# check "'a' == '\n'" failed at t/helper/test-example-tap.c:124
+	#    left: 'a'
+	#   right: '\012'
+	# check "'\\\\' == '\\''" failed at t/helper/test-example-tap.c:125
+	#    left: '\\\\'
+	#   right: '\\''
+	not ok 28 - if_test messages from failing string and char comparison
+	# BUG: test has no checks at t/helper/test-example-tap.c:127
+	not ok 29 - if_test test with no checks
+	1..29
 	EOF

 	! test-tool example-tap >actual &&
diff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c
index 989dc758e6..fa1f95965c 100644
--- a/t/unit-tests/test-lib.c
+++ b/t/unit-tests/test-lib.c
@@ -16,6 +16,8 @@ static struct {
 	unsigned running :1;
 	unsigned skip_all :1;
 	unsigned todo :1;
+	char location[100];
+	char description[100];
 } ctx = {
 	.lazy_plan = 1,
 	.result = RESULT_NONE,
@@ -125,6 +127,8 @@ void test_plan(int count)

 int test_done(void)
 {
+	if (ctx.running && ctx.location[0] && ctx.description[0])
+		test__run_end(1, ctx.location, "%s", ctx.description);
 	assert(!ctx.running);

 	if (ctx.lazy_plan)
@@ -167,13 +171,38 @@ void test_skip_all(const char *format, ...)
 	va_end(ap);
 }

+void test__run_describe(const char *location, const char *format, ...)
+{
+	va_list ap;
+	int len;
+
+	assert(ctx.running);
+	assert(!ctx.location[0]);
+	assert(!ctx.description[0]);
+
+	xsnprintf(ctx.location, sizeof(ctx.location), "%s",
+		  make_relative(location));
+
+	va_start(ap, format);
+	len = vsnprintf(ctx.description, sizeof(ctx.description), format, ap);
+	va_end(ap);
+	if (len < 0)
+		die("unable to format message: %s", format);
+	if (len >= sizeof(ctx.description))
+		BUG("ctx.description too small to format %s", format);
+}
+
 int test__run_begin(void)
 {
+	if (ctx.running && ctx.location[0] && ctx.description[0])
+		test__run_end(1, ctx.location, "%s", ctx.description);
 	assert(!ctx.running);

 	ctx.count++;
 	ctx.result = RESULT_NONE;
 	ctx.running = 1;
+	ctx.location[0] = '\0';
+	ctx.description[0] = '\0';

 	return ctx.skip_all;
 }
diff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h
index 2de6d715d5..f15dceb29e 100644
--- a/t/unit-tests/test-lib.h
+++ b/t/unit-tests/test-lib.h
@@ -14,6 +14,23 @@
 	test__run_end(test__run_begin() ? 0 : (t, 1),	\
 		      TEST_LOCATION(),  __VA_ARGS__)

+/*
+ * Run a test unless test_skip_all() has been called.  Acts like a
+ * conditional; the test body is expected as a statement or block after
+ * the closing parenthesis.  The description for each test should be
+ * unique.  E.g.:
+ *
+ *  if_test ("something else %d %d", arg1, arg2) {
+ *          prepare();
+ *          test_something_else(arg1, arg2);
+ *          cleanup();
+ *  }
+ */
+#define if_test(...)							\
+	if (test__run_begin() ?						\
+	    (test__run_end(0, TEST_LOCATION(),  __VA_ARGS__), 0) :	\
+	    (test__run_describe(TEST_LOCATION(),  __VA_ARGS__), 1))
+
 /*
  * Print a test plan, should be called before any tests. If the number
  * of tests is not known in advance test_done() will automatically
@@ -153,6 +170,9 @@ union test__tmp {

 extern union test__tmp test__tmp[2];

+__attribute__((format (printf, 2, 3)))
+void test__run_describe(const char *, const char *, ...);
+
 int test__run_begin(void);
 __attribute__((format (printf, 3, 4)))
 int test__run_end(int, const char *, const char *, ...);
--
2.46.0

[PATCH v4 4/6] t-ctype: use if_test

[René Scharfe]

Use the documented macro if_test instead of the internal functions
test__run_begin() and test__run_end(), which are supposed to be private
to the unit test framework.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-ctype.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/t/unit-tests/t-ctype.c b/t/unit-tests/t-ctype.c
index d6ac1fe678..e28a7f50f9 100644
--- a/t/unit-tests/t-ctype.c
+++ b/t/unit-tests/t-ctype.c
@@ -4,15 +4,13 @@
 	size_t len = ARRAY_SIZE(string) - 1 + \
 		BUILD_ASSERT_OR_ZERO(ARRAY_SIZE(string) > 0) + \
 		BUILD_ASSERT_OR_ZERO(sizeof(string[0]) == sizeof(char)); \
-	int skip = test__run_begin(); \
-	if (!skip) { \
+	if_test (#class " works") { \
 		for (int i = 0; i < 256; i++) { \
 			if (!check_int(class(i), ==, !!memchr(string, i, len)))\
 				test_msg("      i: 0x%02x", i); \
 		} \
 		check(!class(EOF)); \
 	} \
-	test__run_end(!skip, TEST_LOCATION(), #class " works"); \
 } while (0)

 #define DIGIT "0123456789"
--
2.46.0

[PATCH v4 5/6] t-reftable-basics: use if_test

[René Scharfe]

The macro TEST takes a single expression.  If a test requires multiple
statements then they need to be placed in a function that's called in
the TEST expression.

Remove the overhead of defining and calling single-use functions by
using if_test instead.

Run the tests in the order of definition.  We can reorder them like that
because they are independent.  Technically this changes the output, but
retains the meaning of a full run and allows for easier review e.g. with
diff option --ignore-all-space.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-reftable-basics.c | 228 ++++++++++++++-----------------
 1 file changed, 106 insertions(+), 122 deletions(-)

diff --git a/t/unit-tests/t-reftable-basics.c b/t/unit-tests/t-reftable-basics.c
index 4e80bdf16d..1dd60ab5f0 100644
--- a/t/unit-tests/t-reftable-basics.c
+++ b/t/unit-tests/t-reftable-basics.c
@@ -20,141 +20,125 @@ static int integer_needle_lesseq(size_t i, void *_args)
 	return args->needle <= args->haystack[i];
 }

-static void test_binsearch(void)
+int cmd_main(int argc, const char *argv[])
 {
-	int haystack[] = { 2, 4, 6, 8, 10 };
-	struct {
-		int needle;
-		size_t expected_idx;
-	} testcases[] = {
-		{-9000, 0},
-		{-1, 0},
-		{0, 0},
-		{2, 0},
-		{3, 1},
-		{4, 1},
-		{7, 3},
-		{9, 4},
-		{10, 4},
-		{11, 5},
-		{9000, 5},
-	};
-
-	for (size_t i = 0; i < ARRAY_SIZE(testcases); i++) {
-		struct integer_needle_lesseq_args args = {
-			.haystack = haystack,
-			.needle = testcases[i].needle,
+	if_test ("binary search with binsearch works") {
+		int haystack[] = { 2, 4, 6, 8, 10 };
+		struct {
+			int needle;
+			size_t expected_idx;
+		} testcases[] = {
+			{-9000, 0},
+			{-1, 0},
+			{0, 0},
+			{2, 0},
+			{3, 1},
+			{4, 1},
+			{7, 3},
+			{9, 4},
+			{10, 4},
+			{11, 5},
+			{9000, 5},
 		};
-		size_t idx;

-		idx = binsearch(ARRAY_SIZE(haystack), &integer_needle_lesseq, &args);
-		check_int(idx, ==, testcases[i].expected_idx);
+		for (size_t i = 0; i < ARRAY_SIZE(testcases); i++) {
+			struct integer_needle_lesseq_args args = {
+				.haystack = haystack,
+				.needle = testcases[i].needle,
+			};
+			size_t idx;
+
+			idx = binsearch(ARRAY_SIZE(haystack),
+					&integer_needle_lesseq, &args);
+			check_int(idx, ==, testcases[i].expected_idx);
+		}
 	}
-}

-static void test_names_length(void)
-{
-	const char *a[] = { "a", "b", NULL };
-	check_int(names_length(a), ==, 2);
-}
-
-static void test_names_equal(void)
-{
-	const char *a[] = { "a", "b", "c", NULL };
-	const char *b[] = { "a", "b", "d", NULL };
-	const char *c[] = { "a", "b", NULL };
+	if_test ("names_length retuns size of a NULL-terminated string array") {
+		const char *a[] = { "a", "b", NULL };
+		check_int(names_length(a), ==, 2);
+	}

-	check(names_equal(a, a));
-	check(!names_equal(a, b));
-	check(!names_equal(a, c));
-}
+	if_test ("names_equal compares NULL-terminated string arrays") {
+		const char *a[] = { "a", "b", "c", NULL };
+		const char *b[] = { "a", "b", "d", NULL };
+		const char *c[] = { "a", "b", NULL };

-static void test_parse_names_normal(void)
-{
-	char in1[] = "line\n";
-	char in2[] = "a\nb\nc";
-	char **out = NULL;
-	parse_names(in1, strlen(in1), &out);
-	check_str(out[0], "line");
-	check(!out[1]);
-	free_names(out);
-
-	parse_names(in2, strlen(in2), &out);
-	check_str(out[0], "a");
-	check_str(out[1], "b");
-	check_str(out[2], "c");
-	check(!out[3]);
-	free_names(out);
-}
+		check(names_equal(a, a));
+		check(!names_equal(a, b));
+		check(!names_equal(a, c));
+	}

-static void test_parse_names_drop_empty(void)
-{
-	char in[] = "a\n\nb\n";
-	char **out = NULL;
-	parse_names(in, strlen(in), &out);
-	check_str(out[0], "a");
-	/* simply '\n' should be dropped as empty string */
-	check_str(out[1], "b");
-	check(!out[2]);
-	free_names(out);
-}
+	if_test ("parse_names works for basic input") {
+		char in1[] = "line\n";
+		char in2[] = "a\nb\nc";
+		char **out = NULL;
+		parse_names(in1, strlen(in1), &out);
+		check_str(out[0], "line");
+		check(!out[1]);
+		free_names(out);
+
+		parse_names(in2, strlen(in2), &out);
+		check_str(out[0], "a");
+		check_str(out[1], "b");
+		check_str(out[2], "c");
+		check(!out[3]);
+		free_names(out);
+	}

-static void test_common_prefix(void)
-{
-	struct strbuf a = STRBUF_INIT;
-	struct strbuf b = STRBUF_INIT;
-	struct {
-		const char *a, *b;
-		int want;
-	} cases[] = {
-		{"abcdef", "abc", 3},
-		{ "abc", "ab", 2 },
-		{ "", "abc", 0 },
-		{ "abc", "abd", 2 },
-		{ "abc", "pqr", 0 },
-	};
-
-	for (size_t i = 0; i < ARRAY_SIZE(cases); i++) {
-		strbuf_addstr(&a, cases[i].a);
-		strbuf_addstr(&b, cases[i].b);
-		check_int(common_prefix_size(&a, &b), ==, cases[i].want);
-		strbuf_reset(&a);
-		strbuf_reset(&b);
+	if_test ("parse_names drops empty string") {
+		char in[] = "a\n\nb\n";
+		char **out = NULL;
+		parse_names(in, strlen(in), &out);
+		check_str(out[0], "a");
+		/* simply '\n' should be dropped as empty string */
+		check_str(out[1], "b");
+		check(!out[2]);
+		free_names(out);
 	}
-	strbuf_release(&a);
-	strbuf_release(&b);
-}

-static void test_u24_roundtrip(void)
-{
-	uint32_t in = 0x112233;
-	uint8_t dest[3];
-	uint32_t out;
-	put_be24(dest, in);
-	out = get_be24(dest);
-	check_int(in, ==, out);
-}
+	if_test ("common_prefix_size works") {
+		struct strbuf a = STRBUF_INIT;
+		struct strbuf b = STRBUF_INIT;
+		struct {
+			const char *a, *b;
+			int want;
+		} cases[] = {
+			{"abcdef", "abc", 3},
+			{ "abc", "ab", 2 },
+			{ "", "abc", 0 },
+			{ "abc", "abd", 2 },
+			{ "abc", "pqr", 0 },
+		};

-static void test_u16_roundtrip(void)
-{
-	uint32_t in = 0xfef1;
-	uint8_t dest[3];
-	uint32_t out;
-	put_be16(dest, in);
-	out = get_be16(dest);
-	check_int(in, ==, out);
-}
+		for (size_t i = 0; i < ARRAY_SIZE(cases); i++) {
+			strbuf_addstr(&a, cases[i].a);
+			strbuf_addstr(&b, cases[i].b);
+			check_int(common_prefix_size(&a, &b), ==, cases[i].want);
+			strbuf_reset(&a);
+			strbuf_reset(&b);
+		}
+		strbuf_release(&a);
+		strbuf_release(&b);
+	}

-int cmd_main(int argc, const char *argv[])
-{
-	TEST(test_common_prefix(), "common_prefix_size works");
-	TEST(test_parse_names_normal(), "parse_names works for basic input");
-	TEST(test_parse_names_drop_empty(), "parse_names drops empty string");
-	TEST(test_binsearch(), "binary search with binsearch works");
-	TEST(test_names_length(), "names_length retuns size of a NULL-terminated string array");
-	TEST(test_names_equal(), "names_equal compares NULL-terminated string arrays");
-	TEST(test_u24_roundtrip(), "put_be24 and get_be24 work");
-	TEST(test_u16_roundtrip(), "put_be16 and get_be16 work");
+	if_test ("put_be24 and get_be24 work") {
+		uint32_t in = 0x112233;
+		uint8_t dest[3];
+		uint32_t out;
+		put_be24(dest, in);
+		out = get_be24(dest);
+		check_int(in, ==, out);
+	}
+
+	if_test ("put_be16 and get_be16 work") {
+		uint32_t in = 0xfef1;
+		uint8_t dest[3];
+		uint32_t out;
+		put_be16(dest, in);
+		out = get_be16(dest);
+		check_int(in, ==, out);
+	}

 	return test_done();
 }
--
2.46.0

[PATCH v4 6/6] t-strvec: use if_test

[René Scharfe]

The macro TEST takes a single expression.  If a test requires multiple
statements then they need to be placed in a function that's called in
the TEST expression.

Remove the cognitive overhead of defining and calling single-use
functions by using if_test instead.

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 t/unit-tests/t-strvec.c | 356 ++++++++++++++++++----------------------
 1 file changed, 156 insertions(+), 200 deletions(-)

diff --git a/t/unit-tests/t-strvec.c b/t/unit-tests/t-strvec.c
index d4615ab06d..557e9c835c 100644
--- a/t/unit-tests/t-strvec.c
+++ b/t/unit-tests/t-strvec.c
@@ -36,237 +36,193 @@ static void check_strvec_loc(const char *loc, struct strvec *vec, ...)
 	check_pointer_eq(vec->v[nr], NULL);
 }

-static void t_static_init(void)
+int cmd_main(int argc, const char **argv)
 {
-	struct strvec vec = STRVEC_INIT;
-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
-}
+	if_test ("static initialization") {
+		struct strvec vec = STRVEC_INIT;
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);
+	}

-static void t_dynamic_init(void)
-{
-	struct strvec vec;
-	strvec_init(&vec);
-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
-}
+	if_test ("dynamic initialization") {
+		struct strvec vec;
+		strvec_init(&vec);
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);
+	}

-static void t_clear(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_push(&vec, "foo");
-	strvec_clear(&vec);
-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
-}
+	if_test ("clear") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_push(&vec, "foo");
+		strvec_clear(&vec);
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);
+	}

-static void t_push(void)
-{
-	struct strvec vec = STRVEC_INIT;
+	if_test ("push") {
+		struct strvec vec = STRVEC_INIT;

-	strvec_push(&vec, "foo");
-	check_strvec(&vec, "foo", NULL);
+		strvec_push(&vec, "foo");
+		check_strvec(&vec, "foo", NULL);

-	strvec_push(&vec, "bar");
-	check_strvec(&vec, "foo", "bar", NULL);
+		strvec_push(&vec, "bar");
+		check_strvec(&vec, "foo", "bar", NULL);

-	strvec_clear(&vec);
-}
+		strvec_clear(&vec);
+	}

-static void t_pushf(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushf(&vec, "foo: %d", 1);
-	check_strvec(&vec, "foo: 1", NULL);
-	strvec_clear(&vec);
-}
+	if_test ("pushf") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushf(&vec, "foo: %d", 1);
+		check_strvec(&vec, "foo: 1", NULL);
+		strvec_clear(&vec);
+	}

-static void t_pushl(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	check_strvec(&vec, "foo", "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	if_test ("pushl") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		check_strvec(&vec, "foo", "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_pushv(void)
-{
-	const char *strings[] = {
-		"foo", "bar", "baz", NULL,
-	};
-	struct strvec vec = STRVEC_INIT;
+	if_test ("pushv") {
+		const char *strings[] = {
+			"foo", "bar", "baz", NULL,
+		};
+		struct strvec vec = STRVEC_INIT;

-	strvec_pushv(&vec, strings);
-	check_strvec(&vec, "foo", "bar", "baz", NULL);
+		strvec_pushv(&vec, strings);
+		check_strvec(&vec, "foo", "bar", "baz", NULL);

-	strvec_clear(&vec);
-}
+		strvec_clear(&vec);
+	}

-static void t_replace_at_head(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_replace(&vec, 0, "replaced");
-	check_strvec(&vec, "replaced", "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	if_test ("replace at head") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_replace(&vec, 0, "replaced");
+		check_strvec(&vec, "replaced", "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_replace_at_tail(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_replace(&vec, 2, "replaced");
-	check_strvec(&vec, "foo", "bar", "replaced", NULL);
-	strvec_clear(&vec);
-}
+	if_test ("replace at tail") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_replace(&vec, 2, "replaced");
+		check_strvec(&vec, "foo", "bar", "replaced", NULL);
+		strvec_clear(&vec);
+	}

-static void t_replace_in_between(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_replace(&vec, 1, "replaced");
-	check_strvec(&vec, "foo", "replaced", "baz", NULL);
-	strvec_clear(&vec);
-}
+	if_test ("replace in between") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_replace(&vec, 1, "replaced");
+		check_strvec(&vec, "foo", "replaced", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_replace_with_substring(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", NULL);
-	strvec_replace(&vec, 0, vec.v[0] + 1);
-	check_strvec(&vec, "oo", NULL);
-	strvec_clear(&vec);
-}
+	if_test ("replace with substring") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", NULL);
+		strvec_replace(&vec, 0, vec.v[0] + 1);
+		check_strvec(&vec, "oo", NULL);
+		strvec_clear(&vec);
+	}

-static void t_remove_at_head(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_remove(&vec, 0);
-	check_strvec(&vec, "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	if_test ("remove at head") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_remove(&vec, 0);
+		check_strvec(&vec, "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_remove_at_tail(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_remove(&vec, 2);
-	check_strvec(&vec, "foo", "bar", NULL);
-	strvec_clear(&vec);
-}
+	if_test ("remove at tail") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_remove(&vec, 2);
+		check_strvec(&vec, "foo", "bar", NULL);
+		strvec_clear(&vec);
+	}

-static void t_remove_in_between(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_remove(&vec, 1);
-	check_strvec(&vec, "foo", "baz", NULL);
-	strvec_clear(&vec);
-}
+	if_test ("remove in between") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_remove(&vec, 1);
+		check_strvec(&vec, "foo", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_pop_empty_array(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pop(&vec);
-	check_strvec(&vec, NULL);
-	strvec_clear(&vec);
-}
+	if_test ("pop with empty array") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pop(&vec);
+		check_strvec(&vec, NULL);
+		strvec_clear(&vec);
+	}

-static void t_pop_non_empty_array(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_pushl(&vec, "foo", "bar", "baz", NULL);
-	strvec_pop(&vec);
-	check_strvec(&vec, "foo", "bar", NULL);
-	strvec_clear(&vec);
-}
+	if_test ("pop with non-empty array") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_pushl(&vec, "foo", "bar", "baz", NULL);
+		strvec_pop(&vec);
+		check_strvec(&vec, "foo", "bar", NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_empty_string(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "");
-	check_strvec(&vec, NULL);
-	strvec_clear(&vec);
-}
+	if_test ("split empty string") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "");
+		check_strvec(&vec, NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_single_item(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "foo");
-	check_strvec(&vec, "foo", NULL);
-	strvec_clear(&vec);
-}
+	if_test ("split single item") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "foo");
+		check_strvec(&vec, "foo", NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_multiple_items(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "foo bar baz");
-	check_strvec(&vec, "foo", "bar", "baz", NULL);
-	strvec_clear(&vec);
-}
+	if_test ("split multiple items") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "foo bar baz");
+		check_strvec(&vec, "foo", "bar", "baz", NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_whitespace_only(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, " \t\n");
-	check_strvec(&vec, NULL);
-	strvec_clear(&vec);
-}
+	if_test ("split whitespace only") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, " \t\n");
+		check_strvec(&vec, NULL);
+		strvec_clear(&vec);
+	}

-static void t_split_multiple_consecutive_whitespaces(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	strvec_split(&vec, "foo\n\t bar");
-	check_strvec(&vec, "foo", "bar", NULL);
-	strvec_clear(&vec);
-}
+	if_test ("split multiple consecutive whitespaces") {
+		struct strvec vec = STRVEC_INIT;
+		strvec_split(&vec, "foo\n\t bar");
+		check_strvec(&vec, "foo", "bar", NULL);
+		strvec_clear(&vec);
+	}

-static void t_detach(void)
-{
-	struct strvec vec = STRVEC_INIT;
-	const char **detached;
+	if_test ("detach") {
+		struct strvec vec = STRVEC_INIT;
+		const char **detached;

-	strvec_push(&vec, "foo");
+		strvec_push(&vec, "foo");

-	detached = strvec_detach(&vec);
-	check_str(detached[0], "foo");
-	check_pointer_eq(detached[1], NULL);
+		detached = strvec_detach(&vec);
+		check_str(detached[0], "foo");
+		check_pointer_eq(detached[1], NULL);

-	check_pointer_eq(vec.v, empty_strvec);
-	check_uint(vec.nr, ==, 0);
-	check_uint(vec.alloc, ==, 0);
+		check_pointer_eq(vec.v, empty_strvec);
+		check_uint(vec.nr, ==, 0);
+		check_uint(vec.alloc, ==, 0);

-	free((char *) detached[0]);
-	free(detached);
-}
+		free((char *) detached[0]);
+		free(detached);
+	}

-int cmd_main(int argc, const char **argv)
-{
-	TEST(t_static_init(), "static initialization");
-	TEST(t_dynamic_init(), "dynamic initialization");
-	TEST(t_clear(), "clear");
-	TEST(t_push(), "push");
-	TEST(t_pushf(), "pushf");
-	TEST(t_pushl(), "pushl");
-	TEST(t_pushv(), "pushv");
-	TEST(t_replace_at_head(), "replace at head");
-	TEST(t_replace_in_between(), "replace in between");
-	TEST(t_replace_at_tail(), "replace at tail");
-	TEST(t_replace_with_substring(), "replace with substring");
-	TEST(t_remove_at_head(), "remove at head");
-	TEST(t_remove_in_between(), "remove in between");
-	TEST(t_remove_at_tail(), "remove at tail");
-	TEST(t_pop_empty_array(), "pop with empty array");
-	TEST(t_pop_non_empty_array(), "pop with non-empty array");
-	TEST(t_split_empty_string(), "split empty string");
-	TEST(t_split_single_item(), "split single item");
-	TEST(t_split_multiple_items(), "split multiple items");
-	TEST(t_split_whitespace_only(), "split whitespace only");
-	TEST(t_split_multiple_consecutive_whitespaces(), "split multiple consecutive whitespaces");
-	TEST(t_detach(), "detach");
 	return test_done();
 }
--
2.46.0

Re: [PATCH v3 3/7] unit-tests: add for_test

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 3804 bytes --]

On Tue, Jul 30, 2024 at 04:00:07PM +0200, René Scharfe wrote:
> Am 30.07.24 um 06:49 schrieb Patrick Steinhardt:
> > On Mon, Jul 29, 2024 at 11:55:52AM -0700, Junio C Hamano wrote:
> >> Patrick Steinhardt <ps@pks.im> writes:
> >>
> >>> is standalone nowadays and called "clar" [1]. The biggest downside of it
> >>> is that it depends on Python to auto-generate the test "main" function,
> >>> which is not really a good fit for the Git project.
> >>
> >> Is that because Python is optional (like, we only use it for
> >> optional Perforce thing and import-zip in contrib), or are there
> >> other concerns?
> >>
> >> Unlike these components, unit tests are not even for the end-user
> >> consumers, so if it is Python that you find it a blocker, I do not
> >> see a reason to reject it.  The thing looked like a simple script
> >> that does not use any tricky language construct or modules.
> >
> > No concerns other than adding another mandatory dependency for devs. We
> > already depend on Perl and Shell, so adding Python to the stack feels
> > suboptimal to me.
> >
> > As you say though, the script isn't all that involved. So it would also
> > be possible to port it to Perl if we want to do that, I guess.
> 
> From the point of view of a "minimal C unit testing framework" surely
> the implementation language with the best dependency story would be C.
> Perhaps it could then also test itself.  On the other hand, writing a
> string handling program in 2024 in C is probably not the smartest idea.

I certainly wouldn't want to do it ;) I think Perl would be perfectly
fine given that we already rely on it rather extensively in Git.

> Then again, generate.py uses a non-greedy regex to remove single-line
> comments, which seems wrong, and doesn't seem to support preprocessor
> directives like #if and #ifdef, so whole tests can't be disabled this
> way.

I guess it doesn't have to be perfect, just good enough. clar comes with
some niceties like being able to run only some tests, all tests of a
particular suite or to exclude certain tests:

    # Run only submodule::add tests.
    $ ./libgit2_tests -t -ssubmodule::add::
    TAP version 13
    # start of suite 1: submodule::add
    ok 1 - submodule::add::url_absolute
    ok 2 - submodule::add::url_relative
    ok 3 - submodule::add::url_relative_to_origin
    ok 4 - submodule::add::url_relative_to_workdir
    ok 5 - submodule::add::path_exists_in_index
    ok 6 - submodule::add::file_exists_in_index
    ok 7 - submodule::add::submodule_clone
    ok 8 - submodule::add::submodule_clone_into_nonempty_dir_succeeds
    ok 9 - submodule::add::submodule_clone_twice_fails
    1..9

    # Run only a single test in the submodule::add suite.
    $ ./libgit2_tests -t -ssubmodule::add::url_absolute
    TAP version 13
    # start of suite 1: submodule::add
    ok 1 - submodule::add::url_absolute
    1..1

    # Run all submodule tests, but exclude submodule::add.
    $ ./libgit2_tests -t -ssubmodule::  -xsubmodule::add::
    TAP version 13
    # start of suite 1: submodule::escape
    ok 1 - submodule::escape::from_gitdir
    ok 2 - submodule::escape::from_gitdir_windows
    # start of suite 2: submodule::init
    ok 3 - submodule::init::absolute_url
    ...

So I'm not sure whether it's all that important to be able to recognize
things like #if and #ifdef given that you can pick tests to run on the
command line.

The alternative would be of course to hook up LLVM and properly parse
the C sources. But that feels like overthinking it ;)

> And it uses pickle to cache results; does that mean it would be
> slow without it?

Good question. I plan to find some time this week to draft a PoC and
will remember to benchmark this.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3 3/7] unit-tests: add for_test

[René Scharfe]

Am 31.07.24 um 07:19 schrieb Patrick Steinhardt:
> On Tue, Jul 30, 2024 at 04:00:07PM +0200, René Scharfe wrote:
>>
>> From the point of view of a "minimal C unit testing framework" surely
>> the implementation language with the best dependency story would be C.
>> Perhaps it could then also test itself.  On the other hand, writing a
>> string handling program in 2024 in C is probably not the smartest idea.
>
> I certainly wouldn't want to do it ;) I think Perl would be perfectly
> fine given that we already rely on it rather extensively in Git.

Right.

> clar comes with
> some niceties like being able to run only some tests, all tests of a
> particular suite or to exclude certain tests:
>
>     # Run only submodule::add tests.
>     $ ./libgit2_tests -t -ssubmodule::add::
>     TAP version 13
>     # start of suite 1: submodule::add
>     ok 1 - submodule::add::url_absolute
>     ok 2 - submodule::add::url_relative
>     ok 3 - submodule::add::url_relative_to_origin
>     ok 4 - submodule::add::url_relative_to_workdir
>     ok 5 - submodule::add::path_exists_in_index
>     ok 6 - submodule::add::file_exists_in_index
>     ok 7 - submodule::add::submodule_clone
>     ok 8 - submodule::add::submodule_clone_into_nonempty_dir_succeeds
>     ok 9 - submodule::add::submodule_clone_twice_fails
>     1..9
>
>     # Run only a single test in the submodule::add suite.
>     $ ./libgit2_tests -t -ssubmodule::add::url_absolute
>     TAP version 13
>     # start of suite 1: submodule::add
>     ok 1 - submodule::add::url_absolute
>     1..1
>
>     # Run all submodule tests, but exclude submodule::add.
>     $ ./libgit2_tests -t -ssubmodule::  -xsubmodule::add::
>     TAP version 13
>     # start of suite 1: submodule::escape
>     ok 1 - submodule::escape::from_gitdir
>     ok 2 - submodule::escape::from_gitdir_windows
>     # start of suite 2: submodule::init
>     ok 3 - submodule::init::absolute_url
>     ...
>
> So I'm not sure whether it's all that important to be able to recognize
> things like #if and #ifdef given that you can pick tests to run on the
> command line.

My only reference are the regular (shell-based) tests, and those have
to deal with prerequisites like PTHREADS, PCRE or LIBCURL.  Such tests
wouldn't even compile, and would thus be #if'd out.  We'd have to move
those into separate files instead and skip the whole file, it seems.

> The alternative would be of course to hook up LLVM and properly parse
> the C sources. But that feels like overthinking it ;)

It's as simple as calling clang (or gcc) with the option -E.  We'd need
to disable -Wmissing-prototypes as well, though, as there will be no
declarations before we generate clar-decls.h.  Not sure how to do that
all in a portable way.

Anyway, another consequence is that adding, removing or renaming a
single test function requires rebuilding all unit tests, because
clar-decls.h changes and all of them include it.  Not a problem right
now.  If we ever get to the same number of test files as in the regular
test suite (ca. 1000) then this could become annoying.

Apropos, I wonder how a Clar-style t-ctype would look like, but didn't
dare to even start thinking about it, yet.

René

Re: [PATCH v4 1/6] t0080: use here-doc test body

[Kyle Lippincott]

On Tue, Jul 30, 2024 at 7:06 AM René Scharfe <l.s.r@web.de> wrote:
>
> Improve the readability of the expected output by using a here-doc for
> the test body and replacing the unwieldy ${SQ} references with literal
> single quotes.
>
> Signed-off-by: René Scharfe <l.s.r@web.de>
> ---
>  t/t0080-unit-test-output.sh | 20 ++++++++++----------
>  1 file changed, 10 insertions(+), 10 deletions(-)

I'm not an expert on git's style guide or testing infrastructure, but
this looks like a good improvement to me. Thanks!

Re: [PATCH v4 2/6] unit-tests: show location of checks outside of tests

[Kyle Lippincott]

On Tue, Jul 30, 2024 at 7:07 AM René Scharfe <l.s.r@web.de> wrote:
>
> Checks outside of tests are caught at runtime and reported like this:
>
>  Assertion failed: (ctx.running), function test_assert, file test-lib.c, line 267.
>
> The assert() call aborts the unit test and doesn't reveal the location
> or even the type of the offending check, as test_assert() is called by
> all of them.
>
> Handle it like the opposite case, a test without any checks: Don't
> abort, but report the location of the actual check, along with a message
> explaining the situation.  The output for example above becomes:
>
>  # BUG: check outside of test at t/helper/test-example-tap.c:75
>
> ... and the unit test program continues and indicates the error in its
> exit code at the end.
>
> Signed-off-by: René Scharfe <l.s.r@web.de>
> ---
>  t/helper/test-example-tap.c | 2 ++
>  t/t0080-unit-test-output.sh | 5 +++--
>  t/unit-tests/test-lib.c     | 7 ++++++-
>  3 files changed, 11 insertions(+), 3 deletions(-)
>
> diff --git a/t/helper/test-example-tap.c b/t/helper/test-example-tap.c
> index d072ad559f..79c12b01cd 100644
> --- a/t/helper/test-example-tap.c
> +++ b/t/helper/test-example-tap.c
> @@ -72,6 +72,8 @@ static void t_empty(void)
>
>  int cmd__example_tap(int argc, const char **argv)
>  {
> +       check(1);

Let's include a comment that describes why we have this outside of the
TEST() macros so that people don't try to "fix" it, and so that people
realize it's not meant to be a _good_ example :)

> +
>         test_res = TEST(check_res = check_int(1, ==, 1), "passing test");
>         TEST(t_res(1), "passing test and assertion return 1");
>         test_res = TEST(check_res = check_int(1, ==, 2), "failing test");
> diff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh
> index 9ec47b7360..fe221f3bdb 100755
> --- a/t/t0080-unit-test-output.sh
> +++ b/t/t0080-unit-test-output.sh
> @@ -7,9 +7,10 @@ TEST_PASSES_SANITIZE_LEAK=true
>
>  test_expect_success 'TAP output from unit tests' - <<\EOT
>         cat >expect <<-EOF &&
> +       # BUG: check outside of test at t/helper/test-example-tap.c:75
>         ok 1 - passing test
>         ok 2 - passing test and assertion return 1
> -       # check "1 == 2" failed at t/helper/test-example-tap.c:77
> +       # check "1 == 2" failed at t/helper/test-example-tap.c:79
>         #    left: 1
>         #   right: 2
>         not ok 3 - failing test
> @@ -46,7 +47,7 @@ test_expect_success 'TAP output from unit tests' - <<\EOT
>         #    left: '\\\\'
>         #   right: '\\''
>         not ok 17 - messages from failing string and char comparison
> -       # BUG: test has no checks at t/helper/test-example-tap.c:92
> +       # BUG: test has no checks at t/helper/test-example-tap.c:94
>         not ok 18 - test with no checks
>         ok 19 - test with no checks returns 0
>         1..19
> diff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c
> index 3c513ce59a..989dc758e6 100644
> --- a/t/unit-tests/test-lib.c
> +++ b/t/unit-tests/test-lib.c
> @@ -264,7 +264,12 @@ static void test_todo(void)
>
>  int test_assert(const char *location, const char *check, int ok)
>  {
> -       assert(ctx.running);
> +       if (!ctx.running) {
> +               test_msg("BUG: check outside of test at %s",
> +                        make_relative(location));

Below, `test_msg` emits a message like `skipping check '1 == 2' at
<loc>`. Should we include 'check' as part of the message here, or is
it not possible or not useful for some reason?

> +               ctx.failed = 1;
> +               return 0;
> +       }
>
>         if (ctx.result == RESULT_SKIP) {
>                 test_msg("skipping check '%s' at %s", check,
> --
> 2.46.0

Re: [PATCH v4 3/6] unit-tests: add if_test

[Kyle Lippincott]

On Tue, Jul 30, 2024 at 7:08 AM René Scharfe <l.s.r@web.de> wrote:
>
> The macro TEST only allows defining a test that consists of a single
> expression.  Add a new macro, if_test, which provides a way to define
> unit tests that are made up of one or more statements.
>
> if_test allows defining self-contained tests en bloc, a bit like
> test_expect_success does for regular tests.  It acts like a conditional;
> the test body is executed if test_skip_all() had not been called before.
>
> Signed-off-by: René Scharfe <l.s.r@web.de>
> ---
>  .clang-format               |  5 +++++
>  t/helper/test-example-tap.c | 33 +++++++++++++++++++++++++++++++++
>  t/t0080-unit-test-output.sh | 35 ++++++++++++++++++++++++++++++++++-
>  t/unit-tests/test-lib.c     | 29 +++++++++++++++++++++++++++++
>  t/unit-tests/test-lib.h     | 20 ++++++++++++++++++++
>  5 files changed, 121 insertions(+), 1 deletion(-)
>
> diff --git a/.clang-format b/.clang-format
> index 6408251577..4c6d317508 100644
> --- a/.clang-format
> +++ b/.clang-format
> @@ -169,6 +169,11 @@ ForEachMacros:
>    - 'strmap_for_each_entry'
>    - 'strset_for_each_entry'
>
> +# A list of macros that should be interpreted as conditionals instead of as
> +# function calls.
> +IfMacros:
> +  - 'if_test'
> +
>  # The maximum number of consecutive empty lines to keep.
>  MaxEmptyLinesToKeep: 1
>
> diff --git a/t/helper/test-example-tap.c b/t/helper/test-example-tap.c
> index 79c12b01cd..914af88e0a 100644
> --- a/t/helper/test-example-tap.c
> +++ b/t/helper/test-example-tap.c
> @@ -94,5 +94,38 @@ int cmd__example_tap(int argc, const char **argv)
>         test_res = TEST(t_empty(), "test with no checks");
>         TEST(check_int(test_res, ==, 0), "test with no checks returns 0");
>
> +       if_test ("if_test passing test")
> +               check_int(1, ==, 1);
> +       if_test ("if_test failing test")
> +               check_int(1, ==, 2);
> +       if_test ("if_test passing TEST_TODO()")
> +               TEST_TODO(check(0));
> +       if_test ("if_test failing TEST_TODO()")
> +               TEST_TODO(check(1));
> +       if_test ("if_test test_skip()") {
> +               check(0);
> +               test_skip("missing prerequisite");
> +               check(1);
> +       }
> +       if_test ("if_test test_skip() inside TEST_TODO()")
> +               TEST_TODO((test_skip("missing prerequisite"), 1));
> +       if_test ("if_test TEST_TODO() after failing check") {
> +               check(0);
> +               TEST_TODO(check(0));
> +       }
> +       if_test ("if_test failing check after TEST_TODO()") {
> +               check(1);
> +               TEST_TODO(check(0));
> +               check(0);
> +       }
> +       if_test ("if_test messages from failing string and char comparison") {
> +               check_str("\thello\\", "there\"\n");
> +               check_str("NULL", NULL);
> +               check_char('a', ==, '\n');
> +               check_char('\\', ==, '\'');
> +       }
> +       if_test ("if_test test with no checks")
> +               ; /* nothing */
> +
>         return test_done();
>  }
> diff --git a/t/t0080-unit-test-output.sh b/t/t0080-unit-test-output.sh
> index fe221f3bdb..3c369c88e2 100755
> --- a/t/t0080-unit-test-output.sh
> +++ b/t/t0080-unit-test-output.sh
> @@ -50,7 +50,40 @@ test_expect_success 'TAP output from unit tests' - <<\EOT
>         # BUG: test has no checks at t/helper/test-example-tap.c:94
>         not ok 18 - test with no checks
>         ok 19 - test with no checks returns 0
> -       1..19
> +       ok 20 - if_test passing test
> +       # check "1 == 2" failed at t/helper/test-example-tap.c:100
> +       #    left: 1
> +       #   right: 2
> +       not ok 21 - if_test failing test
> +       not ok 22 - if_test passing TEST_TODO() # TODO
> +       # todo check 'check(1)' succeeded at t/helper/test-example-tap.c:104
> +       not ok 23 - if_test failing TEST_TODO()
> +       # check "0" failed at t/helper/test-example-tap.c:106
> +       # skipping test - missing prerequisite
> +       # skipping check '1' at t/helper/test-example-tap.c:108
> +       ok 24 - if_test test_skip() # SKIP
> +       # skipping test - missing prerequisite
> +       ok 25 - if_test test_skip() inside TEST_TODO() # SKIP
> +       # check "0" failed at t/helper/test-example-tap.c:113
> +       not ok 26 - if_test TEST_TODO() after failing check
> +       # check "0" failed at t/helper/test-example-tap.c:119
> +       not ok 27 - if_test failing check after TEST_TODO()
> +       # check "!strcmp("\thello\\\\", "there\"\n")" failed at t/helper/test-example-tap.c:122
> +       #    left: "\011hello\\\\"
> +       #   right: "there\"\012"
> +       # check "!strcmp("NULL", NULL)" failed at t/helper/test-example-tap.c:123
> +       #    left: "NULL"
> +       #   right: NULL
> +       # check "'a' == '\n'" failed at t/helper/test-example-tap.c:124
> +       #    left: 'a'
> +       #   right: '\012'
> +       # check "'\\\\' == '\\''" failed at t/helper/test-example-tap.c:125
> +       #    left: '\\\\'
> +       #   right: '\\''
> +       not ok 28 - if_test messages from failing string and char comparison
> +       # BUG: test has no checks at t/helper/test-example-tap.c:127
> +       not ok 29 - if_test test with no checks
> +       1..29
>         EOF
>
>         ! test-tool example-tap >actual &&
> diff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c
> index 989dc758e6..fa1f95965c 100644
> --- a/t/unit-tests/test-lib.c
> +++ b/t/unit-tests/test-lib.c
> @@ -16,6 +16,8 @@ static struct {
>         unsigned running :1;
>         unsigned skip_all :1;
>         unsigned todo :1;
> +       char location[100];
> +       char description[100];
>  } ctx = {
>         .lazy_plan = 1,
>         .result = RESULT_NONE,
> @@ -125,6 +127,8 @@ void test_plan(int count)
>
>  int test_done(void)
>  {
> +       if (ctx.running && ctx.location[0] && ctx.description[0])
> +               test__run_end(1, ctx.location, "%s", ctx.description);
>         assert(!ctx.running);
>
>         if (ctx.lazy_plan)
> @@ -167,13 +171,38 @@ void test_skip_all(const char *format, ...)
>         va_end(ap);
>  }
>
> +void test__run_describe(const char *location, const char *format, ...)
> +{
> +       va_list ap;
> +       int len;
> +
> +       assert(ctx.running);
> +       assert(!ctx.location[0]);
> +       assert(!ctx.description[0]);
> +
> +       xsnprintf(ctx.location, sizeof(ctx.location), "%s",
> +                 make_relative(location));
> +
> +       va_start(ap, format);
> +       len = vsnprintf(ctx.description, sizeof(ctx.description), format, ap);
> +       va_end(ap);
> +       if (len < 0)
> +               die("unable to format message: %s", format);
> +       if (len >= sizeof(ctx.description))
> +               BUG("ctx.description too small to format %s", format);
> +}
> +
>  int test__run_begin(void)
>  {
> +       if (ctx.running && ctx.location[0] && ctx.description[0])
> +               test__run_end(1, ctx.location, "%s", ctx.description);
>         assert(!ctx.running);
>
>         ctx.count++;
>         ctx.result = RESULT_NONE;
>         ctx.running = 1;
> +       ctx.location[0] = '\0';
> +       ctx.description[0] = '\0';
>
>         return ctx.skip_all;
>  }
> diff --git a/t/unit-tests/test-lib.h b/t/unit-tests/test-lib.h
> index 2de6d715d5..f15dceb29e 100644
> --- a/t/unit-tests/test-lib.h
> +++ b/t/unit-tests/test-lib.h
> @@ -14,6 +14,23 @@
>         test__run_end(test__run_begin() ? 0 : (t, 1),   \
>                       TEST_LOCATION(),  __VA_ARGS__)
>
> +/*
> + * Run a test unless test_skip_all() has been called.  Acts like a
> + * conditional; the test body is expected as a statement or block after
> + * the closing parenthesis.  The description for each test should be
> + * unique.  E.g.:
> + *
> + *  if_test ("something else %d %d", arg1, arg2) {
> + *          prepare();
> + *          test_something_else(arg1, arg2);
> + *          cleanup();
> + *  }
> + */
> +#define if_test(...)                                                   \
> +       if (test__run_begin() ?                                         \
> +           (test__run_end(0, TEST_LOCATION(),  __VA_ARGS__), 0) :      \
> +           (test__run_describe(TEST_LOCATION(),  __VA_ARGS__), 1))
> +

I personally prefer this implementation much more than the `for_test`
implementation, thanks. I think the extra functionality/usability from
having `test__run_end` automatically called is also very useful.

I'm still not wild about the name, and I'm still not convinced of the
readability improvements. I still want to read `if_test(...)` as
having a condition in the parenthesis. Tests "skipping everything"
seems like an internal implementation detail that there's disagreement
on whether to expose to the test author/reader or not; maybe we should
redesign if we don't migrate to clar or some other test framework. Do
we get significant benefits from a test function continuing to execute
after the "skip everything" flag has been set?

Specifically, the only current user of `test_skip_all` is t-strbuf.c.
Maybe we do this in t-strbuf.c, and remove the "skip everything"
logic? Below, `test_skipping_remainder` does most of the stuff that
`test_skip_all` does, except:
- automatically calls test_done()
- "poison"s some state so that we can't call `test__run_begin`
anymore, if test author forgets to `return` from this

  int cmd_main(int argc, const char **argv)
  {
    if (!TEST(t_static_init(), "static initialization works"))
-     test_skip_all("STRBUF_INIT is broken");
+    return test_skipping_remainder("STRBUF_INIT is broken");
    TEST(t_dynamic_init(), "dynamic initialization works");
    TEST(setup(t_addch, "a"), "strbuf_addch adds char");
    TEST(setup(t_addch, ""), "strbuf_addch adds NUL char");
    TEST(setup_populated(t_addch, "initial value", "a"),
         "strbuf_addch appends to initial value");
    TEST(setup(t_addstr, "hello there"), "strbuf_addstr adds string");
    TEST(setup_populated(t_addstr, "initial value", "hello there"),
         "strbuf_addstr appends string to initial value");

    return test_done();
  }

With that out of the way, I think that `test__run_begin()` always
returns true, and calling `test__run_end()` isn't necessary, so this
would be able to become:

#define if_test(...) \
    if (test__run_describe(TEST_LOCATION(), __VA_ARGS__), 1)  /*
automatically calls test__run_begin() */

At that point, the `if` is just used to allow this syntax:

if_test("example if_test") {
    check_int(1, ==, 1);
    check_int(1, !=, 2);
}

But it's equivalent, since we don't have any conditional that we're
actually checking, to:

{
    if_test("example if_test");  /* note the semicolon; leaving it off
doesn't matter, as long as compiler doesn't complain about empty if
block */
    check_int(1, ==, 1);
    check_int(1, !=, 2);
}

Or even doing that without the {}. At that point, we can remove the
`if` aspect of the name, and get something like this, where you can
use {} blocks if needed or desired:

#define multiline_test(...) test__run_describe(TEST_LOCATION(), __VA_ARGS__)

multiline_test("equality");  /* semicolon required */
check_int(1, ==, 1);
check_int(2, ==, 2);
multiline_test("inequality");
check_int(1, !=, 2);
check_int(2, !=, 1);


>  /*
>   * Print a test plan, should be called before any tests. If the number
>   * of tests is not known in advance test_done() will automatically
> @@ -153,6 +170,9 @@ union test__tmp {
>
>  extern union test__tmp test__tmp[2];
>
> +__attribute__((format (printf, 2, 3)))
> +void test__run_describe(const char *, const char *, ...);
> +
>  int test__run_begin(void);
>  __attribute__((format (printf, 3, 4)))
>  int test__run_end(int, const char *, const char *, ...);
> --
> 2.46.0

Re: [PATCH v3 3/7] unit-tests: add for_test

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 3661 bytes --]

On Wed, Jul 31, 2024 at 06:48:33PM +0200, René Scharfe wrote:
> Am 31.07.24 um 07:19 schrieb Patrick Steinhardt:
> > On Tue, Jul 30, 2024 at 04:00:07PM +0200, René Scharfe wrote:
> Apropos, I wonder how a Clar-style t-ctype would look like, but didn't
> dare to even start thinking about it, yet.

Oh, I already had it converted locally, but didn't end up sending it out
to keep things focussed for now. Below diff is how it could look like.

Patrick

diff --git a/Makefile b/Makefile
index cfdce5770b..ed67feb32e 100644
--- a/Makefile
+++ b/Makefile
@@ -1332,13 +1332,13 @@ THIRD_PARTY_SOURCES += compat/regex/%
 THIRD_PARTY_SOURCES += sha1collisiondetection/%
 THIRD_PARTY_SOURCES += sha1dc/%
 
+UNIT_TESTS_SUITES += ctype
 UNIT_TESTS_SUITES += strvec
 UNIT_TESTS_PROG = $(UNIT_TEST_BIN)/unit-tests$X
 UNIT_TESTS_OBJS = $(patsubst %,$(UNIT_TEST_DIR)/%.o,$(UNIT_TESTS_SUITES))
 UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/clar/clar.o
 UNIT_TESTS_OBJS += $(UNIT_TEST_DIR)/unit-test.o
 
-UNIT_TEST_PROGRAMS += t-ctype
 UNIT_TEST_PROGRAMS += t-example-decorate
 UNIT_TEST_PROGRAMS += t-hash
 UNIT_TEST_PROGRAMS += t-mem-pool
diff --git a/t/unit-tests/t-ctype.c b/t/unit-tests/ctype.c
similarity index 70%
rename from t/unit-tests/t-ctype.c
rename to t/unit-tests/ctype.c
index d6ac1fe678..2f6696e828 100644
--- a/t/unit-tests/t-ctype.c
+++ b/t/unit-tests/ctype.c
@@ -1,18 +1,12 @@
-#include "test-lib.h"
+#include "unit-test.h"
 
 #define TEST_CHAR_CLASS(class, string) do { \
 	size_t len = ARRAY_SIZE(string) - 1 + \
 		BUILD_ASSERT_OR_ZERO(ARRAY_SIZE(string) > 0) + \
 		BUILD_ASSERT_OR_ZERO(sizeof(string[0]) == sizeof(char)); \
-	int skip = test__run_begin(); \
-	if (!skip) { \
-		for (int i = 0; i < 256; i++) { \
-			if (!check_int(class(i), ==, !!memchr(string, i, len)))\
-				test_msg("      i: 0x%02x", i); \
-		} \
-		check(!class(EOF)); \
-	} \
-	test__run_end(!skip, TEST_LOCATION(), #class " works"); \
+	for (int i = 0; i < 256; i++) \
+		cl_assert_equal_i(class(i), !!memchr(string, i, len)); \
+	cl_assert(!class(EOF)); \
 } while (0)
 
 #define DIGIT "0123456789"
@@ -33,21 +27,72 @@
 	"\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f" \
 	"\x7f"
 
-int cmd_main(int argc, const char **argv) {
+void test_ctype__isspace(void)
+{
 	TEST_CHAR_CLASS(isspace, " \n\r\t");
+}
+
+void test_TEST_CHAR_ctype__CLASSisdigit(void)
+{
 	TEST_CHAR_CLASS(isdigit, DIGIT);
+}
+
+void test_TEST_CHAR_ctype__CLASSisalpha(void)
+{
 	TEST_CHAR_CLASS(isalpha, LOWER UPPER);
+}
+
+void test_ctype__isalnum(void)
+{
 	TEST_CHAR_CLASS(isalnum, LOWER UPPER DIGIT);
+}
+
+void test_is_glob_ctype__special(void)
+{
 	TEST_CHAR_CLASS(is_glob_special, "*?[\\");
+}
+
+void test_is_regex_ctype__special(void)
+{
 	TEST_CHAR_CLASS(is_regex_special, "$()*+.?[\\^{|");
+}
+
+void test_is_pathspec_ctype__magic(void)
+{
 	TEST_CHAR_CLASS(is_pathspec_magic, "!\"#%&',-/:;<=>@_`~");
+}
+
+void test_ctype__isascii(void)
+{
 	TEST_CHAR_CLASS(isascii, ASCII);
+}
+
+void test_ctype__islower(void)
+{
 	TEST_CHAR_CLASS(islower, LOWER);
+}
+
+void test_ctype__isupper(void)
+{
 	TEST_CHAR_CLASS(isupper, UPPER);
+}
+
+void test_ctype__iscntrl(void)
+{
 	TEST_CHAR_CLASS(iscntrl, CNTRL);
+}
+
+void test_ctype__ispunct(void)
+{
 	TEST_CHAR_CLASS(ispunct, PUNCT);
+}
+
+void test_ctype__isxdigit(void)
+{
 	TEST_CHAR_CLASS(isxdigit, DIGIT "abcdefABCDEF");
-	TEST_CHAR_CLASS(isprint, LOWER UPPER DIGIT PUNCT " ");
+}
 
-	return test_done();
+void test_ctype__isprint(void)
+{
+	TEST_CHAR_CLASS(isprint, LOWER UPPER DIGIT PUNCT " ");
 }


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 2/6] unit-tests: show location of checks outside of tests

[René Scharfe]

Am 31.07.24 um 23:03 schrieb Kyle Lippincott:
> On Tue, Jul 30, 2024 at 7:07 AM René Scharfe <l.s.r@web.de> wrote:
>>
>> diff --git a/t/helper/test-example-tap.c b/t/helper/test-example-tap.c
>> index d072ad559f..79c12b01cd 100644
>> --- a/t/helper/test-example-tap.c
>> +++ b/t/helper/test-example-tap.c
>> @@ -72,6 +72,8 @@ static void t_empty(void)
>>
>>  int cmd__example_tap(int argc, const char **argv)
>>  {
>> +       check(1);
>
> Let's include a comment that describes why we have this outside of the
> TEST() macros so that people don't try to "fix" it, and so that people
> realize it's not meant to be a _good_ example :)

Well, the other examples have no such comments, either, but they are
tests (using TEST or if_test), so they at least have descriptions.  So
fair enough.

>> diff --git a/t/unit-tests/test-lib.c b/t/unit-tests/test-lib.c
>> index 3c513ce59a..989dc758e6 100644
>> --- a/t/unit-tests/test-lib.c
>> +++ b/t/unit-tests/test-lib.c
>> @@ -264,7 +264,12 @@ static void test_todo(void)
>>
>>  int test_assert(const char *location, const char *check, int ok)
>>  {
>> -       assert(ctx.running);
>> +       if (!ctx.running) {
>> +               test_msg("BUG: check outside of test at %s",
>> +                        make_relative(location));
>
> Below, `test_msg` emits a message like `skipping check '1 == 2' at
> <loc>`. Should we include 'check' as part of the message here, or is
> it not possible or not useful for some reason?

It's possible, of course.  Didn't do it because I didn't consider that
we might have multiple checks per line and the complementary "test has
no checks" message  only mentions the line number.  The latter is
followed by the test description in the next line, though.  So yeah,
good point.

>
>> +               ctx.failed = 1;
>> +               return 0;
>> +       }
>>
>>         if (ctx.result == RESULT_SKIP) {
>>                 test_msg("skipping check '%s' at %s", check,
>> --
>> 2.46.0

Re: [PATCH v4 3/6] unit-tests: add if_test

[René Scharfe]

Am 01.08.24 um 00:04 schrieb Kyle Lippincott:
>
> I'm still not wild about the name, and I'm still not convinced of the
> readability improvements. I still want to read `if_test(...)` as
> having a condition in the parenthesis. Tests "skipping everything"
> seems like an internal implementation detail that there's disagreement
> on whether to expose to the test author/reader or not; maybe we should
> redesign if we don't migrate to clar or some other test framework. Do
> we get significant benefits from a test function continuing to execute
> after the "skip everything" flag has been set?

The idea is that the unit test framework may allow to skip individual
tests in the future; see
https://lore.kernel.org/git/c2de7f4e-e5dc-4324-8238-1d06795a2107@gmail.com/

René

Re: [PATCH v4 3/6] unit-tests: add if_test

[Kyle Lippincott]

On Thu, Aug 1, 2024 at 12:32 AM René Scharfe <l.s.r@web.de> wrote:
>
> Am 01.08.24 um 00:04 schrieb Kyle Lippincott:
> >
> > I'm still not wild about the name, and I'm still not convinced of the
> > readability improvements. I still want to read `if_test(...)` as
> > having a condition in the parenthesis. Tests "skipping everything"
> > seems like an internal implementation detail that there's disagreement
> > on whether to expose to the test author/reader or not; maybe we should
> > redesign if we don't migrate to clar or some other test framework. Do
> > we get significant benefits from a test function continuing to execute
> > after the "skip everything" flag has been set?
>
> The idea is that the unit test framework may allow to skip individual
> tests in the future; see
> https://lore.kernel.org/git/c2de7f4e-e5dc-4324-8238-1d06795a2107@gmail.com/

I see. This functionality isn't currently available, but we may want
it in the future. If that future ever happens, then we may be able to
avoid some cleanup if we keep the setup the way it is, where
`test__run_start` returns whether or not the test actually started.
This future `--run` flag for the unit tests would be based on either
(a) position in the file, like the integration tests, or (b) some
substring of the description. Since the tests don't have a name,
there's no ability for `--run` to operate based on that. Ok.

I still don't like the name, but I might be in the minority on this.
Yes, there's a control structure in use to make this work with the
syntax you desire, but unlike with `for_test`, there's many fewer edge
cases added because of it: `break` and `continue` won't do anything
with `if_test`. I suppose that we would still need to be concerned
about the possibility of someone writing this:

if (condition)
    if_test("...")
        check_int(1, ==, 1);
else
    ...

I'm also still not personally sold on the value here.
Single-expression tests can be written with `TEST(check_int(1, ==, 1),
"equality")`, and having a single large function with all
multi-statement tests in it is harder to follow, in my opinion, than
the alternative where the tests are in a separate function. I can only
really think of the case where the tests have inter-dependencies (ex:
test 2 sets some state that test 5 relies on) for wanting them all in
the same function, because then that dependency becomes a bit more
obvious. However, getting that inter-test dependency to _not_ happen
actually seems like a worthwhile goal (except for cases like strbuf,
where if strbuf is inherently broken, we need to check that first and
bail out). There's increased risk that someone will write a test like
this (purely hypothetical example of some state being carried from one
test to another; yes, you can accomplish the same terribleness with
global variables, but more people have immediate negative reactions
about globals):

int cmd_main(int argc UNUSED, const char **argv UNUSED)
{
    char tmp_path[PATH_MAX];
    if_test("alpha") {
        /* perform basic functionality checks, like strbuf does */
    }
    if_test("beta") {
        /* or some other "expensive" initialization we don't want to
do if everything is broken */
        sprintf("/tmp/tmp.%d", rand());
        do_something(tmp_path, ...);
    }
    if_test("gamma") {
        /*
         * if beta didn't run, or if tests get reordered such that
gamma executes before beta,
         * things are going to go very poorly here
        */
        do_something_else(tmp_path, ...);
    }
    return tests_done();
}

That said, I'm still not opposed enough to the idea to veto (plus I
don't have anywhere near that kind of power, and shouldn't :)). I just
don't think I'd ever use it.