* [PATCH v2 1/3] selftests/nolibc: add a standalone test report macro
2023-06-19 6:52 [PATCH v2 0/3] selftests/nolibc: improve test report support Zhangjin Wu
@ 2023-06-19 6:56 ` Zhangjin Wu
2023-06-19 6:57 ` [PATCH v2 2/3] selftests/nolibc: always print the path to test log file Zhangjin Wu
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Zhangjin Wu @ 2023-06-19 6:56 UTC (permalink / raw)
To: w; +Cc: arnd, falcon, linux-kernel, linux-kselftest, thomas
The run-user, run and rerun targets use the same test report script,
let's add a standalone test report macro for them.
This shrinks code lines and simplify the future maintainability.
Suggested-by: Willy Tarreau <w@1wt.eu>
Link: https://lore.kernel.org/lkml/ZIB792FtG6ibOudp@1wt.eu/
Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
---
tools/testing/selftests/nolibc/Makefile | 19 +++++++------------
1 file changed, 7 insertions(+), 12 deletions(-)
diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 1b7b3c82f8ad..262a9f21d1b4 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -84,6 +84,10 @@ CFLAGS ?= -Os -fno-ident -fno-asynchronous-unwind-tables -std=c89 \
$(CFLAGS_$(ARCH)) $(CFLAGS_STACKPROTECTOR)
LDFLAGS := -s
+REPORT ?= awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \
+ END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
+ if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }'
+
help:
@echo "Supported targets under selftests/nolibc:"
@echo " all call the \"run\" target below"
@@ -131,10 +135,7 @@ libc-test: nolibc-test.c
# qemu user-land test
run-user: nolibc-test
$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
- $(Q)awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \
- END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
- if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \
- $(CURDIR)/run.out
+ $(Q)$(REPORT) $(CURDIR)/run.out
initramfs: nolibc-test
$(QUIET_MKDIR)mkdir -p initramfs
@@ -150,18 +151,12 @@ kernel: initramfs
# run the tests after building the kernel
run: kernel
$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
- $(Q)awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \
- END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
- if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \
- $(CURDIR)/run.out
+ $(Q)$(REPORT) $(CURDIR)/run.out
# re-run the tests from an existing kernel
rerun:
$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
- $(Q)awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \
- END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
- if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \
- $(CURDIR)/run.out
+ $(Q)$(REPORT) $(CURDIR)/run.out
clean:
$(call QUIET_CLEAN, sysroot)
--
2.25.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v2 2/3] selftests/nolibc: always print the path to test log file
2023-06-19 6:52 [PATCH v2 0/3] selftests/nolibc: improve test report support Zhangjin Wu
2023-06-19 6:56 ` [PATCH v2 1/3] selftests/nolibc: add a standalone test report macro Zhangjin Wu
@ 2023-06-19 6:57 ` Zhangjin Wu
2023-06-19 7:01 ` [PATCH v2 3/3] selftests/nolibc: restore the failed tests print Zhangjin Wu
2023-07-02 16:43 ` [PATCH v2 0/3] selftests/nolibc: improve test report support Willy Tarreau
3 siblings, 0 replies; 6+ messages in thread
From: Zhangjin Wu @ 2023-06-19 6:57 UTC (permalink / raw)
To: w; +Cc: arnd, falcon, linux-kernel, linux-kselftest, thomas
Even when there is no failure, developers may be still interested in the
test log file, especially, string alignment, duplicated print, kernel
message and so forth, so, always print the path to test log file.
A new line is added for such a print to avoid annoying people who don't
care about it when the test pass completely.
Suggested-by: Willy Tarreau <w@1wt.eu>
Link: https://lore.kernel.org/lkml/ZIB792FtG6ibOudp@1wt.eu/
Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
---
tools/testing/selftests/nolibc/Makefile | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 262a9f21d1b4..2a0c3f4fa204 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -85,8 +85,8 @@ CFLAGS ?= -Os -fno-ident -fno-asynchronous-unwind-tables -std=c89 \
LDFLAGS := -s
REPORT ?= awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \
- END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
- if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }'
+ END{ printf("%d test(s) passed, %d skipped, %d failed.\n", p, s, f); \
+ printf("See all results in %s\n", ARGV[1]); }'
help:
@echo "Supported targets under selftests/nolibc:"
--
2.25.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v2 3/3] selftests/nolibc: restore the failed tests print
2023-06-19 6:52 [PATCH v2 0/3] selftests/nolibc: improve test report support Zhangjin Wu
2023-06-19 6:56 ` [PATCH v2 1/3] selftests/nolibc: add a standalone test report macro Zhangjin Wu
2023-06-19 6:57 ` [PATCH v2 2/3] selftests/nolibc: always print the path to test log file Zhangjin Wu
@ 2023-06-19 7:01 ` Zhangjin Wu
2023-07-02 16:43 ` [PATCH v2 0/3] selftests/nolibc: improve test report support Willy Tarreau
3 siblings, 0 replies; 6+ messages in thread
From: Zhangjin Wu @ 2023-06-19 7:01 UTC (permalink / raw)
To: w; +Cc: arnd, falcon, linux-kernel, linux-kselftest, thomas
The commit fa0df56a804b ("selftests/nolibc: also count skipped and
failed tests in output") added counting for the skipped and failed
tests, but also removed the 'FAIL' results print, let's restore it for
it really allow users to learn the failed details without opening the
log file.
Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
---
tools/testing/selftests/nolibc/Makefile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 2a0c3f4fa204..000621f21adc 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -84,7 +84,7 @@ CFLAGS ?= -Os -fno-ident -fno-asynchronous-unwind-tables -std=c89 \
$(CFLAGS_$(ARCH)) $(CFLAGS_STACKPROTECTOR)
LDFLAGS := -s
-REPORT ?= awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \
+REPORT ?= awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++;print} /\[SKIPPED\][\r]*$$/{s++} \
END{ printf("%d test(s) passed, %d skipped, %d failed.\n", p, s, f); \
printf("See all results in %s\n", ARGV[1]); }'
--
2.25.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH v2 0/3] selftests/nolibc: improve test report support
2023-06-19 6:52 [PATCH v2 0/3] selftests/nolibc: improve test report support Zhangjin Wu
` (2 preceding siblings ...)
2023-06-19 7:01 ` [PATCH v2 3/3] selftests/nolibc: restore the failed tests print Zhangjin Wu
@ 2023-07-02 16:43 ` Willy Tarreau
2023-07-03 16:24 ` Zhangjin Wu
3 siblings, 1 reply; 6+ messages in thread
From: Willy Tarreau @ 2023-07-02 16:43 UTC (permalink / raw)
To: Zhangjin Wu; +Cc: arnd, linux-kernel, linux-kselftest, thomas
Hi Zhangjin,
On Mon, Jun 19, 2023 at 02:52:31PM +0800, Zhangjin Wu wrote:
> Hi, Willy
>
> Here is the v2 of our old patchset about test report [1].
>
> The trailing '\r' fixup has been merged, so, here only resend the left
> parts with an additional patch to restore the failed tests print.
>
> This patchset is rebased on the dev.2023.06.14a branch of linux-rcu [2].
>
> Tests have passed for 'x86 run':
>
> 138 test(s) passed, 0 skipped, 0 failed.
> See all results in /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/run.out
(...)
> 2. selftests/nolibc: always print the path to test log file
>
> Always print the path to test log file, but move it to a new line to
> avoid annoying people when the test pass without any failures.
I'm still really missing the (s+f > 0) test I added which was a time saver
for me, because I could trivially check in the output reports which ones
were totally OK and which ones required attention. Sure I could also start
to grep for "passed," | grep -v " 0 skipped, 0 failed" but that's quite a
pain, really.
I'm going to merge your series anyway otherwise we'll continue to bikeshed
for many weeks and I know how annoying it is to keep unmerged series. But
I would like that we find a solution that satisfies everyone.
Maybe one possibility would be to add a "status" at the end of the line
that emits "success", "warning", "failure" depending on the highest level
reached like this:
138 test(s) passed, 0 skipped, 0 failed => status: success
136 test(s) passed, 2 skipped, 0 failed => status: warning
136 test(s) passed, 1 skipped, 1 failed => status: failure
This way it's easy to grep -v "status: success" or grep "status: failure"
to instantly get the corresponding details and also grep for them from
multiple files.
Thanks!
Willy
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH v2 0/3] selftests/nolibc: improve test report support
2023-07-02 16:43 ` [PATCH v2 0/3] selftests/nolibc: improve test report support Willy Tarreau
@ 2023-07-03 16:24 ` Zhangjin Wu
0 siblings, 0 replies; 6+ messages in thread
From: Zhangjin Wu @ 2023-07-03 16:24 UTC (permalink / raw)
To: w; +Cc: arnd, falcon, linux-kernel, linux-kselftest, thomas
> Hi Zhangjin,
>
> On Mon, Jun 19, 2023 at 02:52:31PM +0800, Zhangjin Wu wrote:
> > Hi, Willy
> >
> > Here is the v2 of our old patchset about test report [1].
> >
> > The trailing '\r' fixup has been merged, so, here only resend the left
> > parts with an additional patch to restore the failed tests print.
> >
> > This patchset is rebased on the dev.2023.06.14a branch of linux-rcu [2].
> >
> > Tests have passed for 'x86 run':
> >
> > 138 test(s) passed, 0 skipped, 0 failed.
> > See all results in /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/run.out
> (...)
> > 2. selftests/nolibc: always print the path to test log file
> >
> > Always print the path to test log file, but move it to a new line to
> > avoid annoying people when the test pass without any failures.
>
> I'm still really missing the (s+f > 0) test I added which was a time saver
> for me, because I could trivially check in the output reports which ones
> were totally OK and which ones required attention. Sure I could also start
> to grep for "passed," | grep -v " 0 skipped, 0 failed" but that's quite a
> pain, really.
>
> I'm going to merge your series anyway otherwise we'll continue to bikeshed
> for many weeks and I know how annoying it is to keep unmerged series. But
> I would like that we find a solution that satisfies everyone.
>
> Maybe one possibility would be to add a "status" at the end of the line
> that emits "success", "warning", "failure" depending on the highest level
> reached like this:
>
> 138 test(s) passed, 0 skipped, 0 failed => status: success
> 136 test(s) passed, 2 skipped, 0 failed => status: warning
> 136 test(s) passed, 1 skipped, 1 failed => status: failure
>
> This way it's easy to grep -v "status: success" or grep "status: failure"
> to instantly get the corresponding details and also grep for them from
> multiple files.
Ok, it will be a further step, based on your new awk script, it is not
that hard.
This should work:
REPORT ?= awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++;print} /\[SKIPPED\][\r]*$$/{s++} \
END{ printf("\n%d test(s): %d passed, %d skipped, %d failed => status: ", p+s+f, p, s, f); \
if (f) printf("failure\n"); else if (s) printf("warning\n"); else printf("success\n");; \
printf("\nSee all results in %s\n", ARGV[1]); }'
It reports something like this:
...
Total number of errors: 0
Exiting with status 0
143 test(s): 140 passed, 3 skipped, 0 failed => status: warning
See all results in /labs/linux-lab/src/linux-stable/tools/testing/selftests/nolibc/build/x86/run.out
Two newlines are added around the summary line, so, it is not crowded as
before. If this is ok, will send a new patch on your new branch.
Thanks very much.
Best regards,
Zhangjin
>
> Thanks!
> Willy
^ permalink raw reply [flat|nested] 6+ messages in thread