[PATCH 0/4] selftests/nolibc: fix up and improve test report

[PATCH 0/4] selftests/nolibc: fix up and improve test report

[Zhangjin Wu]

Hi, Willy

Thanks very much for your merge of the v3 generic part1 of rv32, just
tested your latest 20230604-nolibc-rv32+stkp6 branch, everything work
well except a trivial test report regression on the 'run' target.

Besides the fixup, a standalone test-report target added to share them
among run, run-user and re-run and allow independent test report check
via direct 'make test-report'.

Best regards,
Zhangjin
---

Zhangjin Wu (4):
  selftests/nolibc: add a test-report target
  selftests/nolibc: allow run test-report directly
  selftests/nolibc: always print the log file
  selftests/nolibc: fix up test-report for run target

 tools/testing/selftests/nolibc/Makefile | 30 ++++++++++++-------------
 1 file changed, 15 insertions(+), 15 deletions(-)

-- 
2.25.1

[PATCH 1/4] selftests/nolibc: add a test-report target

[Zhangjin Wu]

A standalone test-report target is added to let the run, run-user and
rerun targets share them.

Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
---
 tools/testing/selftests/nolibc/Makefile | 26 ++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index be4159837494..8149ace2938a 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -127,14 +127,18 @@ nolibc-test: nolibc-test.c sysroot/$(ARCH)/include
 libc-test: nolibc-test.c
 	$(QUIET_CC)$(CC) -o $@ $<
 
-# qemu user-land test
-run-user: nolibc-test
-	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
+test-report:
 	$(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \
 	         END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
 	         if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \
 	         $(CURDIR)/run.out
 
+# qemu user-land test
+_run-user: nolibc-test
+	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
+
+run-user: _run-user test-report
+
 initramfs: nolibc-test
 	$(QUIET_MKDIR)mkdir -p initramfs
 	$(call QUIET_INSTALL, initramfs/init)
@@ -147,20 +151,16 @@ kernel: initramfs
 	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) $(IMAGE_NAME) CONFIG_INITRAMFS_SOURCE=$(CURDIR)/initramfs
 
 # run the tests after building the kernel
-run: kernel
+_run: kernel
 	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
-	$(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \
-	         END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
-	         if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \
-	         $(CURDIR)/run.out
+
+run: _run test-report
 
 # re-run the tests from an existing kernel
-rerun:
+_rerun:
 	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
-	$(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \
-	         END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
-	         if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \
-	         $(CURDIR)/run.out
+
+rerun: _rerun test-report
 
 clean:
 	$(call QUIET_CLEAN, sysroot)
-- 
2.25.1

Re: [PATCH 1/4] selftests/nolibc: add a test-report target

[Willy Tarreau]

On Mon, Jun 05, 2023 at 11:48:52AM +0800, Zhangjin Wu wrote:
> A standalone test-report target is added to let the run, run-user and
> rerun targets share them.
> 
> Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
> ---
>  tools/testing/selftests/nolibc/Makefile | 26 ++++++++++++-------------
>  1 file changed, 13 insertions(+), 13 deletions(-)
> 
> diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
> index be4159837494..8149ace2938a 100644
> --- a/tools/testing/selftests/nolibc/Makefile
> +++ b/tools/testing/selftests/nolibc/Makefile
> @@ -127,14 +127,18 @@ nolibc-test: nolibc-test.c sysroot/$(ARCH)/include
>  libc-test: nolibc-test.c
>  	$(QUIET_CC)$(CC) -o $@ $<
>  
> -# qemu user-land test
> -run-user: nolibc-test
> -	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
> +test-report:
>  	$(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \
>  	         END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
>  	         if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \
>  	         $(CURDIR)/run.out
>  
> +# qemu user-land test
> +_run-user: nolibc-test
> +	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
> +
> +run-user: _run-user test-report
> +

This will not reliably work, there's no ordering here, nothing guarantees
that test-report will run *after* _run-user (e.g. make -j). Another
approach is needed if you want to factor this, but in general creating
sequences in makefiles is difficult and often more painful than having
3 times the same 3 lines.

Willy

Re: [PATCH 1/4] selftests/nolibc: add a test-report target

[Zhangjin Wu]

> On Mon, Jun 05, 2023 at 11:48:52AM +0800, Zhangjin Wu wrote:
> > A standalone test-report target is added to let the run, run-user and
> > rerun targets share them.
> > 
> > Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
> > ---
> >  tools/testing/selftests/nolibc/Makefile | 26 ++++++++++++-------------
> >  1 file changed, 13 insertions(+), 13 deletions(-)
> > 
> > diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
> > index be4159837494..8149ace2938a 100644
> > --- a/tools/testing/selftests/nolibc/Makefile
> > +++ b/tools/testing/selftests/nolibc/Makefile
> > @@ -127,14 +127,18 @@ nolibc-test: nolibc-test.c sysroot/$(ARCH)/include
> >  libc-test: nolibc-test.c
> >  	$(QUIET_CC)$(CC) -o $@ $<
> >  
> > -# qemu user-land test
> > -run-user: nolibc-test
> > -	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
> > +test-report:
> >  	$(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \
> >  	         END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
> >  	         if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \
> >  	         $(CURDIR)/run.out
> >  
> > +# qemu user-land test
> > +_run-user: nolibc-test
> > +	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
> > +
> > +run-user: _run-user test-report
> > +
> 
> This will not reliably work, there's no ordering here, nothing guarantees
> that test-report will run *after* _run-user (e.g. make -j). Another
> approach is needed if you want to factor this, but in general creating
> sequences in makefiles is difficult and often more painful than having
> 3 times the same 3 lines.
>

Ok, thanks, what about this?

    # LOG_REPORT: report the test results
    LOG_REPORT   := awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \
	                 END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
	                 printf(" See all results in %s\n", ARGV[1]); }'

    run-user: nolibc-test
	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
	$(Q)$(LOG_REPORT) $(CURDIR)/run.out

    run: kernel
	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
	$(Q)$(LOG_REPORT) $(CURDIR)/run.out

    rerun:
	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
	$(Q)$(LOG_REPORT) $(CURDIR)/run.out

Or we directly add a standalone test report script? something like
tools/testing/selftests/nolibc/report.sh

    #!/bin/sh
    #
    # report.sh -- report the test results of nolibc-test
    #
    
    LOG_FILE=$1
    [ ! -f "$LOG_FILE" ] && echo "Usage: $0 /path/to/run.out"
    
    awk '
        /\[OK\][\r]*$$/{ p++ }
        /\[FAIL\][\r]*$$/{ f++ }
        /\[SKIPPED\][\r]*$$/{ s++ }
    
        END {
            printf("%d test(s) passed, %d skipped, %d failed.", p, s, f);
            printf(" See all results in %s\n", ARGV[1]);
        }' $LOG_FILE

And use it like this:

    LOG_REPORT           = $(CURDIR)/report.sh

Best regards,
Zhangjin

> Willy

Re: [PATCH 1/4] selftests/nolibc: add a test-report target

[Zhangjin Wu]

Hi, Willy

> > On Mon, Jun 05, 2023 at 11:48:52AM +0800, Zhangjin Wu wrote:
> > > A standalone test-report target is added to let the run, run-user and
> > > rerun targets share them.
> > > 
> > > Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
> > > ---
> > >  tools/testing/selftests/nolibc/Makefile | 26 ++++++++++++-------------
> > >  1 file changed, 13 insertions(+), 13 deletions(-)
> > > 
> > > diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
> > > index be4159837494..8149ace2938a 100644
> > > --- a/tools/testing/selftests/nolibc/Makefile
> > > +++ b/tools/testing/selftests/nolibc/Makefile
> > > @@ -127,14 +127,18 @@ nolibc-test: nolibc-test.c sysroot/$(ARCH)/include
> > >  libc-test: nolibc-test.c
> > >  	$(QUIET_CC)$(CC) -o $@ $<
> > >  
> > > -# qemu user-land test
> > > -run-user: nolibc-test
> > > -	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
> > > +test-report:
> > >  	$(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \
> > >  	         END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
> > >  	         if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \
> > >  	         $(CURDIR)/run.out
> > >  
> > > +# qemu user-land test
> > > +_run-user: nolibc-test
> > > +	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
> > > +
> > > +run-user: _run-user test-report
> > > +
> > 
> > This will not reliably work, there's no ordering here, nothing guarantees
> > that test-report will run *after* _run-user (e.g. make -j). Another
> > approach is needed if you want to factor this, but in general creating
> > sequences in makefiles is difficult and often more painful than having
> > 3 times the same 3 lines.
> >
> 
> Ok, thanks, what about this?
> 
>     # LOG_REPORT: report the test results
>     LOG_REPORT   := awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \
> 	                 END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
> 	                 printf(" See all results in %s\n", ARGV[1]); }'
> 
>     run-user: nolibc-test
> 	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
> 	$(Q)$(LOG_REPORT) $(CURDIR)/run.out
> 
>     run: kernel
> 	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
> 	$(Q)$(LOG_REPORT) $(CURDIR)/run.out
> 
>     rerun:
> 	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
> 	$(Q)$(LOG_REPORT) $(CURDIR)/run.out
> 
> Or we directly add a standalone test report script? something like
> tools/testing/selftests/nolibc/report.sh
> 
>     #!/bin/sh
>     #
>     # report.sh -- report the test results of nolibc-test
>     #
>     
>     LOG_FILE=$1
>     [ ! -f "$LOG_FILE" ] && echo "Usage: $0 /path/to/run.out"
>     
>     awk '
>         /\[OK\][\r]*$$/{ p++ }
>         /\[FAIL\][\r]*$$/{ f++ }
>         /\[SKIPPED\][\r]*$$/{ s++ }
>     
>         END {
>             printf("%d test(s) passed, %d skipped, %d failed.", p, s, f);
>             printf(" See all results in %s\n", ARGV[1]);
>         }' $LOG_FILE
> 
> And use it like this:
> 
>     LOG_REPORT           = $(CURDIR)/report.sh
>

I plan to renew this patchset, which one of the above methods do you
prefer?

For the always print statement:

    printf(" See all results in %s\n", ARGV[1]); }'

I will paste the reason why I need it, as mentioned in [1], if you still
need a clean test report, I will give up this change ;-)

Thanks,
Zhangjin
---
[1]: https://lore.kernel.org/linux-riscv/20230605070508.153407-1-falcon@tinylab.org/ 

> Best regards,
> Zhangjin
>
> > Willy

Re: [PATCH 1/4] selftests/nolibc: add a test-report target

[Willy Tarreau]

Hi Zhangjin,

On Wed, Jun 07, 2023 at 01:52:00PM +0800, Zhangjin Wu wrote:
> Hi, Willy
> 
> > > On Mon, Jun 05, 2023 at 11:48:52AM +0800, Zhangjin Wu wrote:
> > > > A standalone test-report target is added to let the run, run-user and
> > > > rerun targets share them.
> > > > 
> > > > Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
> > > > ---
> > > >  tools/testing/selftests/nolibc/Makefile | 26 ++++++++++++-------------
> > > >  1 file changed, 13 insertions(+), 13 deletions(-)
> > > > 
> > > > diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
> > > > index be4159837494..8149ace2938a 100644
> > > > --- a/tools/testing/selftests/nolibc/Makefile
> > > > +++ b/tools/testing/selftests/nolibc/Makefile
> > > > @@ -127,14 +127,18 @@ nolibc-test: nolibc-test.c sysroot/$(ARCH)/include
> > > >  libc-test: nolibc-test.c
> > > >  	$(QUIET_CC)$(CC) -o $@ $<
> > > >  
> > > > -# qemu user-land test
> > > > -run-user: nolibc-test
> > > > -	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
> > > > +test-report:
> > > >  	$(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \
> > > >  	         END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
> > > >  	         if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \
> > > >  	         $(CURDIR)/run.out
> > > >  
> > > > +# qemu user-land test
> > > > +_run-user: nolibc-test
> > > > +	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
> > > > +
> > > > +run-user: _run-user test-report
> > > > +
> > > 
> > > This will not reliably work, there's no ordering here, nothing guarantees
> > > that test-report will run *after* _run-user (e.g. make -j). Another
> > > approach is needed if you want to factor this, but in general creating
> > > sequences in makefiles is difficult and often more painful than having
> > > 3 times the same 3 lines.
> > >
> > 
> > Ok, thanks, what about this?
> > 
> >     # LOG_REPORT: report the test results
> >     LOG_REPORT   := awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \
> > 	                 END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
> > 	                 printf(" See all results in %s\n", ARGV[1]); }'
> > 
> >     run-user: nolibc-test
> > 	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
> > 	$(Q)$(LOG_REPORT) $(CURDIR)/run.out
> > 
> >     run: kernel
> > 	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
> > 	$(Q)$(LOG_REPORT) $(CURDIR)/run.out
> > 
> >     rerun:
> > 	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
> > 	$(Q)$(LOG_REPORT) $(CURDIR)/run.out
> > 
> > Or we directly add a standalone test report script? something like
> > tools/testing/selftests/nolibc/report.sh
> > 
> >     #!/bin/sh
> >     #
> >     # report.sh -- report the test results of nolibc-test
> >     #
> >     
> >     LOG_FILE=$1
> >     [ ! -f "$LOG_FILE" ] && echo "Usage: $0 /path/to/run.out"
> >     
> >     awk '
> >         /\[OK\][\r]*$$/{ p++ }
> >         /\[FAIL\][\r]*$$/{ f++ }
> >         /\[SKIPPED\][\r]*$$/{ s++ }
> >     
> >         END {
> >             printf("%d test(s) passed, %d skipped, %d failed.", p, s, f);
> >             printf(" See all results in %s\n", ARGV[1]);
> >         }' $LOG_FILE
> > 
> > And use it like this:
> > 
> >     LOG_REPORT           = $(CURDIR)/report.sh
> >
> 
> I plan to renew this patchset, which one of the above methods do you
> prefer?

IFF it needs to be done I prefer the macro in the Makefile to avoid
depending on external scripts that are useless outside of the makefile.
BUT, my point remains that I adopted this so that I could quickly and
visually check that everything was OK. I'm fine with any other method
but I do not want to have to carefully read all these lines to make
sure I'm not mixing a "8" with a "0" (I'm mentioning this one because
it's exactly the one I had when I decided to add the extra values).
For example if you prepend "FAILURE: ", "WARNING: ", "SUCCESS: " in
front of these lines to summarize them depending on the highest level
encountered (success, skipped, failed), then I'm fine because it's
easy to check that all lines show the same word.

> For the always print statement:
> 
>     printf(" See all results in %s\n", ARGV[1]); }'

Then please put it on its own line without the leading space, this
will be even more readable.

> I will paste the reason why I need it, as mentioned in [1], if you still
> need a clean test report, I will give up this change ;-)

No worries, I don't want to be annoying if you need something, but I
don't want to be annoyed by changes either :-)

thanks,
Willy

Re: [PATCH 1/4] selftests/nolibc: add a test-report target

[Zhangjin Wu]

On Wed, Jun 07, 2023 at 01:52:00PM +0800, Zhangjin Wu wrote:
> > Hi, Willy
> > 
> (...)
> > > 
> > > Ok, thanks, what about this?
> > > 
> > >     # LOG_REPORT: report the test results
> > >     LOG_REPORT   := awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \
> > > 	                 END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
> > > 	                 printf(" See all results in %s\n", ARGV[1]); }'
> > > 
> > >     run-user: nolibc-test
> > > 	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
> > > 	$(Q)$(LOG_REPORT) $(CURDIR)/run.out
> > > 
> > >     run: kernel
> > > 	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
> > > 	$(Q)$(LOG_REPORT) $(CURDIR)/run.out
> > > 
> > >     rerun:
> > > 	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
> > > 	$(Q)$(LOG_REPORT) $(CURDIR)/run.out
> > > 
> > > Or we directly add a standalone test report script? something like
> > > tools/testing/selftests/nolibc/report.sh
> > > 
> > >     #!/bin/sh
> > >     #
> > >     # report.sh -- report the test results of nolibc-test
> > >     #
> > >     
> > >     LOG_FILE=$1
> > >     [ ! -f "$LOG_FILE" ] && echo "Usage: $0 /path/to/run.out"
> > >     
> > >     awk '
> > >         /\[OK\][\r]*$$/{ p++ }
> > >         /\[FAIL\][\r]*$$/{ f++ }
> > >         /\[SKIPPED\][\r]*$$/{ s++ }
> > >     
> > >         END {
> > >             printf("%d test(s) passed, %d skipped, %d failed.", p, s, f);
> > >             printf(" See all results in %s\n", ARGV[1]);
> > >         }' $LOG_FILE
> > > 
> > > And use it like this:
> > > 
> > >     LOG_REPORT           = $(CURDIR)/report.sh
> > >
> > 
> > I plan to renew this patchset, which one of the above methods do you
> > prefer?
> 
> IFF it needs to be done I prefer the macro in the Makefile to avoid
> depending on external scripts that are useless outside of the makefile.
> BUT, my point remains that I adopted this so that I could quickly and
> visually check that everything was OK. I'm fine with any other method
> but I do not want to have to carefully read all these lines to make
> sure I'm not mixing a "8" with a "0" (I'm mentioning this one because
> it's exactly the one I had when I decided to add the extra values).
> For example if you prepend "FAILURE: ", "WARNING: ", "SUCCESS: " in
> front of these lines to summarize them depending on the highest level
> encountered (success, skipped, failed), then I'm fine because it's
> easy to check that all lines show the same word.
> 

Ok.

> > For the always print statement:
> > 
> >     printf(" See all results in %s\n", ARGV[1]); }'
> 
> Then please put it on its own line without the leading space, this
> will be even more readable.
>

It is a good balance.

This may be more useful if we run this from the kselftest framework,
seems it is not able to run it from the 'kselftest' target currently,
here shares something I have tried:

I tried something like this, seems run-user works, but defconfig and run
not.

    diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
    index 4a3a105e1fdf..70693f6501c6 100644
    --- a/tools/testing/selftests/nolibc/Makefile
    +++ b/tools/testing/selftests/nolibc/Makefile
    @@ -83,6 +83,8 @@ CFLAGS  ?= -Os -fno-ident -fno-asynchronous-unwind-tables -std=c89 \
                    $(CFLAGS_$(ARCH)) $(CFLAGS_STACKPROTECTOR)
     LDFLAGS := -s
   
    +NOLIBC_SUBTARGETS ?= run-user
    +
     help:
            @echo "Supported targets under selftests/nolibc:"
            @echo "  all          call the \"run\" target below"
    @@ -110,7 +112,9 @@ help:
            @echo "  IMAGE_NAME    = $(if $(IMAGE_NAME),$(IMAGE_NAME),UNKNOWN_ARCH) [determined from \$$ARCH]"
            @echo ""

    -all: run
    +run_tests: $(NOLIBC_SUBTARGETS)
    +
    +all: $(NOLIBC_SUBTARGETS)

This can be triggered from the top-level kselftest framework:

    $ make -C /path/to/linux-stable kselftest TARGETS=nolibc NOLIBC_SUBTARGETS=run-user

And seems we still not support O= currently either.

> > I will paste the reason why I need it, as mentioned in [1], if you still
> > need a clean test report, I will give up this change ;-)
> 
> No worries, I don't want to be annoying if you need something, but I
> don't want to be annoyed by changes either :-)
> 

Thanks,
Zhangjin

> thanks,
> Willy

[PATCH 2/4] selftests/nolibc: allow run test-report directly

[Zhangjin Wu]

This allows to re-check the test report via 'make test-report' without
the requirement to re-run the dependency targets.

Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
---
 tools/testing/selftests/nolibc/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 8149ace2938a..5b0af8d8f5f3 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -128,10 +128,10 @@ libc-test: nolibc-test.c
 	$(QUIET_CC)$(CC) -o $@ $<
 
 test-report:
-	$(Q)awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \
+	$(Q)[ -f $(CURDIR)/run.out ] && awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \
 	         END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
 	         if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \
-	         $(CURDIR)/run.out
+		 $(CURDIR)/run.out || :
 
 # qemu user-land test
 _run-user: nolibc-test
-- 
2.25.1

[PATCH 3/4] selftests/nolibc: always print the log file

[Zhangjin Wu]

This allows to check the other issues of the output file manually even
when all of them passed.

Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
---
 tools/testing/selftests/nolibc/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 5b0af8d8f5f3..518f85c77fe3 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -130,7 +130,7 @@ libc-test: nolibc-test.c
 test-report:
 	$(Q)[ -f $(CURDIR)/run.out ] && awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \
 	         END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
-	         if (s+f > 0) printf(" See all results in %s\n", ARGV[1]); else print; }' \
+	         printf(" See all results in %s\n", ARGV[1]); }' \
 		 $(CURDIR)/run.out || :
 
 # qemu user-land test
-- 
2.25.1

Re: [PATCH 3/4] selftests/nolibc: always print the log file

[Willy Tarreau]

On Mon, Jun 05, 2023 at 11:57:44AM +0800, Zhangjin Wu wrote:
> This allows to check the other issues of the output file manually even
> when all of them passed.

Till now I preferred not to see it when everything was OK since it was
useless and permitted a quick visual check in the reports. Do you
really think it's useful ? If others prefer it that way we can change
it but I purposely added this test to "improve" the output (for me at
least). I'm interested in opinions here.

Willy

Re: [PATCH 3/4] selftests/nolibc: always print the log file

[Zhangjin Wu]

> On Mon, Jun 05, 2023 at 11:57:44AM +0800, Zhangjin Wu wrote:
> > This allows to check the other issues of the output file manually even
> > when all of them passed.
> 
> Till now I preferred not to see it when everything was OK since it was
> useless and permitted a quick visual check in the reports. Do you
> really think it's useful ? If others prefer it that way we can change
> it but I purposely added this test to "improve" the output (for me at
> least). I'm interested in opinions here.
>

I planed to add detailed potential issues to check in the commit
message, but didn't do that eventually.

The potential 'issues' may be:

1. string alignment, I found one character offset in the user-space
   'efault' handler patchset
2. duplicated print, the one like '30 fork' we have fixed up
3. kernel messages (may provide some important info)

I did add this manually several times in the past weeks, so, if the path
is there, I can simply copy it and cat it, hope we can print the path by
default ;-) 

The commit message may be changed to something like this:

    This allows us to check the details in the log file even when all of
    them passed, from the log details, we can check the string
    alignment, duplicated print and kernel messages.

Best regards,
Zhangjin

> Willy

[PATCH 4/4] selftests/nolibc: fix up test-report for run target

[Zhangjin Wu]

There is a '\r' at the end of every log line when run nolibc-test on
qemu-system (make run), add support for this case, otherwise, the test
result will like this:

    0 test(s) passed, 0 skipped, 0 failed.

Signed-off-by: Zhangjin Wu <falcon@tinylab.org>
---
 tools/testing/selftests/nolibc/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 518f85c77fe3..70a27fc41c22 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -128,7 +128,7 @@ libc-test: nolibc-test.c
 	$(QUIET_CC)$(CC) -o $@ $<
 
 test-report:
-	$(Q)[ -f $(CURDIR)/run.out ] && awk '/\[OK\]$$/{p++} /\[FAIL\]$$/{f++} /\[SKIPPED\]$$/{s++} \
+	$(Q)[ -f $(CURDIR)/run.out ] && awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{f++} /\[SKIPPED\][\r]*$$/{s++} \
 	         END{ printf("%d test(s) passed, %d skipped, %d failed.", p, s, f); \
 	         printf(" See all results in %s\n", ARGV[1]); }' \
 		 $(CURDIR)/run.out || :
-- 
2.25.1

Re: [PATCH 4/4] selftests/nolibc: fix up test-report for run target

[Willy Tarreau]

On Mon, Jun 05, 2023 at 11:58:56AM +0800, Zhangjin Wu wrote:
> There is a '\r' at the end of every log line when run nolibc-test on
> qemu-system (make run), add support for this case, otherwise, the test
> result will like this:
> 
>     0 test(s) passed, 0 skipped, 0 failed.

Argh, sorry for this one! Note that as a general rule, this should have
been the first patch of the series (fixes first so that they can be
squashed into a pending series or backported when already merged).

I'll remerge it into the series for now.

Thank you!
Willy

Re: [PATCH 0/4] selftests/nolibc: fix up and improve test report

[Willy Tarreau]

On Mon, Jun 05, 2023 at 11:47:41AM +0800, Zhangjin Wu wrote:
> Thanks very much for your merge of the v3 generic part1 of rv32, just
> tested your latest 20230604-nolibc-rv32+stkp6 branch, everything work
> well except a trivial test report regression on the 'run' target.
(...)

I've squashed your fix into the pending patch and pushed branch
20230605-nolibc-rv32+stkp7. I have only tested userland (I really
need to leave now, no time for a kernel build).

Zhangjin and Thomas, now that your last two fixes are merged, I'm
assuming that Paul can take the branch any time. If you're seeing a
showstopper that needs to be fixed, please let him know, and I'll
deal with it once I'm connected again, but please no more attempts
to further improve that branch for now (i.e. consider it merged for
any future work so that we can finally settle on something).

Thanks!
Willy

Re: [PATCH 0/4] selftests/nolibc: fix up and improve test report

[Zhangjin Wu]

> On Mon, Jun 05, 2023 at 11:47:41AM +0800, Zhangjin Wu wrote:
> > Thanks very much for your merge of the v3 generic part1 of rv32, just
> > tested your latest 20230604-nolibc-rv32+stkp6 branch, everything work
> > well except a trivial test report regression on the 'run' target.
> (...)
> 
> I've squashed your fix into the pending patch and pushed branch
> 20230605-nolibc-rv32+stkp7. I have only tested userland (I really
> need to leave now, no time for a kernel build).
>

Just did kernel build + nolibc test for arm, aarch64 and rv64, no regressions
found.

> Zhangjin and Thomas, now that your last two fixes are merged, I'm
> assuming that Paul can take the branch any time. If you're seeing a
> showstopper that needs to be fixed, please let him know, and I'll
> deal with it once I'm connected again, but please no more attempts
> to further improve that branch for now (i.e. consider it merged for
> any future work so that we can finally settle on something).

Ok, let's work on v6.6 ;-)

Best regards,
Zhangjin

> 
> Thanks!
> Willy

Re: [PATCH 0/4] selftests/nolibc: fix up and improve test report

[Willy Tarreau]

On Mon, Jun 05, 2023 at 06:53:32PM +0800, Zhangjin Wu wrote:
> > On Mon, Jun 05, 2023 at 11:47:41AM +0800, Zhangjin Wu wrote:
> > > Thanks very much for your merge of the v3 generic part1 of rv32, just
> > > tested your latest 20230604-nolibc-rv32+stkp6 branch, everything work
> > > well except a trivial test report regression on the 'run' target.
> > (...)
> > 
> > I've squashed your fix into the pending patch and pushed branch
> > 20230605-nolibc-rv32+stkp7. I have only tested userland (I really
> > need to leave now, no time for a kernel build).
> >
> 
> Just did kernel build + nolibc test for arm, aarch64 and rv64, no regressions
> found.

Much appreciated, thank you. This evening I'll try the remaining
archs as I already have the cross-compilers, then will give Paul
the go.

Cheers,
Willy