[PATCH][RFC] kvm-unit-tests: report: add report_skip

[PATCH][RFC] kvm-unit-tests: report: add report_skip

[Andrew Jones]

Add report_skip(), which is report(), but with another condition
allowing it to output PASS/FAIL/SKIP, rather than only PASS/FAIL.
This allows report output to stay more consistent between
systems/configurations that may or may not support all tests.
Currently I only have a downstream use case for this, but maybe
report output sequences of the following form

report(msg1, cond1)                           -> FAIL
report(msg2, cond1 && cond2)                  -> FAIL
report(msg3, cond1 && cond3)                  -> FAIL

would look better as

report(msg1, cond1)                           -> FAIL
report_skip(msg2, !cond1, cond1 && cond2)     -> SKIP
report_skip(msg3, !cond1, cond1 && cond3)     -> SKIP

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 lib/libcflat.h |  1 +
 lib/report.c   | 34 ++++++++++++++++++++++++++--------
 2 files changed, 27 insertions(+), 8 deletions(-)

diff --git a/lib/libcflat.h b/lib/libcflat.h
index f734fdee2df18..01f2769ad1773 100644
--- a/lib/libcflat.h
+++ b/lib/libcflat.h
@@ -61,5 +61,6 @@ extern long atol(const char *ptr);
 #define NULL ((void *)0UL)
 
 void report(const char *msg_fmt, bool pass, ...);
+void report_skip(const char *msg_fmt, bool can_skip, bool pass, ...);
 int report_summary(void);
 #endif
diff --git a/lib/report.c b/lib/report.c
index ff562a13c541e..77db22b6c2d8d 100644
--- a/lib/report.c
+++ b/lib/report.c
@@ -5,32 +5,50 @@
  *
  * Authors:
  *  Jan Kiszka <jan.kiszka@siemens.com>
+ *  Andrew Jones <drjones@redhat.com>
  *
  * This work is licensed under the terms of the GNU LGPL, version 2.
  */
 
 #include "libcflat.h"
 
-static unsigned int tests, failures;
+static unsigned int tests, failures, skipped;
 
-void report(const char *msg_fmt, bool pass, ...)
+void va_report_skip(const char *msg_fmt, bool can_skip, bool pass, va_list va)
 {
+	char *fail = can_skip ? "SKIP" : "FAIL";
 	char buf[2000];
-	va_list va;
 
 	tests++;
-	printf("%s: ", pass ? "PASS" : "FAIL");
-	va_start(va, pass);
+	printf("%s: ", pass ? "PASS" : fail);
 	vsnprintf(buf, sizeof(buf), msg_fmt, va);
-	va_end(va);
 	puts(buf);
 	puts("\n");
-	if (!pass)
+	if (!pass && can_skip)
+		skipped++;
+	else if (!pass)
 		failures++;
 }
 
+void report(const char *msg_fmt, bool pass, ...)
+{
+	va_list va;
+	va_start(va, pass);
+	va_report_skip(msg_fmt, false, pass, va);
+	va_end(va);
+}
+
+void report_skip(const char *msg_fmt, bool can_skip, bool pass, ...)
+{
+	va_list va;
+	va_start(va, pass);
+	va_report_skip(msg_fmt, can_skip, pass, va);
+	va_end(va);
+}
+
 int report_summary(void)
 {
-	printf("\nSUMMARY: %d tests, %d failures\n", tests, failures);
+	printf("\nSUMMARY: %d tests, %d failures, %d skipped\n",
+		tests, failures, skipped);
 	return failures > 0 ? 1 : 0;
 }
-- 
1.9.3

Re: [PATCH][RFC] kvm-unit-tests: report: add report_skip

[Paolo Bonzini]

Il 17/06/2014 14:21, Andrew Jones ha scritto:
> would look better as
>
> report(msg1, cond1)                           -> FAIL
> report_skip(msg2, !cond1, cond1 && cond2)     -> SKIP
> report_skip(msg3, !cond1, cond1 && cond3)     -> SKIP

I think a lot of the time there is other code before the report that you 
want to skip.  Is anything more a "report_skip(msg1);" (that print a 
SKIP message) that useful?  So you can do

     if (!(msr & 0x1000) {
         report_skip("Frob bit 12 of msr");
         return;
     }

Unless what you really want is not a SKIP but an expected failure, then 
I agree with the idea.  Something like

   report_xfail(msg2, cond1, cond2)

would print:

   PASS   if cond1 = false, cond2 = true
   FAIL   if cond1 = false, cond2 = false
   XPASS  if cond1 = true, cond2 = true
   XFAIL  if cond1 = true, cond2 = false

An XPASS would ultimately exit with status 1, just like a FAIL.

Paolo

Re: [PATCH][RFC] kvm-unit-tests: report: add report_skip

[Andrew Jones]

On Tue, Jun 17, 2014 at 02:46:30PM +0200, Paolo Bonzini wrote:
> Il 17/06/2014 14:21, Andrew Jones ha scritto:
> >would look better as
> >
> >report(msg1, cond1)                           -> FAIL
> >report_skip(msg2, !cond1, cond1 && cond2)     -> SKIP
> >report_skip(msg3, !cond1, cond1 && cond3)     -> SKIP
> 
> I think a lot of the time there is other code before the report that you
> want to skip.  Is anything more a "report_skip(msg1);" (that print a SKIP
> message) that useful?  So you can do
> 
>     if (!(msr & 0x1000) {
>         report_skip("Frob bit 12 of msr");

This is enough to output "msg: SKIP" in a format consistent with PASS
and FAIL. However, it's not necessarily enough to output the test case
too. E.g. we'd need

if (cond1) {
  /* do stuff to prepare for report conditions */
  report("msg1", cond2);
  report("msg2", cond3);
} else {
  report_skip("msg1");
  report_skip("msg2");
}

It'd be nice to avoid the message redundancy in the cases that we're
able to. However, ...

>         return;
>     }
> 
> Unless what you really want is not a SKIP but an expected failure, then I

...yes, I was thinking along the lines of an 'expected failure' report,
not the above issue, and 'XFAIL' does convey that idea better.

> agree with the idea.  Something like
> 
>   report_xfail(msg2, cond1, cond2)
> 
> would print:
> 
>   PASS   if cond1 = false, cond2 = true
>   FAIL   if cond1 = false, cond2 = false
>   XPASS  if cond1 = true, cond2 = true
>   XFAIL  if cond1 = true, cond2 = false
> 
> An XPASS would ultimately exit with status 1, just like a FAIL.

Yeah, XPASS is good. report_skip would make PASS ambiguous. Not good.
I'll drop the idea of outputting SKIP, switch to report_xfail, and send
a v2.

Thanks,
drew


> 
> Paolo
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html