[PATCH 1/6] Bitfield without explicit sign should be a warning, not an error

[PATCH 1/6] Bitfield without explicit sign should be a warning, not an error

[Pavel Roskin]

The -Wdefault-bitfield-sign is supposed to control a warning, just like
other -W options.

Signed-off-by: Pavel Roskin <proski@gnu.org>
---

 parse.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/parse.c b/parse.c
index cb9f87a..ab3a096 100644
--- a/parse.c
+++ b/parse.c
@@ -1282,7 +1282,7 @@ static struct token *handle_bitfield(struct token *token, struct symbol *decl)
 		    !(bitfield_type->ctype.modifiers & MOD_EXPLICITLY_SIGNED) &&
 		    is_signed) {
 			// The sign of bitfields is unspecified by default.
-			sparse_error(token->pos, "dubious bitfield without explicit `signed' or `unsigned'");
+			warning(token->pos, "dubious bitfield without explicit `signed' or `unsigned'");
 		}
 	}
 	bitfield->bit_size = width;

[PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Pavel Roskin]

Use the actual sizeof values at the compile time to describe the default
target.  If sparse is compiled on a 64-bit system, it will default to a
64-bit system now.

To force 32-bit operation on 64-bit systems, recognize -m32.  Reject
machine options other than -m32 and -m64.

Signed-off-by: Pavel Roskin <proski@gnu.org>
---

 lib.c    |   13 +++++++------
 target.c |   45 +++++++++++++++++++++++++++++++--------------
 target.h |    6 ++++++
 3 files changed, 44 insertions(+), 20 deletions(-)

diff --git a/lib.c b/lib.c
index 7fea474..e695f1c 100644
--- a/lib.c
+++ b/lib.c
@@ -320,12 +320,13 @@ static char **handle_switch_M(char *arg, char **next)
 
 static char **handle_switch_m(char *arg, char **next)
 {
-	if (!strcmp(arg, "m64")) {
-		bits_in_long = 64;
-		max_int_alignment = 8;
-		bits_in_pointer = 64;
-		pointer_alignment = 8;
-	}
+	if (!strcmp(arg, "m32"))
+		target_set_m32();
+	else if (!strcmp(arg, "m64"))
+		target_set_m64();
+	else
+		die("unsupported machine specification: -%s", arg);
+
 	return next;
 }
 
diff --git a/target.c b/target.c
index 22e948e..1d0912d 100644
--- a/target.c
+++ b/target.c
@@ -1,4 +1,5 @@
 #include <stdio.h>
+#include <sys/param.h>
 
 #include "symbol.h"
 #include "target.h"
@@ -15,31 +16,47 @@ int max_alignment = 16;
  * Integer data types
  */
 int bits_in_bool = 1;
-int bits_in_char = 8;
-int bits_in_short = 16;
-int bits_in_int = 32;
-int bits_in_long = 32;
-int bits_in_longlong = 64;
+int bits_in_char = NBBY * sizeof(char);
+int bits_in_short = NBBY * sizeof(short);
+int bits_in_int = NBBY * sizeof(int);
+int bits_in_long = NBBY * sizeof(long);
+int bits_in_longlong = NBBY * sizeof(long long);
 
-int max_int_alignment = 4;
+int max_int_alignment = sizeof(int);
 
 /*
  * Floating point data types
  */
-int bits_in_float = 32;
-int bits_in_double = 64;
-int bits_in_longdouble = 80;
+int bits_in_float = NBBY * sizeof(float);
+int bits_in_double = NBBY * sizeof(double);
+int bits_in_longdouble = NBBY * sizeof(long double);
 
-int max_fp_alignment = 8;
+int max_fp_alignment = sizeof(double);
 
 /*
  * Pointer data type
  */
-int bits_in_pointer = 32;
-int pointer_alignment = 4;
+int bits_in_pointer = NBBY * sizeof(void *);
+int pointer_alignment = sizeof(void *);
 
 /*
  * Enum data types
  */
-int bits_in_enum = 32;
-int enum_alignment = 4;
+int bits_in_enum = NBBY * sizeof(enum {ENUM_BITS});
+int enum_alignment = sizeof(enum {ENUM_ALIGN});
+
+void target_set_m32(void)
+{
+	bits_in_long = 32;
+	max_int_alignment = 4;
+	bits_in_pointer = 32;
+	pointer_alignment = 4;
+}
+
+void target_set_m64(void)
+{
+	bits_in_long = 64;
+	max_int_alignment = 8;
+	bits_in_pointer = 64;
+	pointer_alignment = 8;
+}
diff --git a/target.h b/target.h
index 25f7948..94e182f 100644
--- a/target.h
+++ b/target.h
@@ -42,4 +42,10 @@ extern int pointer_alignment;
 extern int bits_in_enum;
 extern int enum_alignment;
 
+/*
+ * Functions
+ */
+void target_set_m32(void);
+void target_set_m64(void);
+
 #endif

Re: [PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Al Viro]

On Thu, Jun 28, 2007 at 01:39:59AM -0400, Pavel Roskin wrote:
> Use the actual sizeof values at the compile time to describe the default
> target.  If sparse is compiled on a 64-bit system, it will default to a
> 64-bit system now.
> 
> To force 32-bit operation on 64-bit systems, recognize -m32.  Reject
> machine options other than -m32 and -m64.

NAK.  That makes life very painful for cross-builds.

Re: [PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 776 bytes --]

Al Viro wrote:
> On Thu, Jun 28, 2007 at 01:39:59AM -0400, Pavel Roskin wrote:
>> Use the actual sizeof values at the compile time to describe the default
>> target.  If sparse is compiled on a 64-bit system, it will default to a
>> 64-bit system now.
>>
>> To force 32-bit operation on 64-bit systems, recognize -m32.  Reject
>> machine options other than -m32 and -m64.
> 
> NAK.  That makes life very painful for cross-builds.

And the current approach of hard-coding all the sizes doesn't?

While I agree that I'd like a better approach (specifically, I want any Sparse
build to support any target arch), I don't yet have a solution for that, and
this patch does at least seem like an improvement over the current hardcoded
values.

- Josh Triplett



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Al Viro]

On Wed, Jun 27, 2007 at 11:05:20PM -0700, Josh Triplett wrote:
> Al Viro wrote:
> > On Thu, Jun 28, 2007 at 01:39:59AM -0400, Pavel Roskin wrote:
> >> Use the actual sizeof values at the compile time to describe the default
> >> target.  If sparse is compiled on a 64-bit system, it will default to a
> >> 64-bit system now.
> >>
> >> To force 32-bit operation on 64-bit systems, recognize -m32.  Reject
> >> machine options other than -m32 and -m64.
> > 
> > NAK.  That makes life very painful for cross-builds.
> 
> And the current approach of hard-coding all the sizes doesn't?
> 
> While I agree that I'd like a better approach (specifically, I want any Sparse
> build to support any target arch), I don't yet have a solution for that, and
> this patch does at least seem like an improvement over the current hardcoded
> values.

At least it guarantees behaviour that depends only on the arguments you
pass to sparse and is consistent between the boxen you run sparse _on_.

FWIW, one of the pending patches in my tree makes default size_t unsigned
int, switches it upon -m64 and adds explicit -msize-long for forcing the
same in absense of -m64 (i.e. for 32bit target that has size_t declared
as unsigned long).

The reason why it becomes needed is that with proper typechecking on
comparisons we get every bloody target with size_t declared as unsigned
int (i.e. practically all 32bit ones; the only exception I've seen
is s390 31bit which uses unsigned long) complaining about every
min((size_t)n, sizeof(v)) in the kernel - the second argument becomes
sparse idea of size_t, the first one - kernel idea of size_t and
safety check in min() consist of comparing pointers to types of arguments.

That rule (no m64 => size_t is unsigned int, m64 => unsigned long) gives
a very good approximation and all deviations from it are for 32bit
targets with unsigned long instead of unsigned int.

Having types based on the host sparse is built on is utter insanity -
if anything, we need to be very careful about leaking them into
sparse operations.  Just take a look at what portability nightmare
binutils had become due to that approach...

Re: [PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Jeff Garzik]

Al Viro wrote:
> Having types based on the host sparse is built on is utter insanity -

Definitely agree here.

	Jeff

Re: [PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 1607 bytes --]

Al Viro wrote:
> On Wed, Jun 27, 2007 at 11:05:20PM -0700, Josh Triplett wrote:
>> Al Viro wrote:
>>> On Thu, Jun 28, 2007 at 01:39:59AM -0400, Pavel Roskin wrote:
>>>> Use the actual sizeof values at the compile time to describe the default
>>>> target.  If sparse is compiled on a 64-bit system, it will default to a
>>>> 64-bit system now.
>>>>
>>>> To force 32-bit operation on 64-bit systems, recognize -m32.  Reject
>>>> machine options other than -m32 and -m64.
>>> NAK.  That makes life very painful for cross-builds.
>> And the current approach of hard-coding all the sizes doesn't?
>>
>> While I agree that I'd like a better approach (specifically, I want any Sparse
>> build to support any target arch), I don't yet have a solution for that, and
>> this patch does at least seem like an improvement over the current hardcoded
>> values.
> 
> At least it guarantees behaviour that depends only on the arguments you
> pass to sparse and is consistent between the boxen you run sparse _on_.
[...]
> Having types based on the host sparse is built on is utter insanity -
> if anything, we need to be very careful about leaking them into
> sparse operations.  Just take a look at what portability nightmare
> binutils had become due to that approach...

Hmmm.  OK, I buy that argument.

> FWIW, one of the pending patches in my tree makes default size_t unsigned
> int, switches it upon -m64 and adds explicit -msize-long for forcing the
> same in absense of -m64 (i.e. for 32bit target that has size_t declared
> as unsigned long).

Sounds good.

- Josh Triplett


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Jeff Garzik]

Josh Triplett wrote:
> While I agree that I'd like a better approach (specifically, I want any Sparse
> build to support any target arch), I don't yet have a solution for that, and
> this patch does at least seem like an improvement over the current hardcoded
> values.

That's my desire as well:  My ideal sparse backend should be able to 
compile x86, x86-64, ppc64, ia64, arm, etc. with just a change of 
command line switches.

The gcc approach is just bloody awful.

	Jeff

Re: [PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Pavel Roskin]

On Thu, 2007-06-28 at 02:25 -0400, Jeff Garzik wrote:
> Josh Triplett wrote:
> > While I agree that I'd like a better approach (specifically, I want any Sparse
> > build to support any target arch), I don't yet have a solution for that, and
> > this patch does at least seem like an improvement over the current hardcoded
> > values.
> 
> That's my desire as well:  My ideal sparse backend should be able to 
> compile x86, x86-64, ppc64, ia64, arm, etc. with just a change of 
> command line switches.

That would probably mean having some runtime-loadable files describing
the architectures, as you would not want to describe the architecture
with several switches.

Perhaps some machine options could be described in those files, namely
whether they are acceptable and how they affect the architecture
description.

> The gcc approach is just bloody awful.

Ironically, gcc specfiles do something like that.  Of course, they are
not sufficient to actually _compile_ the code, but they may be
sufficient to verify that code.

-- 
Regards,
Pavel Roskin

Re: [PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Jeff Garzik]

Pavel Roskin wrote:
> On Thu, 2007-06-28 at 02:25 -0400, Jeff Garzik wrote:
>> Josh Triplett wrote:
>>> While I agree that I'd like a better approach (specifically, I want any Sparse
>>> build to support any target arch), I don't yet have a solution for that, and
>>> this patch does at least seem like an improvement over the current hardcoded
>>> values.
>> That's my desire as well:  My ideal sparse backend should be able to 
>> compile x86, x86-64, ppc64, ia64, arm, etc. with just a change of 
>> command line switches.
> 
> That would probably mean having some runtime-loadable files describing
> the architectures,

Runtime-loadable, or compiled in.  But in general... agreed.


>> The gcc approach is just bloody awful.
> 
> Ironically, gcc specfiles do something like that.  Of course, they are
> not sufficient to actually _compile_ the code, but they may be
> sufficient to verify that code.

I was mainly grousing about having to recompile gcc for each target, 
which is insanely silly.

	Jeff

Re: [PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 814 bytes --]

Jeff Garzik wrote:
> Pavel Roskin wrote:
>> On Thu, 2007-06-28 at 02:25 -0400, Jeff Garzik wrote:
>>> Josh Triplett wrote:
>>>> While I agree that I'd like a better approach (specifically, I want any Sparse
>>>> build to support any target arch), I don't yet have a solution for that, and
>>>> this patch does at least seem like an improvement over the current hardcoded
>>>> values.
>>> That's my desire as well:  My ideal sparse backend should be able to 
>>> compile x86, x86-64, ppc64, ia64, arm, etc. with just a change of 
>>> command line switches.
>> That would probably mean having some runtime-loadable files describing
>> the architectures,
> 
> Runtime-loadable, or compiled in.  But in general... agreed.

Yes, I definitely want the ability to compile them in.

- Josh Triplett



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 1534 bytes --]

Jeff Garzik wrote:
> Josh Triplett wrote:
>> While I agree that I'd like a better approach (specifically, I want any Sparse
>> build to support any target arch), I don't yet have a solution for that, and
>> this patch does at least seem like an improvement over the current hardcoded
>> values.
> 
> That's my desire as well:  My ideal sparse backend should be able to 
> compile x86, x86-64, ppc64, ia64, arm, etc. with just a change of 
> command line switches.
> 
> The gcc approach is just bloody awful.

I agree that the GCC approach to cross-compilation could use significant
improvement.  In particular, I'd like to just specify a set of architectures,
or "all of them", at compile time, and have GCC support all of those
architectures in the same GCC binary.  Combined with GCC's existing "sysroot"
approach to multiple architecture support, this would make cross-compilation
much easier.

However, in the meantime, GCC doesn't seem to have any such thing.  How do we
want to implement it in Sparse?  I don't think we have enough information on
the GCC command line.  I suppose we could allow specifying an architecture
explicitly; for cgcc, we could also allow specifying the compiler name for
cross-compilation and then deriving the architecture name from the compiler
name if not explicitly specified.

I'd like to see much of cgcc disappear; it should do nothing except invoke GCC
and Sparse, and Sparse should handle the architecture issues that cgcc
currently handles.

- Josh Triplett


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Jeff Garzik]

Josh Triplett wrote:
> However, in the meantime, GCC doesn't seem to have any such thing.  How do we
> want to implement it in Sparse?  I don't think we have enough information on
> the GCC command line.  I suppose we could allow specifying an architecture
> explicitly; for cgcc, we could also allow specifying the compiler name for
> cross-compilation and then deriving the architecture name from the compiler
> name if not explicitly specified.

Judging from existing software, "--target foo" to set the target arch 
should be familiar to many.  If not specified, detect the current arch 
at runtime, and use the results of the detection as the target arch.

	Jeff

Re: [PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 753 bytes --]

Jeff Garzik wrote:
> Josh Triplett wrote:
>> However, in the meantime, GCC doesn't seem to have any such thing.  How do we
>> want to implement it in Sparse?  I don't think we have enough information on
>> the GCC command line.  I suppose we could allow specifying an architecture
>> explicitly; for cgcc, we could also allow specifying the compiler name for
>> cross-compilation and then deriving the architecture name from the compiler
>> name if not explicitly specified.
> 
> Judging from existing software, "--target foo" to set the target arch 
> should be familiar to many.  If not specified, detect the current arch 
> at runtime, and use the results of the detection as the target arch.

Sounds reasonable.

- Josh Triplett



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 2/6] Hardcode actual type sizes, add -m32 support

[Pavel Roskin]

On Wed, 2007-06-27 at 23:05 -0700, Josh Triplett wrote:

> And the current approach of hard-coding all the sizes doesn't?
> 
> While I agree that I'd like a better approach (specifically, I want any Sparse
> build to support any target arch), I don't yet have a solution for that, and
> this patch does at least seem like an improvement over the current hardcoded
> values.

Another problem with this patch is that sparse is actually fed many "-m"
options that it's supposed to ignore, such as "-march=nocona" on my
x86_64.  That's makes live painful for native builds too.  So, adding
strict verification was a bit premature.

Ideally, sparse should know all "-m" options it's given.  I was wrong
that I only tested this change on userspace programs.

So please drop the patch for now.  However, this is not an attempt to
stop the discussion, as it may be very useful.

-- 
Regards,
Pavel Roskin

[PATCH 3/6] cgcc: preserve sparse exit code if -no-compile is used

[Pavel Roskin]

Signed-off-by: Pavel Roskin <proski@gnu.org>
---

 cgcc |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/cgcc b/cgcc
index 226a7a7..2b9f89c 100755
--- a/cgcc
+++ b/cgcc
@@ -50,8 +50,11 @@ if ($do_check) {
 	$check .= &add_specs ('host_os_specs');
     }
     print "$check\n" if $verbose;
-#    exit 1;
-    system ($check);
+    if ($do_compile) {
+	system ($check);
+    } else {
+	exec ($check);
+    }
 }
 
 if ($do_compile) {

Re: [PATCH 3/6] cgcc: preserve sparse exit code if -no-compile is used

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 39 bytes --]

Applied; thanks!

- Josh Triplett


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

[PATCH 4/6] Avoid use of libc headers in the validation suite

[Pavel Roskin]

We don't want libc to affect the validation suite.  Compatibility with
libc headers can be checked by other means, e.g. by compiling some
significant program (even sparse itself) using cgcc as the compiler.

Signed-off-by: Pavel Roskin <proski@gnu.org>
---

 validation/check_byte_count-ice.c |    2 +-
 validation/double-semicolon.c     |    2 +-
 validation/test-be.c              |    4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/validation/check_byte_count-ice.c b/validation/check_byte_count-ice.c
index eaeef96..e58312d 100644
--- a/validation/check_byte_count-ice.c
+++ b/validation/check_byte_count-ice.c
@@ -1,4 +1,4 @@
-#include <string.h>
+extern void *memset (void *s, int c, int n);
 
 static void foo(void *a)
 {
diff --git a/validation/double-semicolon.c b/validation/double-semicolon.c
index 039236f..36cb332 100644
--- a/validation/double-semicolon.c
+++ b/validation/double-semicolon.c
@@ -1,4 +1,4 @@
-#include <string.h>
+extern void *memset (void *s, int c, int n);
 static void test(void)
 {
 	struct { int foo;; } val;
diff --git a/validation/test-be.c b/validation/test-be.c
index 6dfa55f..6b74555 100644
--- a/validation/test-be.c
+++ b/validation/test-be.c
@@ -1,5 +1,5 @@
-#include <stdio.h>
-#include <stdlib.h>
+int printf(char *c, ...);
+void exit(int c);
 
 #undef PRINT_OUTPUTS
 

Re: [PATCH 4/6] Avoid use of libc headers in the validation suite

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 289 bytes --]

Pavel Roskin wrote:
> We don't want libc to affect the validation suite.  Compatibility with
> libc headers can be checked by other means, e.g. by compiling some
> significant program (even sparse itself) using cgcc as the compiler.

Agreed.  Applied; thanks!

- Josh Triplett



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

[PATCH 5/6] Fix warnings about undeclared globals, they are irrelevant to the test

[Pavel Roskin]

Signed-off-by: Pavel Roskin <proski@gnu.org>
---

 validation/bitfields.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/validation/bitfields.c b/validation/bitfields.c
index 292a110..16aa16d 100644
--- a/validation/bitfields.c
+++ b/validation/bitfields.c
@@ -5,13 +5,13 @@
  *
  * "warning: a.c:16:10: incompatible types for operation"
  */
-struct {
+static struct {
 	int x:4;
 } y;
 
 extern int a[];
 
-int b(void)
+static int b(void)
 {
 	return a[y.x];
 }

Re: [PATCH 5/6] Fix warnings about undeclared globals, they are irrelevant to the test

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 215 bytes --]

Pavel Roskin wrote:
> Signed-off-by: Pavel Roskin <proski@gnu.org>
> ---
> 
>  validation/bitfields.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)

Applied; thanks!

- Josh Triplett



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

[PATCH 6/6] Add a simple test script, embed expected results into test files

[Pavel Roskin]

Signed-off-by: Pavel Roskin <proski@gnu.org>
---

 validation/.gitignore                         |    3 +
 validation/address_space.c                    |    6 +++
 validation/bad-array-designated-initializer.c |    6 +++
 validation/bad-assignment.c                   |    5 ++
 validation/bad-cast.c                         |    6 +++
 validation/bad-ternary-cond.c                 |    5 ++
 validation/badtype2.c                         |   11 +++++
 validation/badtype3.c                         |   13 ++++++
 validation/badtype4.c                         |    5 ++
 validation/builtin_safe1.c                    |   12 +++++
 validation/check_byte_count-ice.c             |    5 ++
 validation/choose_expr.c                      |   10 ++++
 validation/cond_expr.c                        |    5 ++
 validation/context.c                          |   17 +++++++
 validation/foul-bitwise.c                     |    7 +++
 validation/initializer-entry-defined-twice.c  |    7 +++
 validation/noderef.c                          |    6 +++
 validation/non-pointer-null.c                 |    4 ++
 validation/old-initializer.c                  |    4 ++
 validation/preprocessor-loop.c                |    4 ++
 validation/preprocessor1.c                    |    5 ++
 validation/preprocessor10.c                   |    5 ++
 validation/preprocessor11.c                   |   12 +++++
 validation/preprocessor12.c                   |    5 ++
 validation/preprocessor13.c                   |    7 +++
 validation/preprocessor14.c                   |    5 ++
 validation/preprocessor15.c                   |    5 ++
 validation/preprocessor18.c                   |    5 ++
 validation/preprocessor19.c                   |    7 +++
 validation/preprocessor2.c                    |    5 ++
 validation/preprocessor20.c                   |    5 ++
 validation/preprocessor3.c                    |    6 +++
 validation/preprocessor4.c                    |    5 ++
 validation/preprocessor5.c                    |    5 ++
 validation/preprocessor7.c                    |    5 ++
 validation/preprocessor8.c                    |    9 ++++
 validation/preprocessor9.c                    |    5 ++
 validation/run-tests                          |   58 +++++++++++++++++++++++++
 38 files changed, 300 insertions(+), 0 deletions(-)

diff --git a/validation/.gitignore b/validation/.gitignore
new file mode 100644
index 0000000..515eae1
--- /dev/null
+++ b/validation/.gitignore
@@ -0,0 +1,3 @@
+*.diff
+*.exp
+*.res
diff --git a/validation/address_space.c b/validation/address_space.c
index 377890a..efd98b7 100644
--- a/validation/address_space.c
+++ b/validation/address_space.c
@@ -6,3 +6,9 @@ static int sys_do_stuff(void __user *user_addr)
 {
 	return poke_memory(user_addr);
 }
+
+/* SPARSE
+address_space.c:7:21: warning: incorrect type in argument 1 (different address spaces)
+address_space.c:7:21:    expected void *addr
+address_space.c:7:21:    got void *user_addr<asn:1>
+   SPARSE */
diff --git a/validation/bad-array-designated-initializer.c b/validation/bad-array-designated-initializer.c
index 636fa63..def6944 100644
--- a/validation/bad-array-designated-initializer.c
+++ b/validation/bad-array-designated-initializer.c
@@ -2,3 +2,9 @@ static int a[] = {
 	[0] = 0,		// OK
 	[\0] = 1,		// KO
 };
+
+/* SPARSE
+bad-array-designated-initializer.c:3:3: error: Expected constant expression
+bad-array-designated-initializer.c:3:3: error: Expected } at end of initializer
+bad-array-designated-initializer.c:3:3: error: got \
+   SPARSE */
diff --git a/validation/bad-assignment.c b/validation/bad-assignment.c
index 3b66a11..3054f86 100644
--- a/validation/bad-assignment.c
+++ b/validation/bad-assignment.c
@@ -4,3 +4,8 @@ static int foo(int a)
 
 	return a;
 }
+
+/* SPARSE
+bad-assignment.c:3:6: error: Expected ; at end of statement
+bad-assignment.c:3:6: error: got \
+   SPARSE */
diff --git a/validation/bad-cast.c b/validation/bad-cast.c
index bafd5a2..ed00150 100644
--- a/validation/bad-cast.c
+++ b/validation/bad-cast.c
@@ -4,3 +4,9 @@ static int foo(int a)
 {
 	return (struct/st *) a;
 }
+
+/* SPARSE
+bad-cast.c:5:16: error: expected declaration
+bad-cast.c:5:16: error: Expected ) at end of cast operator
+bad-cast.c:5:16: error: got /
+   SPARSE */
diff --git a/validation/bad-ternary-cond.c b/validation/bad-ternary-cond.c
index d502931..42dc1a5 100644
--- a/validation/bad-ternary-cond.c
+++ b/validation/bad-ternary-cond.c
@@ -5,3 +5,8 @@ static int foo(int a)
 {
 	return a ?? 1 : 0;
 }
+
+/* SPARSE
+bad-ternary-cond.c:6:12: error: Expected : in conditional expression
+bad-ternary-cond.c:6:12: error: got ?
+   SPARSE */
diff --git a/validation/badtype2.c b/validation/badtype2.c
index aad725d..0920145 100644
--- a/validation/badtype2.c
+++ b/validation/badtype2.c
@@ -8,3 +8,14 @@ static undef foo(char *c)
     return bar();
   }
 }
+
+/* SPARSE
+badtype2.c:2:14: error: Expected ; at end of declaration
+badtype2.c:2:14: error: got bar
+badtype2.c:3:14: error: Expected ; at end of declaration
+badtype2.c:3:14: error: got foo
+badtype2.c:6:3: error: Trying to use reserved word 'switch' as identifier
+badtype2.c:7:3: error: not in switch scope
+badtype2.c:10:1: error: Expected ; end of type declaration
+badtype2.c:10:1: error: got }
+   SPARSE */
diff --git a/validation/badtype3.c b/validation/badtype3.c
index 198ef87..edd90f9 100644
--- a/validation/badtype3.c
+++ b/validation/badtype3.c
@@ -8,3 +8,16 @@ foo (int (*func) (undef, void *), void *data)
   }
   return err;
 }
+
+/* SPARSE
+badtype3.c:2:19: error: Expected ) in nested declarator
+badtype3.c:2:19: error: got undef
+badtype3.c:5:3: error: Trying to use reserved word 'while' as identifier
+badtype3.c:7:7: error: break/continue not in iterator scope
+badtype3.c:9:3: error: Trying to use reserved word 'return' as identifier
+badtype3.c:9:10: error: Expected ; at end of declaration
+badtype3.c:9:10: error: got err
+badtype3.c:10:1: error: Expected ; end of type declaration
+badtype3.c:10:1: error: got }
+badtype3.c:6:11: error: undefined identifier 'func'
+   SPARSE */
diff --git a/validation/badtype4.c b/validation/badtype4.c
index 4f0c494..8020155 100644
--- a/validation/badtype4.c
+++ b/validation/badtype4.c
@@ -5,3 +5,8 @@ void a(void)
 		break;
 	}
 }
+
+/* SPARSE
+badtype4.c:3:9: error: undefined identifier 'x'
+badtype4.c:4:7: error: incompatible types for 'case' statement
+   SPARSE */
diff --git a/validation/builtin_safe1.c b/validation/builtin_safe1.c
index d95e46b..f5adb04 100644
--- a/validation/builtin_safe1.c
+++ b/validation/builtin_safe1.c
@@ -24,3 +24,15 @@ static int foo(int x, int y)
   return x;
 }
 
+
+/* SPARSE
+builtin_safe1.c:13:3: warning: Macro argument with side effects
+builtin_safe1.c:14:3: warning: Macro argument with side effects
+builtin_safe1.c:15:3: warning: Macro argument with side effects
+builtin_safe1.c:16:3: warning: Macro argument with side effects
+builtin_safe1.c:17:3: warning: Macro argument with side effects
+builtin_safe1.c:18:3: warning: Macro argument with side effects
+builtin_safe1.c:19:3: warning: Macro argument with side effects
+builtin_safe1.c:22:3: warning: Macro argument with side effects
+builtin_safe1.c:23:3: warning: Macro argument with side effects
+   SPARSE */
diff --git a/validation/check_byte_count-ice.c b/validation/check_byte_count-ice.c
index e58312d..d400542 100644
--- a/validation/check_byte_count-ice.c
+++ b/validation/check_byte_count-ice.c
@@ -4,3 +4,8 @@ static void foo(void *a)
 {
 	memset(foo, + ', 20);
 }
+
+/* SPARSE
+check_byte_count-ice.c:5:18: error: Bad character constant
+check_byte_count-ice.c:5:8: error: not enough arguments for function memset
+   SPARSE */
diff --git a/validation/choose_expr.c b/validation/choose_expr.c
index 55bfa0c..482ec21 100644
--- a/validation/choose_expr.c
+++ b/validation/choose_expr.c
@@ -2,3 +2,13 @@ static int x = __builtin_choose_expr(0,(char *)0,(void)0);
 static int y = __builtin_choose_expr(1,(char *)0,(void)0);
 static char s[42];
 static int z = 1/(sizeof(__builtin_choose_expr(1,s,0)) - 42);
+
+/* SPARSE
+choose_expr.c:1:51: warning: incorrect type in initializer (different types)
+choose_expr.c:1:51:    expected int static [signed] [toplevel] x
+choose_expr.c:1:51:    got void <noident>
+choose_expr.c:2:41: warning: incorrect type in initializer (different base types)
+choose_expr.c:2:41:    expected int static [signed] [toplevel] y
+choose_expr.c:2:41:    got char *<noident>
+choose_expr.c:4:17: warning: division by zero
+   SPARSE */
diff --git a/validation/cond_expr.c b/validation/cond_expr.c
index d5d9e6f..280d358 100644
--- a/validation/cond_expr.c
+++ b/validation/cond_expr.c
@@ -9,3 +9,8 @@ int a(void)
 {
 	return ~(y ? : x);	/* should warn */
 }
+
+/* SPARSE
+cond_expr.c:10:9: error: incompatible types for operation (~)
+cond_expr.c:10:9:    argument has type double 
+   SPARSE */
diff --git a/validation/context.c b/validation/context.c
index f877ec2..dbbfc4e 100644
--- a/validation/context.c
+++ b/validation/context.c
@@ -314,3 +314,20 @@ static void warn_cond_lock1(void)
         condition2 = 1; /* do stuff */
     r();
 }
+
+/* SPARSE
+context.c:69:13: warning: context imbalance in 'warn_lock1' - wrong count at exit
+context.c:74:13: warning: context imbalance in 'warn_lock2' - wrong count at exit
+context.c:81:13: warning: context imbalance in 'warn_lock3' - wrong count at exit
+context.c:88:13: warning: context imbalance in 'warn_unlock1' - unexpected unlock
+context.c:93:13: warning: context imbalance in 'warn_unlock2' - unexpected unlock
+context.c:131:12: warning: context imbalance in 'warn_if1' - wrong count at exit
+context.c:140:12: warning: context imbalance in 'warn_if2' - different lock contexts for basic block
+context.c:202:2: warning: context imbalance in 'warn_while1' - different lock contexts for basic block
+context.c:210:3: warning: context imbalance in 'warn_while2' - unexpected unlock
+context.c:216:2: warning: context imbalance in 'warn_while3' - wrong count at exit
+context.c:274:13: warning: context imbalance in 'warn_goto1' - wrong count at exit
+context.c:283:13: warning: context imbalance in 'warn_goto2' - wrong count at exit
+context.c:300:5: warning: context imbalance in 'warn_goto3' - different lock contexts for basic block
+context.c:315:5: warning: context imbalance in 'warn_cond_lock1' - different lock contexts for basic block
+   SPARSE */
diff --git a/validation/foul-bitwise.c b/validation/foul-bitwise.c
index ca84be6..d8e8ef2 100644
--- a/validation/foul-bitwise.c
+++ b/validation/foul-bitwise.c
@@ -18,3 +18,10 @@ static __le16 bar(__le16 a)
 {
 	return -a;
 }
+
+/* SPARSE
+foul-bitwise.c:9:9: warning: restricted degrades to integer
+foul-bitwise.c:9:15: warning: restricted degrades to integer
+foul-bitwise.c:19:9: error: incompatible types for operation (-)
+foul-bitwise.c:19:9:    argument has type restricted unsigned short [usertype] a
+   SPARSE */
diff --git a/validation/initializer-entry-defined-twice.c b/validation/initializer-entry-defined-twice.c
index 6c48c52..1c04a62 100644
--- a/validation/initializer-entry-defined-twice.c
+++ b/validation/initializer-entry-defined-twice.c
@@ -41,3 +41,10 @@ static struct same_offset not_an_error = {
 	.field1 = { },
 	.field2 = 0
 };
+
+/* SPARSE
+initializer-entry-defined-twice.c:10:3: error: Initializer entry defined twice
+initializer-entry-defined-twice.c:11:3:   also defined here
+initializer-entry-defined-twice.c:26:4: error: Initializer entry defined twice
+initializer-entry-defined-twice.c:27:4:   also defined here
+   SPARSE */
diff --git a/validation/noderef.c b/validation/noderef.c
index 15be327..0666376 100644
--- a/validation/noderef.c
+++ b/validation/noderef.c
@@ -40,3 +40,9 @@ static void h(void)
 	r = py->a+1;
 	r = &py->a[0];
 }
+
+/* SPARSE
+noderef.c:24:5: warning: incorrect type in assignment (different modifiers)
+noderef.c:24:5:    expected char *[noderef] *q2
+noderef.c:24:5:    got char [noderef] **<noident>
+   SPARSE */
diff --git a/validation/non-pointer-null.c b/validation/non-pointer-null.c
index 2d09aa3..29195ed 100644
--- a/validation/non-pointer-null.c
+++ b/validation/non-pointer-null.c
@@ -1 +1,5 @@
 static void *p = 0;
+
+/* SPARSE
+non-pointer-null.c:1:18: warning: Using plain integer as NULL pointer
+   SPARSE */
diff --git a/validation/old-initializer.c b/validation/old-initializer.c
index 5ae8516..ed1c538 100644
--- a/validation/old-initializer.c
+++ b/validation/old-initializer.c
@@ -3,3 +3,7 @@ struct s {
 };
 
 static struct s the_s = { i: 1 };
+
+/* SPARSE
+old-initializer.c:5:27: warning: obsolete struct initializer, use C99 syntax
+   SPARSE */
diff --git a/validation/preprocessor-loop.c b/validation/preprocessor-loop.c
index 035777e..d51cfc3 100644
--- a/validation/preprocessor-loop.c
+++ b/validation/preprocessor-loop.c
@@ -1,2 +1,6 @@
 #if 1
 #if
+
+/* SPARSE
+preprocessor-loop.c:2:2: error: unterminated preprocessor conditional
+   SPARSE */
diff --git a/validation/preprocessor1.c b/validation/preprocessor1.c
index 5ae20aa..5e9f234 100644
--- a/validation/preprocessor1.c
+++ b/validation/preprocessor1.c
@@ -12,3 +12,8 @@
 #define bar func(
 #define foo bar foo
 foo )
+
+/* SPARSE
+builtin:0:0: error: Expected ; at end of declaration
+builtin:0:0: error: got end-of-input
+   SPARSE */
diff --git a/validation/preprocessor10.c b/validation/preprocessor10.c
index 7fcac36..137bafd 100644
--- a/validation/preprocessor10.c
+++ b/validation/preprocessor10.c
@@ -11,3 +11,8 @@ defined
 #else
 undefined
 #endif
+
+/* SPARSE
+builtin:0:0: error: Expected ; at end of declaration
+builtin:0:0: error: got end-of-input
+   SPARSE */
diff --git a/validation/preprocessor11.c b/validation/preprocessor11.c
index 4c6c1c0..276ccf6 100644
--- a/validation/preprocessor11.c
+++ b/validation/preprocessor11.c
@@ -8,3 +8,15 @@
 #define H(x...,y)
 #define I(...+
 #define J(x,y)
+
+/* SPARSE
+preprocessor11.c:1:11: error: "1" may not appear in macro parameter list
+preprocessor11.c:2:11: error: missing ')' in macro parameter list
+preprocessor11.c:3:12: error: missing ')' in macro parameter list
+preprocessor11.c:4:11: error: parameter name missing
+preprocessor11.c:5:11: error: __VA_ARGS__ can only appear in the expansion of a C99 variadic macro
+preprocessor11.c:6:12: error: "+" may not appear in macro parameter list
+preprocessor11.c:7:12: error: missing ')' in macro parameter list
+preprocessor11.c:8:12: error: missing ')' in macro parameter list
+preprocessor11.c:9:11: error: missing ')' in macro parameter list
+   SPARSE */
diff --git a/validation/preprocessor12.c b/validation/preprocessor12.c
index 385c1a7..b86cab0 100644
--- a/validation/preprocessor12.c
+++ b/validation/preprocessor12.c
@@ -5,3 +5,8 @@
 A(1)
 A(1,2)
 A(1,2,3)
+
+/* SPARSE
+preprocessor12.c:5:1: error: Expected ; end of type declaration
+preprocessor12.c:5:1: error: got 1
+   SPARSE */
diff --git a/validation/preprocessor13.c b/validation/preprocessor13.c
index 96c813e..afbdc4f 100644
--- a/validation/preprocessor13.c
+++ b/validation/preprocessor13.c
@@ -5,3 +5,10 @@
 A(1)
 A(1,2)
 A(1,2,3)
+
+/* SPARSE
+preprocessor13.c:6:1: error: '##' failed: concatenation is not a valid token
+preprocessor13.c:7:1: error: '##' failed: concatenation is not a valid token
+preprocessor13.c:5:1: error: Expected ; end of type declaration
+preprocessor13.c:5:1: error: got 1
+   SPARSE */
diff --git a/validation/preprocessor14.c b/validation/preprocessor14.c
index 001f1f2..85210d1 100644
--- a/validation/preprocessor14.c
+++ b/validation/preprocessor14.c
@@ -5,3 +5,8 @@
 A(,1)
 #define B(x,y,...) x##,##__VA_ARGS__
 B(,1)
+
+/* SPARSE
+preprocessor14.c:5:1: error: Expected ; end of type declaration
+preprocessor14.c:5:1: error: got ,
+   SPARSE */
diff --git a/validation/preprocessor15.c b/validation/preprocessor15.c
index 0a2dfd7..63f64fe 100644
--- a/validation/preprocessor15.c
+++ b/validation/preprocessor15.c
@@ -5,3 +5,8 @@
 #if D(A,B) B
 D(1,2)
 #endif
+
+/* SPARSE
+preprocessor15.c:6:1: error: Expected ; end of type declaration
+preprocessor15.c:6:1: error: got 12
+   SPARSE */
diff --git a/validation/preprocessor18.c b/validation/preprocessor18.c
index b5988bf..8fe58e4 100644
--- a/validation/preprocessor18.c
+++ b/validation/preprocessor18.c
@@ -1,3 +1,8 @@
 /* one warning for each, please... */
 #define 1
 #undef 1
+
+/* SPARSE
+preprocessor18.c:2:2: error: expected identifier to 'define'
+preprocessor18.c:3:2: error: expected identifier to 'undef'
+   SPARSE */
diff --git a/validation/preprocessor19.c b/validation/preprocessor19.c
index 0f7da47..f0c01de 100644
--- a/validation/preprocessor19.c
+++ b/validation/preprocessor19.c
@@ -3,3 +3,10 @@
 #define A x
 #define A y
 A
+
+/* SPARSE
+preprocessor19.c:4:9: warning: preprocessor token A redefined
+preprocessor19.c:3:9: this was the original definition
+builtin:0:0: error: Expected ; at end of declaration
+builtin:0:0: error: got end-of-input
+   SPARSE */
diff --git a/validation/preprocessor2.c b/validation/preprocessor2.c
index 340938e..36740b0 100644
--- a/validation/preprocessor2.c
+++ b/validation/preprocessor2.c
@@ -13,3 +13,8 @@
 #define BINARY(x, y) x + y
 
 UNARY(TWO)
+
+/* SPARSE
+preprocessor2.c:15:1: error: Expected ; at end of declaration
+preprocessor2.c:15:1: error: got +
+   SPARSE */
diff --git a/validation/preprocessor20.c b/validation/preprocessor20.c
index 68c0ccc..3101e93 100644
--- a/validation/preprocessor20.c
+++ b/validation/preprocessor20.c
@@ -2,3 +2,8 @@
 #define X
 #define Y
 #include "preprocessor20.h"
+
+/* SPARSE
+preprocessor20.h:2:1: error: Expected ; at end of declaration
+preprocessor20.h:2:1: error: got B
+   SPARSE */
diff --git a/validation/preprocessor3.c b/validation/preprocessor3.c
index 71b9acd..738154e 100644
--- a/validation/preprocessor3.c
+++ b/validation/preprocessor3.c
@@ -35,3 +35,9 @@ A()                     // B ( )
 SCAN( A() )             // A ( )
 SCAN(SCAN( A() ))       // B ( )
 SCAN(SCAN(SCAN( A() ))) // A ( )
+
+/* SPARSE
+preprocessor3.c:34:1: warning: non-ANSI function declaration of function 'B'
+preprocessor3.c:35:1: error: Expected ; at end of declaration
+preprocessor3.c:35:1: error: got A
+   SPARSE */
diff --git a/validation/preprocessor4.c b/validation/preprocessor4.c
index 8b8c4da..1620a8b 100644
--- a/validation/preprocessor4.c
+++ b/validation/preprocessor4.c
@@ -8,3 +8,8 @@
 
 mac(foo)
 
+
+/* SPARSE
+builtin:0:0: error: Expected ; at end of declaration
+builtin:0:0: error: got end-of-input
+   SPARSE */
diff --git a/validation/preprocessor5.c b/validation/preprocessor5.c
index fa38937..8dbfcba 100644
--- a/validation/preprocessor5.c
+++ b/validation/preprocessor5.c
@@ -7,3 +7,8 @@
 #define b(x) x
 
 b(a)
+
+/* SPARSE
+preprocessor5.c:9:1: error: Expected ; at end of declaration
+preprocessor5.c:9:1: error: got |
+   SPARSE */
diff --git a/validation/preprocessor7.c b/validation/preprocessor7.c
index e7e1d37..23dd0cf 100644
--- a/validation/preprocessor7.c
+++ b/validation/preprocessor7.c
@@ -3,3 +3,8 @@
 #define C(x,y) E(y)
 #define E(y) #y
 A(2))
+
+/* SPARSE
+preprocessor7.c:5:1: error: Expected ; end of type declaration
+preprocessor7.c:5:1: error: got "\"D\""
+   SPARSE */
diff --git a/validation/preprocessor8.c b/validation/preprocessor8.c
index 861dfe5..f724692 100644
--- a/validation/preprocessor8.c
+++ b/validation/preprocessor8.c
@@ -14,3 +14,12 @@
 "x#y"			: E
 "ab GH \"G\" 12"	: F(G,H)
 "a ## b"		: I(a,##,b)
+
+/* SPARSE
+preprocessor8.c:1:14: error: '##' cannot appear at the ends of macro expansion
+preprocessor8.c:2:16: error: '##' cannot appear at the ends of macro expansion
+preprocessor8.c:3:22: error: '##' cannot appear at the ends of macro expansion
+preprocessor8.c:4:15: error: '#' is not followed by a macro parameter
+preprocessor8.c:10:1: error: Expected ; end of type declaration
+preprocessor8.c:10:1: error: got "A(x)"
+   SPARSE */
diff --git a/validation/preprocessor9.c b/validation/preprocessor9.c
index 76d6e41..5c73378 100644
--- a/validation/preprocessor9.c
+++ b/validation/preprocessor9.c
@@ -8,3 +8,8 @@
 #define A # define X 1
 A
 X
+
+/* SPARSE
+preprocessor9.c:9:1: error: Expected ; end of type declaration
+preprocessor9.c:9:1: error: got #
+   SPARSE */
diff --git a/validation/run-tests b/validation/run-tests
new file mode 100755
index 0000000..ebe8ffc
--- /dev/null
+++ b/validation/run-tests
@@ -0,0 +1,58 @@
+#!/bin/sh
+
+: ${SPARSE=../sparse}
+: ${SPARSE_FLAGS=}
+
+# Use "--update" to update test files with the actual output
+update=
+if test "$1" = "--update"; then
+	update=1
+	shift
+fi
+
+if test -n "$1"; then
+	tests="$@"
+else
+	tests=`echo *.c`
+fi
+
+bad=
+for test in $tests; do
+	base="`basename $test .c`"
+	exp="$base.exp"
+	res="$base.res"
+	diff="$base.diff"
+	$SPARSE $SPARSE_FLAGS "$test" 2>"$res"
+	if test "$?" != 0; then
+		echo "FATAL: $test"
+		bad="$bad $test"
+		continue
+	fi
+
+	if test -n "$update"; then
+		sed -i '/SPARSE/,/SPARSE/d' "$test"
+		if test -s "$res"; then
+			echo "/* SPARSE" >>"$test"
+			cat "$res" >>"$test"
+			echo "   SPARSE */" >>"$test"
+		fi
+	fi
+
+	sed -n '/SPARSE/,/SPARSE/p' "$test" |grep -v SPARSE >"$exp"
+	diff -u "$exp" "$res" >"$diff"
+	if test -s "$diff"; then
+		echo "FAIL: $test"
+		bad="$bad $test"
+		continue
+	fi
+
+	rm -f "$exp" "$res" "$diff"
+done
+
+if test -n "$bad"; then
+	echo "Failed tests:$bad"
+	exit 1
+fi
+
+echo "Testsuite completed successfully"
+exit 0

Re: [PATCH 6/6] Add a simple test script, embed expected results into test files

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 3715 bytes --]

Awesome!  I plan to merge this tomorrow.

I did notice a few issues.  If you get to them before tomorrow then please
submit an updated patch, but otherwise I will go ahead and fix them myself and
apply the patch; I really want to see this go into Sparse as soon as possible.

* The files starting with "preprocessor" shouldn't run through all of Sparse,
  just the preprocessor (sparse -E).  I've moved them to a
  validation/preprocessor/ subdirectory to keep them separated.

* Ideally, the test suite should support a SPARSE_FLAGS comment, so you can
  specify different -Wfoo options and the corresponding output.  This would
  also handle the preprocessor tests.

Pavel Roskin wrote:
> diff --git a/validation/preprocessor1.c b/validation/preprocessor1.c
> index 5ae20aa..5e9f234 100644
> --- a/validation/preprocessor1.c
> +++ b/validation/preprocessor1.c
> @@ -12,3 +12,8 @@
>  #define bar func(
>  #define foo bar foo
>  foo )
> +
> +/* SPARSE
> +builtin:0:0: error: Expected ; at end of declaration
> +builtin:0:0: error: got end-of-input
> +   SPARSE */

Errors in builtin seem strange and probably broken.

> diff --git a/validation/preprocessor10.c b/validation/preprocessor10.c
> index 7fcac36..137bafd 100644
> --- a/validation/preprocessor10.c
> +++ b/validation/preprocessor10.c
> @@ -11,3 +11,8 @@ defined
>  #else
>  undefined
>  #endif
> +
> +/* SPARSE
> +builtin:0:0: error: Expected ; at end of declaration
> +builtin:0:0: error: got end-of-input
> +   SPARSE */

Ditto.

> diff --git a/validation/preprocessor4.c b/validation/preprocessor4.c
> index 8b8c4da..1620a8b 100644
> --- a/validation/preprocessor4.c
> +++ b/validation/preprocessor4.c
> @@ -8,3 +8,8 @@
>  
>  mac(foo)
>  
> +
> +/* SPARSE
> +builtin:0:0: error: Expected ; at end of declaration
> +builtin:0:0: error: got end-of-input
> +   SPARSE */

Ditto

> diff --git a/validation/run-tests b/validation/run-tests
> new file mode 100755
> index 0000000..ebe8ffc
> --- /dev/null
> +++ b/validation/run-tests
> @@ -0,0 +1,58 @@
> +#!/bin/sh
> +
> +: ${SPARSE=../sparse}
> +: ${SPARSE_FLAGS=}
> +
> +# Use "--update" to update test files with the actual output
> +update=
> +if test "$1" = "--update"; then
> +	update=1
> +	shift
> +fi
> +
> +if test -n "$1"; then
> +	tests="$@"
> +else
> +	tests=`echo *.c`
> +fi
> +
> +bad=
> +for test in $tests; do
> +	base="`basename $test .c`"
> +	exp="$base.exp"
> +	res="$base.res"
> +	diff="$base.diff"
> +	$SPARSE $SPARSE_FLAGS "$test" 2>"$res"
> +	if test "$?" != 0; then
> +		echo "FATAL: $test"
> +		bad="$bad $test"
> +		continue
> +	fi
> +
> +	if test -n "$update"; then
> +		sed -i '/SPARSE/,/SPARSE/d' "$test"
> +		if test -s "$res"; then
> +			echo "/* SPARSE" >>"$test"
> +			cat "$res" >>"$test"
> +			echo "   SPARSE */" >>"$test"
> +		fi
> +	fi

Rather than treating the absence of a SPARSE comment as meaning that the test
has no output, I think I'd rather explicitly include an empty SPARSE comment
in such tests, and then make the absence of a SPARSE comment an error.

> +	sed -n '/SPARSE/,/SPARSE/p' "$test" |grep -v SPARSE >"$exp"
> +	diff -u "$exp" "$res" >"$diff"
> +	if test -s "$diff"; then
> +		echo "FAIL: $test"
> +		bad="$bad $test"
> +		continue
> +	fi
> +
> +	rm -f "$exp" "$res" "$diff"
> +done
> +
> +if test -n "$bad"; then
> +	echo "Failed tests:$bad"
> +	exit 1
> +fi

How about a --verbose that prints the diffs?  I'd prefer that with automated
build systems, for example, so that I can immediately see the test suite
failure in the logs.  Also, --verbose --update should print the diffs it
applies.

- Josh Triplett


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 6/6] Add a simple test script, embed expected results into test files

[Damien Lespiau]

[-- Attachment #1: Type: text/plain, Size: 419 bytes --]

Josh, Pavel,

I happen to have 3 patches in my tree that look like this one. I
thought you might
want to have a look as they appear to handle the points you were wanting to fix.
Patch 3 is a "documentation patch" that explains what I wanted to do.

I attached those patches as gmail does not allow me to answer a thread and
post a patch without losing its formatting.

hope it can help in a way or another.

-- 
Damien

[-- Attachment #2: 0001-test-suite-a-tiny-test-automation-script.patch --]
[-- Type: text/x-patch, Size: 6497 bytes --]

From 68d040139383ef69e3d79776815844c14fa3f496 Mon Sep 17 00:00:00 2001
From: Damien Lespiau <damien.lespiau@gmail.com>
Date: Fri, 15 Jun 2007 18:54:23 +0200
Subject: [PATCH] test-suite: a tiny test automation script

This patch introduces test-suite, a simple script that makes test cases
verification easier. Test cases in the validation directory are annotated
and this script parses them to check if the actual result is the one
expected by the test writer.

Signed-off-by: Damien Lespiau <damien.lespiau@gmail.com>
---
 .gitignore            |    5 +
 Makefile              |    4 +
 validation/test-suite |  239 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 248 insertions(+), 0 deletions(-)
 create mode 100755 validation/test-suite

diff --git a/.gitignore b/.gitignore
index e22a8c6..faa046b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -32,3 +32,8 @@ patches-*
 patches
 series
 
+# test-suite
+*.diff
+*.got
+*.expected
+
diff --git a/Makefile b/Makefile
index 039fe38..7ce9969 100644
--- a/Makefile
+++ b/Makefile
@@ -168,3 +168,7 @@ dist:
 		exit 1 ; \
 	fi
 	git archive --format=tar --prefix=sparse-$(VERSION)/ HEAD^{tree} | gzip -9 > sparse-$(VERSION).tar.gz
+
+check: all
+	$(Q)cd validation && ./test-suite
+
diff --git a/validation/test-suite b/validation/test-suite
new file mode 100755
index 0000000..af1a729
--- /dev/null
+++ b/validation/test-suite
@@ -0,0 +1,239 @@
+#!/bin/sh
+
+#set -x
+
+default_cmd="../sparse \$file"
+tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
+prog_name=`basename $0`
+
+# counts:
+#	- tests that have not been converted to test-suite format
+#	- tests that passed
+#	- tests that failed
+#	- tests that failed but are known to fail
+unhandled_tests=0
+ok_tests=0
+ko_tests=0
+known_ko_tests=0
+
+# defaults to not verbose
+[ -z "$V" ] && V=0
+
+##
+# get_value(key, file) - gets the value of a (key, value) pair in file.
+#
+# returns 0 on success, 1 if the file does not have the key
+get_value()
+{
+	last_result=`grep $1: $2 | sed -e "s/^.*$1:\(.*\)$/\1/"`
+	[ -z "$last_result" ] && return 1
+	return 0
+}
+
+##
+# get_tag(key, file) - does file has the tag key in it ?
+#
+# returns 0 if present, 1 otherwise
+get_tag()
+{
+	last_result=`grep $1 $2`
+	return $?
+}
+
+##
+# verbose(string) - prints string if we are in verbose mode
+verbose()
+{
+	[ "$V" -eq "1" ] && echo "        $1"
+	return 0
+}
+
+##
+# error(string[, die]) - prints an error and exits with value die if given
+error()
+{
+	echo "error: $1"
+	[ -n "$2" ] && exit $2
+	return 0
+}
+
+do_usage()
+{
+echo "$prog_name - a tiny automatic testing script"
+echo "Usage: $prog_name [command] [command arguments]"
+echo
+echo "commands:"
+echo "    none                       runs the whole test suite"
+echo "    single file                runs the test in 'file'"
+echo "    format file [name [cmd]]   helps writing a new test case using cmd"
+echo
+echo "    help                       prints usage"
+}
+
+##
+# do_test(file) - tries to validate a test case
+#
+# it "parses" file, looking for check-* tags and tries to validate
+# the test against an expected result
+# returns:
+#	- 0 if the test passed,
+#	- 1 if it failed,
+#	- 2 if it is not a "test-suite" test.
+do_test()
+{
+	test_failed=0
+	file="$1"
+
+	# can this test be handled by test-suite ?
+	# (it has to have a check-name key in it)
+	get_value "check-name" $file
+	if [ "$?" -eq 1 ]; then
+		unhandled_tests=`expr $unhandled_tests + 1`
+		return 2
+	fi
+	test_name=$last_result
+
+	echo "     TEST    $test_name ($file)"
+
+	# does the test provide a specific command ?
+	cmd=`eval echo $default_cmd`
+	get_value "check-command" $file
+	if [ "$?" -eq "0" ]; then
+		cmd=`eval echo $last_result`
+	fi
+	verbose "Using command       : $cmd"
+
+	# grab the expected exit value
+	get_value "check-exit-value" $file
+	if [ "$?" -eq "0" ]; then
+		expected_exit_value=`echo $last_result | tr -d ' '`
+	else
+		expected_exit_value=0
+	fi
+	verbose "Expecting exit value: $expected_exit_value"
+
+	# grab the expected output
+	get_value "check-output" $file
+	if [ "$?" -eq "0" ]; then
+		echo "$last_result" > "$file".expected
+	else
+		echo -n "" > "$file".expected
+	fi
+
+	# grab the actual output & exit value
+	$cmd 1> $file.got 2> $file.got
+	actual_exit_value=$?
+
+	diff -u "$file".expected "$file".got > "$file".diff
+	if [ "$?" -ne "0" ]; then
+		error "actual output does not match the expected one."
+		error  "see $file.* for further investigation."
+		test_failed=1
+	fi
+
+	if [ "$actual_exit_value" -ne "$expected_exit_value" ]; then
+		error "Actual exit value does not match the expected one."
+		error "expected $expected_exit_value, got $actual_exit_value."
+		test_failed=1
+	fi
+
+	if [ "$test_failed" -eq "1" ]; then
+		ko_tests=`expr $ko_tests + 1`
+		get_tag "check-known-to-fail" $file
+		[ "$?" -eq "0" ] && known_ko_tests=`expr $known_ko_tests + 1`
+		return 1
+	else
+		ok_tests=`expr $ok_tests + 1`
+		return 0
+	fi
+}
+
+do_test_suite()
+{
+	for i in $tests_list; do
+		do_test "$i"
+	done
+
+	# prints some numbers
+	tests_nr=`expr $ok_tests + $ko_tests`
+	echo -n "Out of $tests_nr tests, $ok_tests passed, $ko_tests failed"
+	echo " ($known_ko_tests of them are known to fail)"
+	if [ "$unhandled_tests" -ne "0" ]; then
+		echo "$unhandled_tests tests could not be handled by $prog_name"
+	fi
+}
+
+##
+# do_format(file[, name[, cmd]]) - helps a test writer to format test-suite tags
+do_format()
+{
+	if [ -z "$2" ]; then
+		fname="$1"
+		fcmd=$default_cmd
+	elif [ -z "$3" ]; then
+		fname="$2"
+		fcmd=$default_cmd
+	else
+		fname="$2"
+		fcmd="$3"
+	fi
+	file="$1"
+	cmd=`eval echo $fcmd`
+	$cmd 1> $file.got 2> $file.got
+	fexit_value=$?
+	foutput=`sed -e "s/^\(.*\)/ * check-output:\1/" $file.got`
+	format=`cat <<_EOF
+/*
+ * check-name: $fname
+ *
+ * check-command: $fcmd
+ * check-exit-value: $fexit_value
+ *
+$foutput
+ */
+_EOF
+`
+	echo "$format"
+	return 0
+}
+
+##
+# arg_file(filename) - checks if filename exists
+arg_file()
+{
+	[ -z "$1" ] && {
+		do_usage
+		exit 1
+	}
+	[ -e "$1" ] || {
+		error "Can't open file $1"
+		exit 1
+	}
+	return 0
+}
+
+case "$1" in
+	'')
+		do_test_suite
+		;;
+	single)
+		arg_file "$2"
+		do_test "$2"
+		case "$?" in
+			0) echo "$2 passed !";;
+			1) echo "$2 failed !";;
+			2) echo "$2 can't be handled by $prog_name";;
+		esac
+		;;
+	format)
+		arg_file "$2"
+		do_format "$2" "$3" "$4"
+		;;
+	help | *)
+		do_usage
+		exit 1
+		;;
+esac
+
+exit $?
+
-- 
1.5.2.1.280.g38570


[-- Attachment #3: 0002-Sample-test-suite-test-cases.patch --]
[-- Type: text/x-patch, Size: 2775 bytes --]

From 2a705ec0c1936bd1effae53184cdf4c9ff7ec73d Mon Sep 17 00:00:00 2001
From: Damien Lespiau <damien.lespiau@gmail.com>
Date: Fri, 15 Jun 2007 18:57:06 +0200
Subject: [PATCH] Sample test-suite test cases

A few examples meant to show the use of test-suite

Signed-off-by: Damien Lespiau <damien.lespiau@gmail.com>
---
 validation/bad-assignment.c |   11 +++++++++++
 validation/preprocessor1.c  |    9 +++++++++
 validation/preprocessor2.c  |    9 +++++++++
 validation/preprocessor3.c  |   12 ++++++++++++
 validation/struct-as.c      |    8 ++++++++
 5 files changed, 49 insertions(+), 0 deletions(-)

diff --git a/validation/bad-assignment.c b/validation/bad-assignment.c
index 3b66a11..efdd9a6 100644
--- a/validation/bad-assignment.c
+++ b/validation/bad-assignment.c
@@ -4,3 +4,14 @@ static int foo(int a)
 
 	return a;
 }
+/*
+ * check-name: bad assignement
+ *
+ * check-command: ../sparse $file
+ * check-exit-value: 1
+ *
+ * check-output:bad-assignment.c:3:6: error: Expected ; at end of statement
+ * check-output:bad-assignment.c:3:6: error: got \
+ *
+ * check-known-to-fail
+ */
diff --git a/validation/preprocessor1.c b/validation/preprocessor1.c
index 5ae20aa..ecc8c39 100644
--- a/validation/preprocessor1.c
+++ b/validation/preprocessor1.c
@@ -12,3 +12,12 @@
 #define bar func(
 #define foo bar foo
 foo )
+/*
+ * check-name: preprocessor 1
+ *
+ * check-command: ../sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output:
+ * check-output:foo
+ */
diff --git a/validation/preprocessor2.c b/validation/preprocessor2.c
index 340938e..de00d4f 100644
--- a/validation/preprocessor2.c
+++ b/validation/preprocessor2.c
@@ -13,3 +13,12 @@
 #define BINARY(x, y) x + y
 
 UNARY(TWO)
+/*
+ * check-name: preprocessor 2
+ *
+ * check-command: ../sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output:
+ * check-output:a + b
+ */
diff --git a/validation/preprocessor3.c b/validation/preprocessor3.c
index 71b9acd..8f3cffd 100644
--- a/validation/preprocessor3.c
+++ b/validation/preprocessor3.c
@@ -35,3 +35,15 @@ A()                     // B ( )
 SCAN( A() )             // A ( )
 SCAN(SCAN( A() ))       // B ( )
 SCAN(SCAN(SCAN( A() ))) // A ( )
+/*
+ * check-name: preprocessor 3
+ *
+ * check-command: ../sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output:
+ * check-output:B ( )
+ * check-output:A ( )
+ * check-output:B ( )
+ * check-output:A ( )
+ */
diff --git a/validation/struct-as.c b/validation/struct-as.c
index 86b90d3..19fd3d1 100644
--- a/validation/struct-as.c
+++ b/validation/struct-as.c
@@ -14,3 +14,11 @@ static int broken(struct hello __user *sp)
 {
 	test(&sp->a);
 }
+/*
+ * check-name: address space of structure members
+ *
+ * check-command: ../sparse $file
+ * check-exit-value: 0
+ *
+
+ */
-- 
1.5.2.1.280.g38570


[-- Attachment #4: 0003-test-suite-documentation.patch --]
[-- Type: text/x-patch, Size: 3517 bytes --]

From 22e02751dc5e29536f8e015e8c9880d26eb7b3d5 Mon Sep 17 00:00:00 2001
From: Damien Lespiau <damien.lespiau@gmail.com>
Date: Fri, 15 Jun 2007 18:58:15 +0200
Subject: [PATCH] test-suite documentation

A quick description of test-suite usage.

Signed-off-by: Damien Lespiau <damien.lespiau@gmail.com>
---
 Documentation/test-suite |  112 ++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 112 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/test-suite

diff --git a/Documentation/test-suite b/Documentation/test-suite
new file mode 100644
index 0000000..480355f
--- /dev/null
+++ b/Documentation/test-suite
@@ -0,0 +1,112 @@
+
+
+	Sparse test suite
+	~~~~~~~~~~~~~~~~~
+
+Sparse has a number of test cases in its validation directory. The test-suite
+script aims at making automated checking of these tests possible. It works by
+embedding tags in C comments in the test cases.
+
+check-name: (mandatory)
+	Name of the test.
+
+check-description: (optional)
+	A description of what checks the test.
+
+check-command: (optional)
+	There are different kinds of tests. Some can validate sparse
+	preprocessor while others will use sparse of even other backends of
+	the library. check-command allows you to
+	The '$file' string is special with check-command. It will be expanded
+	to the file name at run time.
+	It defaults to "../sparse $file".
+
+check-exit-value: (optional)
+	The expected exit value of check-command. It defaults to 0.
+
+check-output: (optional)
+	The expected output (stdout and stderr) of check-command. It defaults
+	to no output.
+
+check-known-to-fail (optional)
+	Mark the test as being known to fail.
+
+
+	Using test-suite
+	~~~~~~~~~~~~~~~~
+
+The test-suite script is called trough the check target of the Makefile. It
+will try to check every test case it finds (find validation -name '*.c').
+It can be called to check a single test with:
+
+$ cd validation
+$ ./test-suite single preprocessor1.c
+    TEST  Preprocessor 1 (preprocessor1.c)
+preprocessor1.c passed !
+
+
+	Writing a test
+	~~~~~~~~~~~~~~
+
+test-suite comes with the format command to make a test easier to write:
+
+	test-suite format file [name [cmd]]
+
+name:
+	check-name value. If no name is provided, it defaults to the file name.
+cmd:
+	check-command value. It no cmd if provided, it defaults to
+	"../sparse $file".
+
+The output of the test-suite format command can be redirected into the
+test case to create a test-suite formated file.
+
+$ ./test-suite format bad-assignment.c Assignement >> bad-assignment.c
+$ cat !$
+cat bad-assignment.c
+static int foo(int a)
+{
+        a |=\1;
+
+        return a;
+}
+/*
+ * check-name: Assignement
+ *
+ * check-command: ../sparse $file
+ * check-exit-value: 0
+ *
+ * check-output:bad-assignment.c:3:6: error: Expected ; at end of statement
+ * check-output:bad-assignment.c:3:6: error: got \
+ */
+
+You can define the check-command you want to use for the test. $file will be
+extended to the file name at run time.
+
+$ ./test-suite format preprocessor2.c "Preprocessor 2" \
+		"../sparse -E \$file" >> preprocessor2.c
+$ cat !$
+/*
+ * This one we happen to get right.
+ *
+ * It should result in a simple
+ *
+ *      a + b
+ *
+ * for a proper preprocessor.
+ */
+#define TWO a, b
+
+#define UNARY(x) BINARY(x)
+#define BINARY(x, y) x + y
+
+UNARY(TWO)
+/*
+ * check-name: Preprocessor 2
+ *
+ * check-command: ../sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output:
+ * check-output:a + b
+ */
-- 
1.5.2.1.280.g38570

Re: [PATCH 6/6] Add a simple test script, embed expected results into test files

[Pavel Roskin]

Hello Damien,

On Thu, 2007-06-28 at 20:59 +0200, Damien Lespiau wrote:

> I happen to have 3 patches in my tree that look like this one. I
> thought you might
> want to have a look as they appear to handle the points you were wanting to fix.
> Patch 3 is a "documentation patch" that explains what I wanted to do.

I see your testsuite is much more sophisticated, since it can handle:

- known bad tests
- return codes other than 0
- stdout (combined with stderr)
- test-specific sparse flags

More verbose keywords also look better.  And having the documentation is
another advantage.

I don't think I'll be able to match what you have done (unless you see a
patch from me within the next hour).  Maybe your testsuite could be
applied instead?

-- 
Regards,
Pavel Roskin

Re: [PATCH 6/6] Add a simple test script, embed expected results into test files

[Josh Triplett]

On Thu, 2007-06-28 at 20:59 +0200, Damien Lespiau wrote:
> I happen to have 3 patches in my tree that look like this one. I
> thought you might
> want to have a look as they appear to handle the points you were wanting to fix.
> Patch 3 is a "documentation patch" that explains what I wanted to do.
> 
> I attached those patches as gmail does not allow me to answer a thread and
> post a patch without losing its formatting.

Will review in more detail later, but at first glance this looks quite
good, and very capable.

Two minor things that jump out at me:
      * The need to specify the command line as "../sparse args $file"
        seems somewhat inelegant.  I do like the ability to specify a
        command other than sparse; for example, this could allow
        checking the output of c2xml.  However, I'd prefer not to need a
        path; how about allowing "sparse args $file"?  Also, allowing an
        alternate option "check-options" that just specifies sparse
        flags seems useful, and the default command could do something
        like "sparse $options $file"; that way, you can just say
        "check-options: -E", or "check-options: -Wthingy".
      * The need to prefix every line of output, rather than delimiting
        the start and end of the output, seems painful with large
        amounts of output.

Other than that, I really like this test suite.  Thanks!

- Josh Triplett

Re: [PATCH 6/6] Add a simple test script, embed expected results into test files

[Damien Lespiau]

[-- Attachment #1: Type: text/plain, Size: 2163 bytes --]

> Will review in more detail later, but at first glance this looks quite
> good, and very capable.

I rebased the series against current HEAD to make testing easier.

> Two minor things that jump out at me:
>       * The need to specify the command line as "../sparse args $file"
>         seems somewhat inelegant.
solved: no path in check-command now.

>         Also, allowing an
>         alternate option "check-options" that just specifies sparse
>         flags seems useful, and the default command could do something
>         like "sparse $options $file"; that way, you can just say
>         "check-options: -E", or "check-options: -Wthingy".
Humm, not sure if it's _that_ much useful to have
         check-options: -E -Wthingy
instead of
         check-command: sparse -E -Wthingy $file
for some reason I find the later less confusing, a matter of taste I guess.

>       * The need to prefix every line of output, rather than delimiting
>         the start and end of the output, seems painful with large
>         amounts of output.
Agreed. that's why ./test-suite format helps building such tags

However I still run into a behavior that I cannot explain:

validation$ echo $SHELL
/bin/bash
----------
validation$ ../sparse -E preprocessor/preprocessor19.c

preprocessor/preprocessor19.c:4:9: warning: preprocessor token A redefined
preprocessor/preprocessor19.c:3:9: this was the original definition
y
----------
validation$ ../sparse -E preprocessor/preprocessor19.c 2> o 1> o && cat o

y
processor/preprocessor19.c:4:9: warning: preprocessor token A redefined
preprocessor/preprocessor19.c:3:9: this was the original definition
----------
validation$ ../sparse -E preprocessor/preprocessor19.c &> o && cat o
preprocessor/preprocessor19.c:4:9: warning: preprocessor token A redefined
preprocessor/preprocessor19.c:3:9: this was the original definition

y

If you look carefully the 2> 1> redirections have eaten "pre" of the first
"preprocessor". Using &> show a more suitable behavior but it seems
that &> is not supported by every Bourne shell (for instance dash (the
default /bin/sh of Ubuntu 7.04 does not support &>) Any idea ?

-- 
Damien

[-- Attachment #2: 0001-test-suite-a-tiny-test-automation-script.patch --]
[-- Type: text/x-patch, Size: 6608 bytes --]

From 31cb2f2eafb9e7b88f3fb40500908853bfad24bf Mon Sep 17 00:00:00 2001
From: Damien Lespiau <damien.lespiau@gmail.com>
Date: Fri, 29 Jun 2007 00:57:36 +0200
Subject: [PATCH] test-suite: a tiny test automation script

This patch introduces test-suite, a simple script that makes test cases
verification easier. Test cases in the validation directory are annotated
and this script parses them to check if the actual result is the one
expected by the test writer.

Signed-off-by: Damien Lespiau <damien.lespiau@gmail.com>
---
 .gitignore            |    5 +
 Makefile              |    4 +
 validation/test-suite |  241 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 250 insertions(+), 0 deletions(-)
 create mode 100755 validation/test-suite

diff --git a/.gitignore b/.gitignore
index e22a8c6..faa046b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -32,3 +32,8 @@ patches-*
 patches
 series
 
+# test-suite
+*.diff
+*.got
+*.expected
+
diff --git a/Makefile b/Makefile
index 039fe38..7ce9969 100644
--- a/Makefile
+++ b/Makefile
@@ -168,3 +168,7 @@ dist:
 		exit 1 ; \
 	fi
 	git archive --format=tar --prefix=sparse-$(VERSION)/ HEAD^{tree} | gzip -9 > sparse-$(VERSION).tar.gz
+
+check: all
+	$(Q)cd validation && ./test-suite
+
diff --git a/validation/test-suite b/validation/test-suite
new file mode 100755
index 0000000..c2e0cbb
--- /dev/null
+++ b/validation/test-suite
@@ -0,0 +1,241 @@
+#!/bin/sh
+
+#set -x
+
+default_path=".."
+default_cmd="sparse \$file"
+tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
+prog_name=`basename $0`
+
+# counts:
+#	- tests that have not been converted to test-suite format
+#	- tests that passed
+#	- tests that failed
+#	- tests that failed but are known to fail
+unhandled_tests=0
+ok_tests=0
+ko_tests=0
+known_ko_tests=0
+
+# defaults to not verbose
+[ -z "$V" ] && V=0
+
+##
+# get_value(key, file) - gets the value of a (key, value) pair in file.
+#
+# returns 0 on success, 1 if the file does not have the key
+get_value()
+{
+	last_result=`grep $1: $2 | sed -e "s/^.*$1:\(.*\)$/\1/"`
+	[ -z "$last_result" ] && return 1
+	return 0
+}
+
+##
+# get_tag(key, file) - does file has the tag key in it ?
+#
+# returns 0 if present, 1 otherwise
+get_tag()
+{
+	last_result=`grep $1 $2`
+	return $?
+}
+
+##
+# verbose(string) - prints string if we are in verbose mode
+verbose()
+{
+	[ "$V" -eq "1" ] && echo "        $1"
+	return 0
+}
+
+##
+# error(string[, die]) - prints an error and exits with value die if given
+error()
+{
+	echo "error: $1"
+	[ -n "$2" ] && exit $2
+	return 0
+}
+
+do_usage()
+{
+echo "$prog_name - a tiny automatic testing script"
+echo "Usage: $prog_name [command] [command arguments]"
+echo
+echo "commands:"
+echo "    none                       runs the whole test suite"
+echo "    single file                runs the test in 'file'"
+echo "    format file [name [cmd]]   helps writing a new test case using cmd"
+echo
+echo "    help                       prints usage"
+}
+
+##
+# do_test(file) - tries to validate a test case
+#
+# it "parses" file, looking for check-* tags and tries to validate
+# the test against an expected result
+# returns:
+#	- 0 if the test passed,
+#	- 1 if it failed,
+#	- 2 if it is not a "test-suite" test.
+do_test()
+{
+	test_failed=0
+	file="$1"
+
+	# can this test be handled by test-suite ?
+	# (it has to have a check-name key in it)
+	get_value "check-name" $file
+	if [ "$?" -eq 1 ]; then
+		unhandled_tests=`expr $unhandled_tests + 1`
+		return 2
+	fi
+	test_name=$last_result
+
+	echo "     TEST    $test_name ($file)"
+
+	# does the test provide a specific command ?
+	cmd=`eval echo $default_path/$default_cmd`
+	get_value "check-command" $file
+	if [ "$?" -eq "0" ]; then
+		last_result=`echo $last_result | sed -e 's/^ *//'`
+		cmd=`eval echo $default_path/$last_result`
+	fi
+	verbose "Using command       : $cmd"
+
+	# grab the expected exit value
+	get_value "check-exit-value" $file
+	if [ "$?" -eq "0" ]; then
+		expected_exit_value=`echo $last_result | tr -d ' '`
+	else
+		expected_exit_value=0
+	fi
+	verbose "Expecting exit value: $expected_exit_value"
+
+	# grab the expected output
+	get_value "check-output" $file
+	if [ "$?" -eq "0" ]; then
+		echo "$last_result" > "$file".expected
+	else
+		echo -n "" > "$file".expected
+	fi
+
+	# grab the actual output & exit value
+	$cmd 1> $file.got 2> $file.got
+	actual_exit_value=$?
+
+	diff -u "$file".expected "$file".got > "$file".diff
+	if [ "$?" -ne "0" ]; then
+		error "actual output does not match the expected one."
+		error  "see $file.* for further investigation."
+		test_failed=1
+	fi
+
+	if [ "$actual_exit_value" -ne "$expected_exit_value" ]; then
+		error "Actual exit value does not match the expected one."
+		error "expected $expected_exit_value, got $actual_exit_value."
+		test_failed=1
+	fi
+
+	if [ "$test_failed" -eq "1" ]; then
+		ko_tests=`expr $ko_tests + 1`
+		get_tag "check-known-to-fail" $file
+		[ "$?" -eq "0" ] && known_ko_tests=`expr $known_ko_tests + 1`
+		return 1
+	else
+		ok_tests=`expr $ok_tests + 1`
+		return 0
+	fi
+}
+
+do_test_suite()
+{
+	for i in $tests_list; do
+		do_test "$i"
+	done
+
+	# prints some numbers
+	tests_nr=`expr $ok_tests + $ko_tests`
+	echo -n "Out of $tests_nr tests, $ok_tests passed, $ko_tests failed"
+	echo " ($known_ko_tests of them are known to fail)"
+	if [ "$unhandled_tests" -ne "0" ]; then
+		echo "$unhandled_tests tests could not be handled by $prog_name"
+	fi
+}
+
+##
+# do_format(file[, name[, cmd]]) - helps a test writer to format test-suite tags
+do_format()
+{
+	if [ -z "$2" ]; then
+		fname="$1"
+		fcmd=$default_cmd
+	elif [ -z "$3" ]; then
+		fname="$2"
+		fcmd=$default_cmd
+	else
+		fname="$2"
+		fcmd="$3"
+	fi
+	file="$1"
+	cmd=`eval echo $default_path/$fcmd`
+	$cmd 1> $file.got 2> $file.got
+	fexit_value=$?
+	foutput=`sed -e "s/^\(.*\)/ * check-output:\1/" $file.got`
+	format=`cat <<_EOF
+/*
+ * check-name: $fname
+ *
+ * check-command: $fcmd
+ * check-exit-value: $fexit_value
+ *
+$foutput
+ */
+_EOF
+`
+	echo "$format"
+	return 0
+}
+
+##
+# arg_file(filename) - checks if filename exists
+arg_file()
+{
+	[ -z "$1" ] && {
+		do_usage
+		exit 1
+	}
+	[ -e "$1" ] || {
+		error "Can't open file $1"
+		exit 1
+	}
+	return 0
+}
+
+case "$1" in
+	'')
+		do_test_suite
+		;;
+	single)
+		arg_file "$2"
+		do_test "$2"
+		case "$?" in
+			0) echo "$2 passed !";;
+			1) echo "$2 failed !";;
+			2) echo "$2 can't be handled by $prog_name";;
+		esac
+		;;
+	format)
+		arg_file "$2"
+		do_format "$2" "$3" "$4"
+		;;
+	help | *)
+		do_usage
+		exit 1
+		;;
+esac
+
+exit 0
+
-- 
1.5.2.1.280.g38570


[-- Attachment #3: 0002-Sample-test-suite-test-cases.patch --]
[-- Type: text/x-patch, Size: 2976 bytes --]

From 35fb2463bb71fefdd6865db38cb70bdd0f28d9a4 Mon Sep 17 00:00:00 2001
From: Damien Lespiau <damien.lespiau@gmail.com>
Date: Fri, 29 Jun 2007 00:57:38 +0200
Subject: [PATCH] Sample test-suite test cases

A few examples meant to show the use of test-suite

Signed-off-by: Damien Lespiau <damien.lespiau@gmail.com>
---
 validation/bad-assignment.c             |   11 +++++++++++
 validation/preprocessor/preprocessor1.c |    9 +++++++++
 validation/preprocessor/preprocessor2.c |    9 +++++++++
 validation/preprocessor/preprocessor3.c |   12 ++++++++++++
 validation/struct-as.c                  |    8 ++++++++
 5 files changed, 49 insertions(+), 0 deletions(-)

diff --git a/validation/bad-assignment.c b/validation/bad-assignment.c
index 3b66a11..72b40d3 100644
--- a/validation/bad-assignment.c
+++ b/validation/bad-assignment.c
@@ -4,3 +4,14 @@ static int foo(int a)
 
 	return a;
 }
+/*
+ * check-name: bad assignement
+ *
+ * check-command: sparse $file
+ * check-exit-value: 1
+ *
+ * check-output:bad-assignment.c:3:6: error: Expected ; at end of statement
+ * check-output:bad-assignment.c:3:6: error: got \
+ *
+ * check-known-to-fail
+ */
diff --git a/validation/preprocessor/preprocessor1.c b/validation/preprocessor/preprocessor1.c
index 5ae20aa..0eb8147 100644
--- a/validation/preprocessor/preprocessor1.c
+++ b/validation/preprocessor/preprocessor1.c
@@ -12,3 +12,12 @@
 #define bar func(
 #define foo bar foo
 foo )
+/*
+ * check-name: preprocessor 1
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output:
+ * check-output:foo
+ */
diff --git a/validation/preprocessor/preprocessor2.c b/validation/preprocessor/preprocessor2.c
index 340938e..098fe8c 100644
--- a/validation/preprocessor/preprocessor2.c
+++ b/validation/preprocessor/preprocessor2.c
@@ -13,3 +13,12 @@
 #define BINARY(x, y) x + y
 
 UNARY(TWO)
+/*
+ * check-name: preprocessor 2
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output:
+ * check-output:a + b
+ */
diff --git a/validation/preprocessor/preprocessor3.c b/validation/preprocessor/preprocessor3.c
index 71b9acd..d689a1a 100644
--- a/validation/preprocessor/preprocessor3.c
+++ b/validation/preprocessor/preprocessor3.c
@@ -35,3 +35,15 @@ A()                     // B ( )
 SCAN( A() )             // A ( )
 SCAN(SCAN( A() ))       // B ( )
 SCAN(SCAN(SCAN( A() ))) // A ( )
+/*
+ * check-name: preprocessor 3
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output:
+ * check-output:B ( )
+ * check-output:A ( )
+ * check-output:B ( )
+ * check-output:A ( )
+ */
diff --git a/validation/struct-as.c b/validation/struct-as.c
index 86b90d3..21ee218 100644
--- a/validation/struct-as.c
+++ b/validation/struct-as.c
@@ -14,3 +14,11 @@ static int broken(struct hello __user *sp)
 {
 	test(&sp->a);
 }
+/*
+ * check-name: address space of structure members
+ *
+ * check-command: sparse $file
+ * check-exit-value: 0
+ *
+
+ */
-- 
1.5.2.1.280.g38570


[-- Attachment #4: 0003-test-suite-documentation.patch --]
[-- Type: text/x-patch, Size: 3535 bytes --]

From 2b5aa5e2525fef36335502eb37462c1ceb93e43d Mon Sep 17 00:00:00 2001
From: Damien Lespiau <damien.lespiau@gmail.com>
Date: Fri, 29 Jun 2007 01:18:12 +0200
Subject: [PATCH] test-suite documentation

A quick description of test-suite usage.

Signed-off-by: Damien Lespiau <damien.lespiau@gmail.com>
---
 Documentation/test-suite |  113 ++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 113 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/test-suite

diff --git a/Documentation/test-suite b/Documentation/test-suite
new file mode 100644
index 0000000..b6ef22c
--- /dev/null
+++ b/Documentation/test-suite
@@ -0,0 +1,113 @@
+
+
+	Sparse test suite
+	~~~~~~~~~~~~~~~~~
+
+Sparse has a number of test cases in its validation directory. The test-suite
+script aims at making automated checking of these tests possible. It works by
+embedding tags in C comments in the test cases.
+
+check-name: (mandatory)
+	Name of the test.
+
+check-description: (optional)
+	A description of what checks the test.
+
+check-command: (optional)
+	There are different kinds of tests. Some can validate sparse
+	preprocessor while others will use sparse, cgcc of even other backends
+	of the library. check-command allows you to give a custom command to
+	run the test-case.
+	The '$file' string is special. It will be expanded to the file name at
+	run time.
+	It defaults to "sparse $file".
+
+check-exit-value: (optional)
+	The expected exit value of check-command. It defaults to 0.
+
+check-output: (optional)
+	The expected output (stdout and stderr) of check-command. It defaults
+	to no output.
+
+check-known-to-fail (optional)
+	Mark the test as being known to fail.
+
+
+	Using test-suite
+	~~~~~~~~~~~~~~~~
+
+The test-suite script is called trough the check target of the Makefile. It
+will try to check every test case it finds (find validation -name '*.c').
+It can be called to check a single test with:
+
+$ cd validation
+$ ./test-suite single preprocessor1.c
+    TEST  Preprocessor 1 (preprocessor1.c)
+preprocessor1.c passed !
+
+
+	Writing a test
+	~~~~~~~~~~~~~~
+
+test-suite comes with the format command to make a test easier to write:
+
+	test-suite format file [name [cmd]]
+
+name:
+	check-name value. If no name is provided, it defaults to the file name.
+cmd:
+	check-command value. It no cmd if provided, it defaults to
+	"sparse $file".
+
+The output of the test-suite format command can be redirected into the
+test case to create a test-suite formated file.
+
+$ ./test-suite format bad-assignment.c Assignement >> bad-assignment.c
+$ cat !$
+cat bad-assignment.c
+static int foo(int a)
+{
+        a |=\1;
+
+        return a;
+}
+/*
+ * check-name: Assignement
+ *
+ * check-command: sparse $file
+ * check-exit-value: 0
+ *
+ * check-output:bad-assignment.c:3:6: error: Expected ; at end of statement
+ * check-output:bad-assignment.c:3:6: error: got \
+ */
+
+You can define the check-command you want to use for the test. $file will be
+extended to the file name at run time.
+
+$ ./test-suite format preprocessor2.c "Preprocessor 2" \
+		"sparse -E \$file" >> preprocessor2.c
+$ cat !$
+/*
+ * This one we happen to get right.
+ *
+ * It should result in a simple
+ *
+ *      a + b
+ *
+ * for a proper preprocessor.
+ */
+#define TWO a, b
+
+#define UNARY(x) BINARY(x)
+#define BINARY(x, y) x + y
+
+UNARY(TWO)
+/*
+ * check-name: Preprocessor 2
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output:
+ * check-output:a + b
+ */
-- 
1.5.2.1.280.g38570

Re: [PATCH 6/6] Add a simple test script, embed expected results into test files

[Josh Triplett]

On Fri, 2007-06-29 at 02:13 +0200, Damien Lespiau wrote:
> > Will review in more detail later, but at first glance this looks quite
> > good, and very capable.
> 
> I rebased the series against current HEAD to make testing easier.

Thanks.

> > Two minor things that jump out at me:
> >       * The need to specify the command line as "../sparse args $file"
> >         seems somewhat inelegant.
> solved: no path in check-command now.

Excellent.

> >         Also, allowing an
> >         alternate option "check-options" that just specifies sparse
> >         flags seems useful, and the default command could do something
> >         like "sparse $options $file"; that way, you can just say
> >         "check-options: -E", or "check-options: -Wthingy".
> Humm, not sure if it's _that_ much useful to have
>          check-options: -E -Wthingy
> instead of
>          check-command: sparse -E -Wthingy $file
> for some reason I find the later less confusing, a matter of taste I guess.

I don't mind going with the latter for now and considering the former
for a later patch.

> >       * The need to prefix every line of output, rather than delimiting
> >         the start and end of the output, seems painful with large
> >         amounts of output.
> Agreed. that's why ./test-suite format helps building such tags

Yes, but I still prefer the delimited format for readability.

> However I still run into a behavior that I cannot explain:
> 
> validation$ echo $SHELL
> /bin/bash
> ----------
> validation$ ../sparse -E preprocessor/preprocessor19.c
> 
> preprocessor/preprocessor19.c:4:9: warning: preprocessor token A redefined
> preprocessor/preprocessor19.c:3:9: this was the original definition
> y
> ----------
> validation$ ../sparse -E preprocessor/preprocessor19.c 2> o 1> o && cat o
> 
> y
> processor/preprocessor19.c:4:9: warning: preprocessor token A redefined
> preprocessor/preprocessor19.c:3:9: this was the original definition
> ----------
> validation$ ../sparse -E preprocessor/preprocessor19.c &> o && cat o
> preprocessor/preprocessor19.c:4:9: warning: preprocessor token A redefined
> preprocessor/preprocessor19.c:3:9: this was the original definition
> 
> y
> 
> If you look carefully the 2> 1> redirections have eaten "pre" of the first
> "preprocessor". Using &> show a more suitable behavior but it seems
> that &> is not supported by every Bourne shell (for instance dash (the
> default /bin/sh of Ubuntu 7.04 does not support &>) Any idea ?

You can't redirect two things independently to the same file; that will
open the file twice, and the writes will conflict, giving exactly the
result you saw.  > o 2>&1 should work; it has exactly the same effect as
&> .

- Josh Triplett

Re: [PATCH 6/6] Add a simple test script, embed expected results into test files

[Damien Lespiau]

[-- Attachment #1: Type: text/plain, Size: 1386 bytes --]

A small update,

> Yes, but I still prefer the delimited format for readability.

Ok: expected output is now between check-output-start / check-output-end
tags.
I've added a clean-check target in the Makefile, updated the provided
test-cases and documentation.

> You can't redirect two things independently to the same file; that will
> open the file twice, and the writes will conflict, giving exactly the
> result you saw.  > o 2>&1 should work; it has exactly the same effect as
> &> .

For some reason I was expecting that:

[pid  8877] open("o", O_WRONLY|O_CREAT|O_TRUNC| \
        O_LARGEFILE, 0666) = 3
[pid  8877] dup2(3, 1)                  = 1
[pid  8877] close(3)                    = 0
[pid  8877] open("o", O_WRONLY|O_CREAT|O_TRUNC| \
        O_LARGEFILE, 0666) = 3
[pid  8877] dup2(3, 2)                  = 2
[pid  8877] close(3)                    = 0

and

[pid 10777] open("o", O_WRONLY|O_CREAT|O_TRUNC| \
        O_LARGEFILE, 0666) = 3
[pid 10777] dup2(3, 1)                  = 1
[pid 10777] close(3)                    = 0
[pid 10777] dup2(1, 2)                  = 2

followed by:

[pid  xxxxx] write(2, "validation/preprocessor19.c:4:9:"..., 73) = 73
[pid  xxxxx] write(2, "validation/preprocessor19.c:3:9:"..., 66) = 66
[pid  xxxxx] write(1, "\ny\n", 3)        = o

to give the same result and that the first one would not eat characters.
I was wrong.

-- 
Damien

[-- Attachment #2: 0001-test-suite-a-tiny-test-automation-script.patch --]
[-- Type: text/x-patch, Size: 6857 bytes --]

From 4977f1c73fcb53d260facbee0be60c78bdb7af76 Mon Sep 17 00:00:00 2001
From: Damien Lespiau <damien.lespiau@gmail.com>
Date: Mon, 2 Jul 2007 06:41:34 +0200
Subject: [PATCH] test-suite: a tiny test automation script

This patch introduces test-suite, a simple script that makes test cases
verification easier. Test cases in the validation directory are annotated
and this script parses them to check if the actual result is the one
expected by the test writer.

Signed-off-by: Damien Lespiau <damien.lespiau@gmail.com>
---
 .gitignore            |    5 +
 Makefile              |    7 ++
 validation/test-suite |  252 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 264 insertions(+), 0 deletions(-)
 create mode 100755 validation/test-suite

diff --git a/.gitignore b/.gitignore
index e22a8c6..faa046b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -32,3 +32,8 @@ patches-*
 patches
 series
 
+# test-suite
+*.diff
+*.got
+*.expected
+
diff --git a/Makefile b/Makefile
index 039fe38..6bfb2ea 100644
--- a/Makefile
+++ b/Makefile
@@ -168,3 +168,10 @@ dist:
 		exit 1 ; \
 	fi
 	git archive --format=tar --prefix=sparse-$(VERSION)/ HEAD^{tree} | gzip -9 > sparse-$(VERSION).tar.gz
+
+check: all
+	$(Q)cd validation && ./test-suite
+
+clean-check:
+	find validation/ -name "*.c.[egd]*" -exec rm {} \;
+
diff --git a/validation/test-suite b/validation/test-suite
new file mode 100755
index 0000000..6b9d1b7
--- /dev/null
+++ b/validation/test-suite
@@ -0,0 +1,252 @@
+#!/bin/sh
+
+#set -x
+
+default_path=".."
+default_cmd="sparse \$file"
+tests_list=`find . -name '*.c' | sed -e 's#^\./\(.*\)#\1#' | sort`
+prog_name=`basename $0`
+
+# counts:
+#	- tests that have not been converted to test-suite format
+#	- tests that passed
+#	- tests that failed
+#	- tests that failed but are known to fail
+unhandled_tests=0
+ok_tests=0
+ko_tests=0
+known_ko_tests=0
+
+# defaults to not verbose
+[ -z "$V" ] && V=0
+
+##
+# get_value(key, file) - gets the value of a (key, value) pair in file.
+#
+# returns 0 on success, 1 if the file does not have the key
+get_value()
+{
+	last_result=`grep $1: $2 | sed -e "s/^.*$1:\(.*\)$/\1/"`
+	[ -z "$last_result" ] && return 1
+	return 0
+}
+
+##
+# get_tag(key, file) - does file has the tag key in it ?
+#
+# returns 0 if present, 1 otherwise
+get_tag()
+{
+	last_result=`grep $1 $2`
+	return $?
+}
+
+##
+# verbose(string) - prints string if we are in verbose mode
+verbose()
+{
+	[ "$V" -eq "1" ] && echo "        $1"
+	return 0
+}
+
+##
+# error(string[, die]) - prints an error and exits with value die if given
+error()
+{
+	echo "error: $1"
+	[ -n "$2" ] && exit $2
+	return 0
+}
+
+do_usage()
+{
+echo "$prog_name - a tiny automatic testing script"
+echo "Usage: $prog_name [command] [command arguments]"
+echo
+echo "commands:"
+echo "    none                       runs the whole test suite"
+echo "    single file                runs the test in 'file'"
+echo "    format file [name [cmd]]   helps writing a new test case using cmd"
+echo
+echo "    help                       prints usage"
+}
+
+##
+# do_test(file) - tries to validate a test case
+#
+# it "parses" file, looking for check-* tags and tries to validate
+# the test against an expected result
+# returns:
+#	- 0 if the test passed,
+#	- 1 if it failed,
+#	- 2 if it is not a "test-suite" test.
+do_test()
+{
+	test_failed=0
+	file="$1"
+
+	# can this test be handled by test-suite ?
+	# (it has to have a check-name key in it)
+	get_value "check-name" $file
+	if [ "$?" -eq 1 ]; then
+		unhandled_tests=`expr $unhandled_tests + 1`
+		return 2
+	fi
+	test_name=$last_result
+
+	echo "     TEST    $test_name ($file)"
+
+	# does the test provide a specific command ?
+	cmd=`eval echo $default_path/$default_cmd`
+	get_value "check-command" $file
+	if [ "$?" -eq "0" ]; then
+		last_result=`echo $last_result | sed -e 's/^ *//'`
+		cmd=`eval echo $default_path/$last_result`
+	fi
+	verbose "Using command       : $cmd"
+
+	# grab the expected exit value
+	get_value "check-exit-value" $file
+	if [ "$?" -eq "0" ]; then
+		expected_exit_value=`echo $last_result | tr -d ' '`
+	else
+		expected_exit_value=0
+	fi
+	verbose "Expecting exit value: $expected_exit_value"
+
+	# grab the expected output
+	sed -n '/check-output-start/,/check-output-end/p' $file \
+		| grep -v check-output > "$file".expected
+
+	# grab the actual output & exit value
+	$cmd 1> $file.got 2>&1
+	actual_exit_value=$?
+
+	diff -u "$file".expected "$file".got > "$file".diff
+	if [ "$?" -ne "0" ]; then
+		error "actual output does not match the expected one."
+		error  "see $file.* for further investigation."
+		test_failed=1
+	fi
+
+	if [ "$actual_exit_value" -ne "$expected_exit_value" ]; then
+		error "Actual exit value does not match the expected one."
+		error "expected $expected_exit_value, got $actual_exit_value."
+		test_failed=1
+	fi
+
+	if [ "$test_failed" -eq "1" ]; then
+		ko_tests=`expr $ko_tests + 1`
+		get_tag "check-known-to-fail" $file
+		[ "$?" -eq "0" ] && known_ko_tests=`expr $known_ko_tests + 1`
+		return 1
+	else
+		ok_tests=`expr $ok_tests + 1`
+		return 0
+	fi
+}
+
+do_test_suite()
+{
+	for i in $tests_list; do
+		do_test "$i"
+	done
+
+	# prints some numbers
+	tests_nr=`expr $ok_tests + $ko_tests`
+	echo -n "Out of $tests_nr tests, $ok_tests passed, $ko_tests failed"
+	echo " ($known_ko_tests of them are known to fail)"
+	if [ "$unhandled_tests" -ne "0" ]; then
+		echo "$unhandled_tests tests could not be handled by $prog_name"
+	fi
+}
+
+##
+# do_format(file[, name[, cmd]]) - helps a test writer to format test-suite tags
+do_format()
+{
+	if [ -z "$2" ]; then
+		fname="$1"
+		fcmd=$default_cmd
+	elif [ -z "$3" ]; then
+		fname="$2"
+		fcmd=$default_cmd
+	else
+		fname="$2"
+		fcmd="$3"
+	fi
+	file="$1"
+	cmd=`eval echo $default_path/$fcmd`
+	$cmd 1> $file.got 2>&1
+	fexit_value=$?
+	#foutput=`sed -e "s/^\(.*\)/ * check-output:\1/" $file.got`
+	foutput=`cat $file.got`
+	if [ -z "$foutput" ]; then
+	format=`cat <<_EOF
+/*
+ * check-name: $fname
+ *
+ * check-command: $fcmd
+ * check-exit-value: $fexit_value
+ */
+_EOF
+`
+	else
+	format=`cat <<_EOF
+/*
+ * check-name: $fname
+ *
+ * check-command: $fcmd
+ * check-exit-value: $fexit_value
+ *
+ * check-output-start
+$foutput
+ * check-output-end
+ */
+_EOF
+`
+	fi
+	echo "$format"
+	return 0
+}
+
+##
+# arg_file(filename) - checks if filename exists
+arg_file()
+{
+	[ -z "$1" ] && {
+		do_usage
+		exit 1
+	}
+	[ -e "$1" ] || {
+		error "Can't open file $1"
+		exit 1
+	}
+	return 0
+}
+
+case "$1" in
+	'')
+		do_test_suite
+		;;
+	single)
+		arg_file "$2"
+		do_test "$2"
+		case "$?" in
+			0) echo "$2 passed !";;
+			1) echo "$2 failed !";;
+			2) echo "$2 can't be handled by $prog_name";;
+		esac
+		;;
+	format)
+		arg_file "$2"
+		do_format "$2" "$3" "$4"
+		;;
+	help | *)
+		do_usage
+		exit 1
+		;;
+esac
+
+exit 0
+
-- 
1.5.2.1.280.g38570


[-- Attachment #3: 0002-Sample-test-suite-test-cases.patch --]
[-- Type: text/x-patch, Size: 3613 bytes --]

From 922bed97f061491ff2e7ca11e22161726fd6d7e5 Mon Sep 17 00:00:00 2001
From: Damien Lespiau <damien.lespiau@gmail.com>
Date: Mon, 2 Jul 2007 06:41:39 +0200
Subject: [PATCH] Sample test-suite test cases

A few examples meant to show the use of test-suite

Signed-off-by: Damien Lespiau <damien.lespiau@gmail.com>
---
 validation/bad-assignment.c              |   13 +++++++++++++
 validation/preprocessor/preprocessor1.c  |   11 +++++++++++
 validation/preprocessor/preprocessor19.c |   13 +++++++++++++
 validation/preprocessor/preprocessor2.c  |   11 +++++++++++
 validation/preprocessor/preprocessor3.c  |   14 ++++++++++++++
 validation/struct-as.c                   |    6 ++++++
 6 files changed, 68 insertions(+), 0 deletions(-)

diff --git a/validation/bad-assignment.c b/validation/bad-assignment.c
index 3b66a11..28e8c8d 100644
--- a/validation/bad-assignment.c
+++ b/validation/bad-assignment.c
@@ -4,3 +4,16 @@ static int foo(int a)
 
 	return a;
 }
+/*
+ * check-name: bad assignement
+ *
+ * check-command: sparse $file
+ * check-exit-value: 1
+ *
+ * check-output-start
+bad-assignment.c:3:6: error: Expected ; at end of statement
+bad-assignment.c:3:6: error: got \
+ * check-output-end
+ *
+ * check-known-to-fail
+ */
diff --git a/validation/preprocessor/preprocessor1.c b/validation/preprocessor/preprocessor1.c
index 5ae20aa..a02ccf6 100644
--- a/validation/preprocessor/preprocessor1.c
+++ b/validation/preprocessor/preprocessor1.c
@@ -12,3 +12,14 @@
 #define bar func(
 #define foo bar foo
 foo )
+/*
+ * check-name: Preprocessor #1
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output-start
+
+foo
+ * check-output-end
+ */
diff --git a/validation/preprocessor/preprocessor19.c b/validation/preprocessor/preprocessor19.c
index 0f7da47..c036c2e 100644
--- a/validation/preprocessor/preprocessor19.c
+++ b/validation/preprocessor/preprocessor19.c
@@ -3,3 +3,16 @@
 #define A x
 #define A y
 A
+/*
+ * check-name: Preprocessor #19
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output-start
+preprocessor/preprocessor19.c:4:9: warning: preprocessor token A redefined
+preprocessor/preprocessor19.c:3:9: this was the original definition
+
+y
+ * check-output-end
+ */
diff --git a/validation/preprocessor/preprocessor2.c b/validation/preprocessor/preprocessor2.c
index 340938e..c95641d 100644
--- a/validation/preprocessor/preprocessor2.c
+++ b/validation/preprocessor/preprocessor2.c
@@ -13,3 +13,14 @@
 #define BINARY(x, y) x + y
 
 UNARY(TWO)
+/*
+ * check-name: Preprocessor #2
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output-start
+
+a + b
+ * check-output-end
+ */
diff --git a/validation/preprocessor/preprocessor3.c b/validation/preprocessor/preprocessor3.c
index 71b9acd..0735667 100644
--- a/validation/preprocessor/preprocessor3.c
+++ b/validation/preprocessor/preprocessor3.c
@@ -35,3 +35,17 @@ A()                     // B ( )
 SCAN( A() )             // A ( )
 SCAN(SCAN( A() ))       // B ( )
 SCAN(SCAN(SCAN( A() ))) // A ( )
+/*
+ * check-name: Preprocessor #3
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output-start
+
+B ( )
+A ( )
+B ( )
+A ( )
+ * check-output-end
+ */
diff --git a/validation/struct-as.c b/validation/struct-as.c
index 86b90d3..31bdc88 100644
--- a/validation/struct-as.c
+++ b/validation/struct-as.c
@@ -14,3 +14,9 @@ static int broken(struct hello __user *sp)
 {
 	test(&sp->a);
 }
+/*
+ * check-name: Address space of a struct member
+ *
+ * check-command: sparse $file
+ * check-exit-value: 0
+ */
-- 
1.5.2.1.280.g38570


[-- Attachment #4: 0003-test-suite-documentation.patch --]
[-- Type: text/x-patch, Size: 3640 bytes --]

From 3881b34f3e103aecdc4581bfcef1925d7a8636b6 Mon Sep 17 00:00:00 2001
From: Damien Lespiau <damien.lespiau@gmail.com>
Date: Mon, 2 Jul 2007 06:49:04 +0200
Subject: [PATCH] test-suite documentation

A quick description of test-suite usage.

Signed-off-by: Damien Lespiau <damien.lespiau@gmail.com>
---
 Documentation/test-suite |  112 ++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 112 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/test-suite

diff --git a/Documentation/test-suite b/Documentation/test-suite
new file mode 100644
index 0000000..1755156
--- /dev/null
+++ b/Documentation/test-suite
@@ -0,0 +1,112 @@
+
+
+	Sparse test suite
+	~~~~~~~~~~~~~~~~~
+
+Sparse has a number of test cases in its validation directory. The test-suite
+script aims at making automated checking of these tests possible. It works by
+embedding tags in C comments in the test cases.
+
+check-name: (mandatory)
+	Name of the test.
+
+check-description: (optional)
+	A description of what checks the test.
+
+check-command: (optional)
+	There are different kinds of tests. Some can validate sparse
+	preprocessor while others will use sparse, cgcc of even other backends
+	of the library. check-command allows you to give a custom command to
+	run the test-case.
+	The '$file' string is special. It will be expanded to the file name at
+	run time.
+	It defaults to "sparse $file".
+
+check-exit-value: (optional)
+	The expected exit value of check-command. It defaults to 0.
+
+check-output-start / check-output-end (optional)
+	The expected output (stdout and stderr) of check-command lies between
+	those two tags. It defaults to no output.
+
+check-known-to-fail (optional)
+	Mark the test as being known to fail.
+
+
+	Using test-suite
+	~~~~~~~~~~~~~~~~
+
+The test-suite script is called trough the check target of the Makefile. It
+will try to check every test case it finds (find validation -name '*.c').
+
+It can be called to check a single test with:
+$ cd validation
+$ ./test-suite single preprocessor/preprocessor1.c
+     TEST     Preprocessor #1 (preprocessor/preprocessor1.c)
+preprocessor/preprocessor1.c passed !
+
+
+	Writing a test
+	~~~~~~~~~~~~~~
+
+test-suite comes with a format command to make a test easier to write:
+
+	test-suite format file [name [cmd]]
+
+name:
+	check-name value. If no name is provided, it defaults to the file name.
+cmd:
+	check-command value. It no cmd if provided, it defaults to
+	"sparse $file".
+
+The output of the test-suite format command can be redirected into the
+test case to create a test-suite formated file.
+
+$ ./test-suite format bad-assignment.c Assignement >> bad-assignment.c
+$ cat !$
+cat bad-assignment.c
+/*
+ * check-name: bad assignement
+ *
+ * check-command: sparse $file
+ * check-exit-value: 1
+ *
+ * check-output-start
+bad-assignment.c:3:6: error: Expected ; at end of statement
+bad-assignment.c:3:6: error: got \
+ * check-output-end
+ */
+
+You can define the check-command you want to use for the test. $file will be
+extended to the file name at run time.
+
+$ ./test-suite format validation/preprocessor2.c "Preprocessor #2" \
+		"sparse -E \$file" >> validation/preprocessor2.c
+$ cat !$
+cat validation/preprocessor2.c
+/*
+ * This one we happen to get right.
+ *
+ * It should result in a simple
+ *
+ *	a + b
+ *
+ * for a proper preprocessor.
+ */
+#define TWO a, b
+
+#define UNARY(x) BINARY(x)
+#define BINARY(x, y) x + y
+
+UNARY(TWO)
+/*
+ * check-name: Preprocessor #2
+ *
+ * check-command: sparse -E $file
+ * check-exit-value: 0
+ *
+ * check-output-start
+
+a + b
+ * check-output-end
+ */
-- 
1.5.2.1.280.g38570

Re: [PATCH 6/6] Add a simple test script, embed expected results into test files

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 1884 bytes --]

Damien Lespiau wrote:
> A small update,
> 
>> Yes, but I still prefer the delimited format for readability.
> 
> Ok: expected output is now between check-output-start / check-output-end
> tags.
> I've added a clean-check target in the Makefile, updated the provided
> test-cases and documentation.

Thanks!  Will review and apply soon.

>> You can't redirect two things independently to the same file; that will
>> open the file twice, and the writes will conflict, giving exactly the
>> result you saw.  > o 2>&1 should work; it has exactly the same effect as
>> &> .
> 
> For some reason I was expecting that:
> 
> [pid  8877] open("o", O_WRONLY|O_CREAT|O_TRUNC| \
>         O_LARGEFILE, 0666) = 3
> [pid  8877] dup2(3, 1)                  = 1
> [pid  8877] close(3)                    = 0
> [pid  8877] open("o", O_WRONLY|O_CREAT|O_TRUNC| \
>         O_LARGEFILE, 0666) = 3
> [pid  8877] dup2(3, 2)                  = 2
> [pid  8877] close(3)                    = 0
> 
> and
> 
> [pid 10777] open("o", O_WRONLY|O_CREAT|O_TRUNC| \
>         O_LARGEFILE, 0666) = 3
> [pid 10777] dup2(3, 1)                  = 1
> [pid 10777] close(3)                    = 0
> [pid 10777] dup2(1, 2)                  = 2
> 
> followed by:
> 
> [pid  xxxxx] write(2, "validation/preprocessor19.c:4:9:"..., 73) = 73
> [pid  xxxxx] write(2, "validation/preprocessor19.c:3:9:"..., 66) = 66
> [pid  xxxxx] write(1, "\ny\n", 3)        = o
> 
> to give the same result and that the first one would not eat characters.

In the first case, fd 1 and fd 2 represent separate open files, and have
independent file offsets; thus, writes through the two file descriptors
overwrite each other.  In the second case, fd 1 and fd 2 represent the same
open file, and have the same file offset, so writes to either one appear in
the file in sequence.

- Josh Triplett


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 6/6] Add a simple test script, embed expected results into test files

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

I've applied Damien's test-suite patches.  I've started making use of the test
suite, and adding the test-suite comments to more files in validation.  I find
it a pleasure to work with; thanks, Damien!

Thanks to Pavel as well, for pushing automated testing to begin with, and for
contributing to the final result.

Damien, I noticed one minor issue with test-suite: it seems to eat whitespace
in the Sparse output, making "test-suite format" for several of the
preprocessor tests (such as preprocessor18.c) not match the actual output.  I
would guess some kind of quoting problem.  Here documents may not do what you
want.  Could you take a look at that, please?

- Josh Triplett


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 6/6] Add a simple test script, embed expected results into test files

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 739 bytes --]

Josh Triplett wrote:
> Damien, I noticed one minor issue with test-suite: it seems to eat whitespace
> in the Sparse output, making "test-suite format" for several of the
> preprocessor tests (such as preprocessor18.c) not match the actual output.  I
> would guess some kind of quoting problem.  Here documents may not do what you
> want.  Could you take a look at that, please?

Actually, this led me to a more important issue: I don't think the test suite
output can mix stdout and stderr safely, because the order may differ if both
have data.  Look at preprocessor18.c for an example.  I think the test suite
needs to have a block for check-stdout-{start,end} and a block for
check-stderr-{start,end}.

- Josh Triplett


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [PATCH 6/6] Add a simple test script, embed expected results into test files

[Damien Lespiau]

(I removed Pavel from the cc: list as he probably does not want to read such
mails)

> Actually, this led me to a more important issue: I don't think the test suite
> output can mix stdout and stderr safely, because the order may differ if both
> have data.  Look at preprocessor18.c for an example.  I think the test suite
> needs to have a block for check-stdout-{start,end} and a block for
> check-stderr-{start,end}.

Absolutely !

 ../sparse -E preprocessor/preprocessor18.c 1> foo 2>&1 && cat foo
and
../sparse -E preprocessor/preprocessor18.c

are not giving the same result and I had the same conclusion as yours
before going to bed. I was actually thinking about splitting the output
between stderr and stdout too. I added 1: or 2: at the beginning of
output lines but I think I like check-(stdout|stderr)-(start,end) better.

However I bumped into another issue:
$ hexdump -C bar
00000000  0a 0a 78 78 78 0a 0a 0a                       |..xxx...|
$ foo=`cat bar`
$ echo "$foo" | hexdump -C
00000000  0a 0a 78 78 78 0a                                |..xxx.|

it eats the last new lines but one and not the first '\n's !
I still have to sort this out...

-- 
Damien

Re: [PATCH 6/6] Add a simple test script, embed expected results into test files

[Anderson Lizardo]

On 7/9/07, Damien Lespiau <damien.lespiau@gmail.com> wrote:
> However I bumped into another issue:
> $ hexdump -C bar
> 00000000  0a 0a 78 78 78 0a 0a 0a                       |..xxx...|
> $ foo=`cat bar`
> $ echo "$foo" | hexdump -C
> 00000000  0a 0a 78 78 78 0a                                |..xxx.|
>
> it eats the last new lines but one and not the first '\n's !
> I still have to sort this out...

Actually, it eats both newlines... The trailing newline that still
appears is from "echo" command, which appends a newline by default.
Try:

echo -n "$foo" | hexdump -C

instead. Anyway, this is from "man bash":

"Command Substitution
[...]
Bash performs the expansion by executing command and replacing the
command substitution with the standard output of the command, with any
trailing newlines  deleted. Embedded newlines are not deleted, but
they may be removed during word splitting."

So I suppose this is an expected behaviour.

Regards,
-- 
Anderson Lizardo

Re: [PATCH 1/6] Bitfield without explicit sign should be a warning, not an error

[Josh Triplett]

[-- Attachment #1: Type: text/plain, Size: 174 bytes --]

Pavel Roskin wrote:
> The -Wdefault-bitfield-sign is supposed to control a warning, just like
> other -W options.

I've applied this patch; thanks!

- Josh Triplett


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]