[Cocci] [RFC 0/2] coccinelle: extend test coverage

[Cocci] [RFC 0/2] coccinelle: extend test coverage

[Luis R. Rodriguez]

I wanted to extend tests coverage for coccinelle but noticed
that tests are limited to single c file, these are attached
to respective .res files, but this also means that you have
a direct relationship and requirement between tests from
a single C file and a respective cocci file.

To go wild we want to be able to support really complex
types of examples. Fortunatley the backports project has
been using Coccinelle extensively daily, when a backports
developers needed to replace a series of patches with a
Coccinelle patch pycocci was extended to provide 
SmPL <--> Patch equivalence support. We can easily take
advantage of this to extend Coccinelle test cases way
beyond a single C file, but to more complex real life
style projects.

I don't think we should convert all test cases to this
new mechanism but, I'd like to see what are folks' thoughts
over using pycocci SmPL <--> Patch equivalence proof support
to help expand the tests cases for Coccinelle.

Luis R. Rodriguez (2):
  pycocci: add missing new line
  tests: convert sp.cocci test to pycocci proof

 Makefile                            | 1 +
 scripts/test.sh                     | 8 ++++++++
 tests/{sp.cocci => 0001-sp.cocci}   | 0
 tests/0001-sp/replace-typedef.patch | 9 +++++++++
 tests/{ => code/0001-sp}/sp.c       | 0
 tests/sp.res                        | 4 ----
 tools/pycocci                       | 2 +-
 7 files changed, 19 insertions(+), 5 deletions(-)
 create mode 100755 scripts/test.sh
 rename tests/{sp.cocci => 0001-sp.cocci} (100%)
 create mode 100644 tests/0001-sp/replace-typedef.patch
 rename tests/{ => code/0001-sp}/sp.c (100%)
 delete mode 100644 tests/sp.res

-- 
2.7.2

[Cocci] [RFC 1/2] pycocci: add missing new line

[Luis R. Rodriguez]

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 tools/pycocci | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/pycocci b/tools/pycocci
index 266957bd90cd..43bb9a4dd412 100755
--- a/tools/pycocci
+++ b/tools/pycocci
@@ -608,7 +608,7 @@ def _main():
         cmd = [ '--stat', patch_branch_name + ".." + smpl_branch_name ]
         diff_stat = git_diff(tree=args.target_dir, extra_args = cmd)
         if len(diff_stat) == 0:
-            logwrite('\nSmPL patch fully replaces patch series!')
+            logwrite('\nSmPL patch fully replaces patch series!\n')
         else:
             logwrite('\nDifferences found:\n\n')
             logwrite('Change directory to %s and run:\n\n\tgit diff %s..%s\n\n' % (args.target_dir, patch_branch_name, smpl_branch_name))
-- 
2.7.2

[Cocci] [RFC 2/2] tests: convert sp.cocci test to pycocci proof

[Luis R. Rodriguez]

Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
---
 Makefile                            | 1 +
 scripts/test.sh                     | 8 ++++++++
 tests/{sp.cocci => 0001-sp.cocci}   | 0
 tests/0001-sp/replace-typedef.patch | 9 +++++++++
 tests/{ => code/0001-sp}/sp.c       | 0
 tests/sp.res                        | 4 ----
 6 files changed, 18 insertions(+), 4 deletions(-)
 create mode 100755 scripts/test.sh
 rename tests/{sp.cocci => 0001-sp.cocci} (100%)
 create mode 100644 tests/0001-sp/replace-typedef.patch
 rename tests/{ => code/0001-sp}/sp.c (100%)
 delete mode 100644 tests/sp.res

diff --git a/Makefile b/Makefile
index a98eefd2d83b..312f287fd61d 100644
--- a/Makefile
+++ b/Makefile
@@ -488,6 +488,7 @@ check: scripts/spatch
 		COCCINELLE_HOME="$$(pwd)" ./scripts/spatch --sp-file demos/hello/hello-python.cocci demos/hello/helloworld.c --very-quiet | grep -q 'Hello at: 2'; fi
 	@$(ECHO) running the test suite
 	COCCINELLE_HOME="$$(pwd)" ./scripts/spatch --testall --no-update-score-file
+	COCCINELLE_HOME="$$(pwd)" ./scripts/test.sh
 
 # -inline 0  to see all the functions in the profile.
 # Can also use the profile framework in commons/ and run your program
diff --git a/scripts/test.sh b/scripts/test.sh
new file mode 100755
index 000000000000..efaaa88e67be
--- /dev/null
+++ b/scripts/test.sh
@@ -0,0 +1,8 @@
+#!/bin/bash
+set -e
+
+for i in tests/0001-sp.cocci; do
+	TEST=${i%%.cocci}
+	CODE_TEST=tests/code/$(basename $TEST)
+	./tools/pycocci -v -s $i $CODE_TEST
+done
diff --git a/tests/sp.cocci b/tests/0001-sp.cocci
similarity index 100%
rename from tests/sp.cocci
rename to tests/0001-sp.cocci
diff --git a/tests/0001-sp/replace-typedef.patch b/tests/0001-sp/replace-typedef.patch
new file mode 100644
index 000000000000..5097b94cbfca
--- /dev/null
+++ b/tests/0001-sp/replace-typedef.patch
@@ -0,0 +1,9 @@
+--- a/sp.c
++++ b/sp.c
+@@ -1,4 +1,4 @@
+-typedef struct {
++struct name {
+ 	unsigned long gcr;
+ 	struct pci_dev *pci;
+-} snd_card_als4000_t;
++};
diff --git a/tests/sp.c b/tests/code/0001-sp/sp.c
similarity index 100%
rename from tests/sp.c
rename to tests/code/0001-sp/sp.c
diff --git a/tests/sp.res b/tests/sp.res
deleted file mode 100644
index a97fecceb3f0..000000000000
--- a/tests/sp.res
+++ /dev/null
@@ -1,4 +0,0 @@
-struct name {
-	unsigned long gcr;
-	struct pci_dev *pci;
-};
-- 
2.7.2

[Cocci] [RFC 1/2] pycocci: add missing new line

[Julia Lawall]

On Wed, 4 May 2016, Luis R. Rodriguez wrote:

> Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>

Acked-by: Julia Lawall <julia.lawall@lip6.fr>

> ---
>  tools/pycocci | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tools/pycocci b/tools/pycocci
> index 266957bd90cd..43bb9a4dd412 100755
> --- a/tools/pycocci
> +++ b/tools/pycocci
> @@ -608,7 +608,7 @@ def _main():
>          cmd = [ '--stat', patch_branch_name + ".." + smpl_branch_name ]
>          diff_stat = git_diff(tree=args.target_dir, extra_args = cmd)
>          if len(diff_stat) == 0:
> -            logwrite('\nSmPL patch fully replaces patch series!')
> +            logwrite('\nSmPL patch fully replaces patch series!\n')
>          else:
>              logwrite('\nDifferences found:\n\n')
>              logwrite('Change directory to %s and run:\n\n\tgit diff %s..%s\n\n' % (args.target_dir, patch_branch_name, smpl_branch_name))
> -- 
> 2.7.2
> 
> _______________________________________________
> Cocci mailing list
> Cocci at systeme.lip6.fr
> https://systeme.lip6.fr/mailman/listinfo/cocci
> 

[Cocci] [RFC 0/2] coccinelle: extend test coverage

[SF Markus Elfring]

> I wanted to extend tests coverage for coccinelle but noticed
> that tests are limited to single c file, these are attached
> to respective .res files, but this also means that you have
> a direct relationship and requirement between tests from
> a single C file and a respective cocci file.

Do you imagine to measure test coverage with any advanced techniques?


> To go wild we want to be able to support really complex
> types of examples.

Is there any kind of target conflict between nice software
demonstrations and the desire for better test procedures?


> We can easily take advantage of this to extend Coccinelle test cases
> way beyond a single C file, but to more complex real life
> style projects.

How much would you like to improve the automatic checking
of generated patches?

Regards,
Markus