[Qemu-devel] [RFC PATCH 0/4] travis: run all coccinelle scripts

[Qemu-devel] [RFC PATCH 0/4] travis: run all coccinelle scripts

[Philippe Mathieu-Daudé]

Another item from my 'automated testing' list: use travis-ci to run coccinelle
scripts. This series is more of a PoC. The idea would be to run it once a day
only on /master.

Patch 1 is here only to speedup travis testing.

Patch 3 add a script which run each cocci script sequencially. If the script
modified any file from the repo, changes are commited. If the script generated
some output (i.e. using python), this output is logged in an empty commit.
Spatch is run thru a debian-based Docker image.

Patch 4 is the travis job: it calls the previous script. To respect travis time
limit timeout, each script is limited to <10min. If any commit were
generated, they are pushed to my gh-repo:

https://github.com/philmd/qemu/compare/travis-cocci_v1...philmd:autogenerated-coccinelle-20170625-126

Build output (Ran for 23 min 7 sec):
https://travis-ci.org/philmd/qemu/builds/246848085

Any idea is welcome :)

Regards,

Phil.

Philippe Mathieu-Daudé (4):
  travis: simplify matrix
  coccinelle: ignore ASTs pre-parsed cached C files
  coccinelle: script to run them all
  travis: add job to run coccinelle scripts

 .gitignore                     |   2 +
 .travis.sshkey                 | Bin 0 -> 1296 bytes
 .travis.yml                    | 145 ++++++-----------------------------------
 scripts/check-cocci-scripts.sh |  47 +++++++++++++
 4 files changed, 69 insertions(+), 125 deletions(-)
 create mode 100644 .travis.sshkey
 create mode 100755 scripts/check-cocci-scripts.sh

-- 
2.13.1

[Qemu-devel] [RFC PATCH 1/4] XXX travis: simplify matrix

[Philippe Mathieu-Daudé]

to speedup further testing

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .travis.yml | 119 ------------------------------------------------------------
 1 file changed, 119 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 27a2d9cfb3..4761569183 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -48,11 +48,6 @@ env:
     - TEST_CMD="make check"
   matrix:
     - CONFIG=""
-    - CONFIG="--enable-debug --enable-debug-tcg --enable-trace-backends=log"
-    - CONFIG="--disable-linux-aio --disable-cap-ng --disable-attr --disable-brlapi --disable-uuid --disable-libusb"
-    - CONFIG="--enable-modules"
-    - CONFIG="--with-coroutine=ucontext"
-    - CONFIG="--with-coroutine=sigaltstack"
 git:
   # we want to do this ourselves
   submodules: false
@@ -67,39 +62,6 @@ script:
   - make -j3 && ${TEST_CMD}
 matrix:
   include:
-    # Test with CLang for compile portability
-    - env: CONFIG=""
-      compiler: clang
-    # gprof/gcov are GCC features
-    - env: CONFIG="--enable-gprof --enable-gcov --disable-pie"
-      compiler: gcc
-    # We manually include builds which we disable "make check" for
-    - env: CONFIG="--enable-debug --enable-tcg-interpreter"
-           TEST_CMD=""
-      compiler: gcc
-    - env: CONFIG="--enable-trace-backends=simple"
-           TEST_CMD=""
-      compiler: gcc
-    - env: CONFIG="--enable-trace-backends=ftrace"
-           TEST_CMD=""
-      compiler: gcc
-    - env: CONFIG="--enable-trace-backends=ust"
-           TEST_CMD=""
-      compiler: gcc
-    - env: CONFIG=""
-      os: osx
-      compiler: clang
-    # Plain Trusty System Build
-    - env: CONFIG="--disable-linux-user"
-      sudo: required
-      addons:
-      dist: trusty
-      compiler: gcc
-      before_install:
-        - sudo apt-get update -qq
-        - sudo apt-get build-dep -qq qemu
-        - wget -O - http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz | tar -xvJ
-        - git submodule update --init --recursive
     # Plain Trusty Linux User Build
     - env: CONFIG="--disable-system"
       sudo: required
@@ -111,84 +73,3 @@ matrix:
         - sudo apt-get build-dep -qq qemu
         - wget -O - http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz | tar -xvJ
         - git submodule update --init --recursive
-    # Trusty System build with latest stable clang
-    - sudo: required
-      addons:
-      dist: trusty
-      language: generic
-      compiler: none
-      env:
-        - COMPILER_NAME=clang CXX=clang++-3.9 CC=clang-3.9
-        - CONFIG="--disable-linux-user --cc=clang-3.9 --cxx=clang++-3.9"
-      before_install:
-        - wget -nv -O - http://llvm.org/apt/llvm-snapshot.gpg.key | sudo apt-key add -
-        - sudo apt-add-repository -y 'deb http://llvm.org/apt/trusty llvm-toolchain-trusty-3.9 main'
-        - sudo apt-get update -qq
-        - sudo apt-get install -qq -y clang-3.9
-        - sudo apt-get build-dep -qq qemu
-        - wget -O - http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz | tar -xvJ
-        - git submodule update --init --recursive
-      before_script:
-        - ./configure ${CONFIG} || cat config.log
-    # Trusty Linux User build with latest stable clang
-    - sudo: required
-      addons:
-      dist: trusty
-      language: generic
-      compiler: none
-      env:
-        - COMPILER_NAME=clang CXX=clang++-3.9 CC=clang-3.9
-        - CONFIG="--disable-system --cc=clang-3.9 --cxx=clang++-3.9"
-      before_install:
-        - wget -nv -O - http://llvm.org/apt/llvm-snapshot.gpg.key | sudo apt-key add -
-        - sudo apt-add-repository -y 'deb http://llvm.org/apt/trusty llvm-toolchain-trusty-3.9 main'
-        - sudo apt-get update -qq
-        - sudo apt-get install -qq -y clang-3.9
-        - sudo apt-get build-dep -qq qemu
-        - wget -O - http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz | tar -xvJ
-        - git submodule update --init --recursive
-      before_script:
-        - ./configure ${CONFIG} || cat config.log
-    # Using newer GCC with sanitizers
-    - addons:
-        apt:
-          sources:
-            # PPAs for newer toolchains
-            - ubuntu-toolchain-r-test
-          packages:
-            # Extra toolchains
-            - gcc-5
-            - g++-5
-            # Build dependencies
-            - libaio-dev
-            - libattr1-dev
-            - libbrlapi-dev
-            - libcap-ng-dev
-            - libgnutls-dev
-            - libgtk-3-dev
-            - libiscsi-dev
-            - liblttng-ust-dev
-            - libnfs-dev
-            - libncurses5-dev
-            - libnss3-dev
-            - libpixman-1-dev
-            - libpng12-dev
-            - librados-dev
-            - libsdl1.2-dev
-            - libseccomp-dev
-            - libspice-protocol-dev
-            - libspice-server-dev
-            - libssh2-1-dev
-            - liburcu-dev
-            - libusb-1.0-0-dev
-            - libvte-2.90-dev
-            - sparse
-            - uuid-dev
-      language: generic
-      compiler: none
-      env:
-        - COMPILER_NAME=gcc CXX=g++-5 CC=gcc-5
-        - CONFIG="--cc=gcc-5 --cxx=g++-5 --disable-pie --disable-linux-user"
-        - TEST_CMD=""
-      before_script:
-        - ./configure ${CONFIG} --extra-cflags="-g3 -O0 -fsanitize=thread -fuse-ld=gold" || cat config.log
-- 
2.13.1

[Qemu-devel] [RFC PATCH 2/4] coccinelle: ignore ASTs pre-parsed cached C files

[Philippe Mathieu-Daudé]

files generated using coccinelle tool: 'spatch --use-cache'

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .gitignore | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.gitignore b/.gitignore
index 09c2363acf..cf65316863 100644
--- a/.gitignore
+++ b/.gitignore
@@ -116,6 +116,8 @@ tags
 TAGS
 docker-src.*
 *~
+*.ast_raw
+*.depend_raw
 trace.h
 trace.c
 trace-ust.h
-- 
2.13.1

[Qemu-devel] [RFC PATCH 3/4] coccinelle: script to run them all

[Philippe Mathieu-Daudé]

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 scripts/check-cocci-scripts.sh | 47 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 47 insertions(+)
 create mode 100755 scripts/check-cocci-scripts.sh

diff --git a/scripts/check-cocci-scripts.sh b/scripts/check-cocci-scripts.sh
new file mode 100755
index 0000000000..ec3a8b39ef
--- /dev/null
+++ b/scripts/check-cocci-scripts.sh
@@ -0,0 +1,47 @@
+#! /usr/bin/env bash
+
+test -d scripts/coccinelle || exit 1
+
+SPATCH_IMAGE="philmd/coccinelle:1.0.4"
+
+GIT_AUTHOR_NAME="Coccinelle Spatch"
+GIT_COMMITTER_NAME="Coccinelle Spatch"
+
+if [ -n "$TRAVIS" ]; then
+	# avoid stalling builds: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
+	TIMEOUT_S=530
+	TIMEOUT_CMD="timeout -k 550 500"
+	EXTRA_ARGS="--timeout ${TIMEOUT_S}"
+else
+	TIMEOUT_S=0
+fi
+
+HEAD=19 #`echo -n scripts/coccinelle/ | wc -c`
+TAIL=6 #`echo -n .cocci | wc -c`
+
+test -z "$(${SUDO} docker images -q ${SPATCH_IMAGE})" && ${SUDO} docker pull ${SPATCH_IMAGE}
+
+LOG=/tmp/cocci-spatch-$$
+for script in scripts/coccinelle/*.cocci; do
+	desc=${script:$HEAD:-$TAIL}
+	echo -e "\nRunning ${script}...\n"
+	echo -e "coccinelle: committing changes after running \"$desc\" script\n" > ${LOG}.topic
+	${TIMEOUT_CMD} ${SUDO} \
+	docker run --rm -v `pwd`:`pwd` -w `pwd` -u `id -u` \
+		${SPATCH_IMAGE} --use-cache --use-gitgrep --keep-comments \
+			--very-quiet ${EXTRA_ARGS} \
+			--sp-file ${script} \
+			--macro-file scripts/cocci-macro-file.h \
+			--dir . \
+			--in-place | tee ${LOG}.content
+	git add -u
+	git diff --cached --exit-code -s
+	if [ $? -ne 0 ]; then
+		:> ${LOG}.content
+	else
+		test -s ${LOG}.content || continue
+	fi
+	cat ${LOG}.{topic,content} | git commit --allow-empty -F -
+done
+
+rm -f ${LOG}.{topic,content}
-- 
2.13.1

[Qemu-devel] [RFC PATCH 4/4] travis: add job to run coccinelle scripts

[Philippe Mathieu-Daudé]

result pushed to my gh repo via ssh deploy key

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 .travis.sshkey | Bin 0 -> 1296 bytes
 .travis.yml    |  26 ++++++++++++++++++++------
 2 files changed, 20 insertions(+), 6 deletions(-)
 create mode 100644 .travis.sshkey

diff --git a/.travis.sshkey b/.travis.sshkey
new file mode 100644
index 0000000000000000000000000000000000000000..5921e953c8d91461ae193c15922d658834f8d6c5
GIT binary patch
literal 1296
zcmV+r1@HPCoP~Uo`}Gc5AacL0i>ciVZGDC=x67D__o=U=ig02!0StKv2oE-TL@8$>
zU0$F^xr)f=@8?MPI@}e&`i8?|`SXQ<@(;6hZnSBYT!2m}p#!07TN{#iIdPyKQH>(O
z66UEPdqEDb%~tpi+Wxb^jo$Z$8V9xkj&Zf#;-Em>m@d<_%#}N$?$af5jn}TO^)Vlf
zv~Wb!S$UnnnwWL0duGIR)=(GR15y%p)Twy93gpSGUnb!u&lx@E3~(dg5R~~?^OObe
zl>p3F$Wk~ik$4);k?zOq%6LU@vd$}DY@%-TIBpF^@?;3!jJIsNPFFjay`~et1EgI_
zMpprT%uH4i;`<##f$cFhAh<wU5Kf0DrZ;jhjDau4Ok2*)RZJBH)bRa;n>k$2#|jPI
zwFds#72`WsQ;CX*$iOAWaBaq(yR_<(-4OyZ)W8r7tdsTF)tQm5Qm`^JN5YD<v3%dA
z39F9&$!*C)dlcP(>L*JlT~`LrPzh9I_~Tjf$99HM${l=EjEK6KQ{9GBvA~8p(*N%B
zJjX9tqMli6nykS@*a;;AVjl4dK$JliASAv_9zckAQ(;U=eB6m4R>^&EcZlFAo)OTn
zhPz5?A<|tKZVuhRnfQaTkv2h_>sWH#oVnH&eV29jt8gH>IZRo%g<pmQf9WvMp12)6
zzA&ePL%Ck<UKBYINIZ*5qkCBtNi{J7u<6j}s6K9rOC}*{+k>Y?A3HSG`Ale19DS3X
z$iXSp$>v;MD2S0N6g53BQ(RU;;@E5r{H8kkSs{|&8&eF%15L|}>u8)}p({!79kYTk
z;$4d%3*h6V@-)s;l4i{7st?~{!Z}gK+`j*^jc?cPZ#Qr|g~^Be+DW&OJr@k<JSjM>
z%MDv7<M<zh$f)GeGy}`Kwq5SPVza#L(|G@Z(XwF21&kdU60#^UvH$YVgT+|me;CMi
z^3F;XGd4zXp$Qh+(?gqjjL3mA$fXW&Bovhv+tnR5`^JMBY^bmI?bZ}y7nYvBRRL2O
zukxxyZ*U`RJk6v1{R=Hbo}EJoOFL-|66Qz`0x|&j?U`N4MRmaGQWszs$;fdKYAyju
zQ8D-=%A0(bknuH?Cu{4a-*2urJW)WgAcZV}`x?0ul&7X#DRkSomZVz7DDrt}{v+ip
zuJuF{_^15DstUbL3=~JQ_w;^A3)B^-dkpmq+1w-Zu)&IU=sgX()?49tUO=nG7*p6R
zTMhvNqAu(q=y~FbeR=HuajRX-sm$AUVa;~0e0KQuR18f-HxE~>*PmucM^qRj*rK36
zfqH&%?QJB>cjUoZwq;2=%@5{=((`nIe_+#`n{ivY53v36@`pdZdyiN3iDy^T1ltVW
zYXJh3g4aNoPZQNn7a$l}6B;&sWe8n=CCj>F>s;W!(Dd-BcWFcnhHn?XZ|&RY7MUC=
z=!4=WHQ==S-C?Ms3cH)W^?%>IHJ>ZD*0le4eX5_T4Rs&?k;0~OG5Ludm-uuHi-{vX
zWKVyW?2=Uy6RCl8gNjW+e|AfO^6EJ}=upJqIUHpXNd(-E*ZgN%A0-L4r<`IbRB?4y
z)w%*$FPwBYRmfJ7U}7|Wpd2&o+nj#Dk9Dk=w`L}&Y@Zs4DKinczsaDO&ifLs6YnIA
zAcKDL;g+m+!<gE4a)^z;MR0~o0S^@`D9OA*FH)N64kN$7>ByZV67En&W{lNMy7g?{
GW~j`fDT25F

literal 0
HcmV?d00001

diff --git a/.travis.yml b/.travis.yml
index 4761569183..52665273ba 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -51,6 +51,9 @@ env:
 git:
   # we want to do this ourselves
   submodules: false
+branches:
+  except:
+  - /^autogenerated-.*/
 before_install:
   - if [ "$TRAVIS_OS_NAME" == "osx" ]; then brew update ; fi
   - if [ "$TRAVIS_OS_NAME" == "osx" ]; then brew install libffi gettext glib pixman ; fi
@@ -62,14 +65,25 @@ script:
   - make -j3 && ${TEST_CMD}
 matrix:
   include:
-    # Plain Trusty Linux User Build
-    - env: CONFIG="--disable-system"
+    # Coccinelle using Trusty
+    - env: TEST_CMD=""
+           GIT_AUTHOR_EMAIL="coccinelle@travis-ci"
+           GIT_COMMITTER_EMAIL="f4bug@amsat.org"
       sudo: required
-      addons:
+      services:
+      - docker
       dist: trusty
+      group: deprecated-2017Q2
       compiler: gcc
       before_install:
-        - sudo apt-get update -qq
         - sudo apt-get build-dep -qq qemu
-        - wget -O - http://people.linaro.org/~alex.bennee/qemu-submodule-git-seed.tar.xz | tar -xvJ
-        - git submodule update --init --recursive
+        - git submodule update --init dtc
+      script:
+        - ORIGIN=$(git rev-parse HEAD)
+        - scripts/check-cocci-scripts.sh
+      after_success:
+        - if [ ${ORIGIN} = $(git rev-parse HEAD) ]; then exit; fi
+        - eval "$(ssh-agent -s)"
+        - openssl aes-256-cbc -K $encrypted_8b1bab3b4d04_key -iv $encrypted_8b1bab3b4d04_iv -in .travis.sshkey -d | ssh-add -
+        - BRANCH=autogenerated-coccinelle-`date +%Y%m%d`-${TRAVIS_BUILD_NUMBER}
+        - git checkout -b ${BRANCH} && git remote add deploy git@github.com:${TRAVIS_REPO_SLUG}.git && git push -f deploy ${BRANCH}
-- 
2.13.1

Re: [Qemu-devel] [RFC PATCH 0/4] travis: run all coccinelle scripts

[Philippe Mathieu-Daudé]

> Patch 4 is the travis job: it calls the previous script. To respect travis time
> limit timeout, each script is limited to <10min. If any commit were
> generated, they are pushed to my gh-repo

Initially I wanted to use git format-patch/send-email to send to the
list as a series but it seems too verbose and Travis doesn't seem to
allow to send mails. Another possibility is to send to a S3 bucket or
gh-page. The push-to-repo way involves deploy ssh key but is easier to
manage with regular git commands, I can fetch this repo and
cherry-pick/rework/amend commits.

I also plan to add Marc-André Lureau clang-tidy work:
http://lists.nongnu.org/archive/html/qemu-devel/2017-06/msg05034.html

Re: [Qemu-devel] [RFC PATCH 3/4] coccinelle: script to run them all

[Eduardo Habkost]

On Sun, Jun 25, 2017 at 04:29:49PM -0300, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  scripts/check-cocci-scripts.sh | 47 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 47 insertions(+)
>  create mode 100755 scripts/check-cocci-scripts.sh
> 
> diff --git a/scripts/check-cocci-scripts.sh b/scripts/check-cocci-scripts.sh
> new file mode 100755
> index 0000000000..ec3a8b39ef
> --- /dev/null
> +++ b/scripts/check-cocci-scripts.sh
> @@ -0,0 +1,47 @@
> +#! /usr/bin/env bash
> +
> +test -d scripts/coccinelle || exit 1
> +
> +SPATCH_IMAGE="philmd/coccinelle:1.0.4"
> +
> +GIT_AUTHOR_NAME="Coccinelle Spatch"
> +GIT_COMMITTER_NAME="Coccinelle Spatch"
> +
> +if [ -n "$TRAVIS" ]; then
> +	# avoid stalling builds: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
> +	TIMEOUT_S=530
> +	TIMEOUT_CMD="timeout -k 550 500"
> +	EXTRA_ARGS="--timeout ${TIMEOUT_S}"
> +else
> +	TIMEOUT_S=0
> +fi
> +
> +HEAD=19 #`echo -n scripts/coccinelle/ | wc -c`
> +TAIL=6 #`echo -n .cocci | wc -c`
> +
> +test -z "$(${SUDO} docker images -q ${SPATCH_IMAGE})" && ${SUDO} docker pull ${SPATCH_IMAGE}
> +

Can the docker magic be made optional (and disabled by default), so
people who already have Coccinelle installed can run this script more
quickly?


> +LOG=/tmp/cocci-spatch-$$
> +for script in scripts/coccinelle/*.cocci; do
> +	desc=${script:$HEAD:-$TAIL}
> +	echo -e "\nRunning ${script}...\n"
> +	echo -e "coccinelle: committing changes after running \"$desc\" script\n" > ${LOG}.topic
> +	${TIMEOUT_CMD} ${SUDO} \
> +	docker run --rm -v `pwd`:`pwd` -w `pwd` -u `id -u` \
> +		${SPATCH_IMAGE} --use-cache --use-gitgrep --keep-comments \
> +			--very-quiet ${EXTRA_ARGS} \
> +			--sp-file ${script} \
> +			--macro-file scripts/cocci-macro-file.h \
> +			--dir . \
> +			--in-place | tee ${LOG}.content
> +	git add -u
> +	git diff --cached --exit-code -s
> +	if [ $? -ne 0 ]; then
> +		:> ${LOG}.content
> +	else
> +		test -s ${LOG}.content || continue
> +	fi
> +	cat ${LOG}.{topic,content} | git commit --allow-empty -F -
> +done
> +
> +rm -f ${LOG}.{topic,content}
> -- 
> 2.13.1
> 

-- 
Eduardo

Re: [Qemu-devel] [RFC PATCH 3/4] coccinelle: script to run them all

[Philippe Mathieu-Daudé]

On Mon, Jun 26, 2017 at 11:17 PM, Eduardo Habkost <ehabkost@redhat.com> wrote:
> On Sun, Jun 25, 2017 at 04:29:49PM -0300, Philippe Mathieu-Daudé wrote:
[...]
>> +test -z "$(${SUDO} docker images -q ${SPATCH_IMAGE})" && ${SUDO} docker pull ${SPATCH_IMAGE}
>> +
>
> Can the docker magic be made optional (and disabled by default), so
> people who already have Coccinelle installed can run this script more
> quickly?

Ok. Actually I didn't even think about running out of docker, the list
of ocaml requisites scared me a bit.
Using a dockerized spatch allow to stick to a common version (so I
know the scripts I run locally will run exactly the same on Travis).
But it makes sens ;)
I tried the multi-threaded options and for some reason they did not
work within docker, but I did not try out of it.

Regards,

Phil.

Re: [Qemu-devel] [RFC PATCH 3/4] coccinelle: script to run them all

[Fam Zheng]

On Sun, 06/25 16:29, Philippe Mathieu-Daudé wrote:
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  scripts/check-cocci-scripts.sh | 47 ++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 47 insertions(+)
>  create mode 100755 scripts/check-cocci-scripts.sh
> 
> diff --git a/scripts/check-cocci-scripts.sh b/scripts/check-cocci-scripts.sh
> new file mode 100755
> index 0000000000..ec3a8b39ef
> --- /dev/null
> +++ b/scripts/check-cocci-scripts.sh
> @@ -0,0 +1,47 @@
> +#! /usr/bin/env bash
> +
> +test -d scripts/coccinelle || exit 1
> +
> +SPATCH_IMAGE="philmd/coccinelle:1.0.4"
> +
> +GIT_AUTHOR_NAME="Coccinelle Spatch"
> +GIT_COMMITTER_NAME="Coccinelle Spatch"
> +
> +if [ -n "$TRAVIS" ]; then
> +	# avoid stalling builds: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
> +	TIMEOUT_S=530
> +	TIMEOUT_CMD="timeout -k 550 500"
> +	EXTRA_ARGS="--timeout ${TIMEOUT_S}"
> +else
> +	TIMEOUT_S=0
> +fi
> +
> +HEAD=19 #`echo -n scripts/coccinelle/ | wc -c`
> +TAIL=6 #`echo -n .cocci | wc -c`
> +
> +test -z "$(${SUDO} docker images -q ${SPATCH_IMAGE})" && ${SUDO} docker pull ${SPATCH_IMAGE}
> +
> +LOG=/tmp/cocci-spatch-$$
> +for script in scripts/coccinelle/*.cocci; do
> +	desc=${script:$HEAD:-$TAIL}
> +	echo -e "\nRunning ${script}...\n"
> +	echo -e "coccinelle: committing changes after running \"$desc\" script\n" > ${LOG}.topic
> +	${TIMEOUT_CMD} ${SUDO} \
> +	docker run --rm -v `pwd`:`pwd` -w `pwd` -u `id -u` \
> +		${SPATCH_IMAGE} --use-cache --use-gitgrep --keep-comments \
> +			--very-quiet ${EXTRA_ARGS} \
> +			--sp-file ${script} \
> +			--macro-file scripts/cocci-macro-file.h \
> +			--dir . \
> +			--in-place | tee ${LOG}.content

Is it possible to do an "incremental" run of coccinelle to only process affected
files by a series (maybe on those files that are updated by an incremental make
invocation)? That way we can save much time.

BTW I think you should use whitespaces instead of tabs for indentation.

Fam

Re: [Qemu-devel] [RFC PATCH 0/4] travis: run all coccinelle scripts

[Markus Armbruster]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Another item from my 'automated testing' list: use travis-ci to run coccinelle
> scripts. This series is more of a PoC. The idea would be to run it once a day
> only on /master.
>
> Patch 1 is here only to speedup travis testing.
>
> Patch 3 add a script which run each cocci script sequencially. If the script
> modified any file from the repo, changes are commited. If the script generated
> some output (i.e. using python), this output is logged in an empty commit.
> Spatch is run thru a debian-based Docker image.
>
> Patch 4 is the travis job: it calls the previous script. To respect travis time
> limit timeout, each script is limited to <10min. If any commit were
> generated, they are pushed to my gh-repo:
>
> https://github.com/philmd/qemu/compare/travis-cocci_v1...philmd:autogenerated-coccinelle-20170625-126
>
> Build output (Ran for 23 min 7 sec):
> https://travis-ci.org/philmd/qemu/builds/246848085
>
> Any idea is welcome :)

Are we sure all the Coccinelle scripts always produce desirable and
correct results?  I often use Coccinelle to do the tedious 95% of the
job, followed by a bit of manual cleanup.  What if manual cleanup is
required to make things compile cleanly?  Say because the transformation
leaves unused variables behind.  What if the transformation is wanted in
19 out of 20 cases?  Say because it replaces a bulky expression by the
helper call created for this purpose (good), including in the helper
function (bad)?