[RFC/PATCH] Bisect: implement "git bisect run <cmd>..." to automatically bisect.

[RFC/PATCH] Bisect: implement "git bisect run <cmd>..." to automatically bisect.

[Christian Couder]

This idea was suggested by Bill Lear
(Message-ID: <17920.38942.364466.642979@lisa.zopyra.com>)
and I think it is a very good one.

This patch adds a new test file for "git bisect run", but there
is currently only one basic test.

Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---
 git-bisect.sh         |   54 +++++++++++++++++++++++++++++++++++++++++++--
 t/t6030-bisect-run.sh |   57 +++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 108 insertions(+), 3 deletions(-)
 create mode 100755 t/t6030-bisect-run.sh

diff --git a/git-bisect.sh b/git-bisect.sh
index b1c3a6b..53dff7d 100755
--- a/git-bisect.sh
+++ b/git-bisect.sh
@@ -1,14 +1,15 @@
 #!/bin/sh
 
-USAGE='[start|bad|good|next|reset|visualize|replay|log]'
+USAGE='[start|bad|good|next|reset|visualize|replay|log|run]'
 LONG_USAGE='git bisect start [<pathspec>]	reset bisect state and start bisection.
 git bisect bad [<rev>]		mark <rev> a known-bad revision.
 git bisect good [<rev>...]	mark <rev>... known-good revisions.
 git bisect next			find next bisection to test and check it out.
 git bisect reset [<branch>]	finish bisection search and go back to branch.
 git bisect visualize            show bisect status in gitk.
-git bisect replay <logfile>	replay bisection log
-git bisect log			show bisect log.'
+git bisect replay <logfile>	replay bisection log.
+git bisect log			show bisect log.
+git bisect run <cmd>... 	use <cmd>... to automatically bisect.'
 
 . git-sh-setup
 require_work_tree
@@ -185,6 +186,7 @@ bisect_reset() {
 		rm -f "$GIT_DIR/refs/heads/bisect" "$GIT_DIR/head-name"
 		rm -f "$GIT_DIR/BISECT_LOG"
 		rm -f "$GIT_DIR/BISECT_NAMES"
+		rm -f "$GIT_DIR/BISECT_RUN"
 	fi
 }
 
@@ -220,6 +222,50 @@ bisect_replay () {
 	bisect_auto_next
 }
 
+bisect_run () {
+    while true
+    do
+      echo "running $@"
+      "$@"
+      res=$?
+
+      # Check for really bad run error.
+      if [ $res -lt 0 -o $res -ge 128 ]; then
+	  echo >&2 "bisect run failed:"
+	  echo >&2 "exit code $res from '$@' is < 0 or >= 128"
+	  exit $res
+      fi
+
+      # Use "git-bisect good" or "git-bisect bad"
+      # depending on run success or failure.
+      # We cannot use bisect_good or bisect_bad functions
+      # because they can exit.
+      if [ $res -gt 0 ]; then
+	  next_bisect='git-bisect bad'
+      else
+	  next_bisect='git-bisect good'
+      fi
+
+      $next_bisect > "$GIT_DIR/BISECT_RUN"
+      res=$?
+
+      cat "$GIT_DIR/BISECT_RUN"
+
+      if [ $res -ne 0 ]; then
+	  echo >&2 "bisect run failed:"
+	  echo >&2 "$next_bisect exited with error code $res"
+	  exit $res
+      fi
+
+      if grep "is first bad commit" "$GIT_DIR/BISECT_RUN" > /dev/null; then
+	  echo "bisect run success"
+	  exit 0;
+      fi
+
+    done
+}
+
+
 case "$#" in
 0)
     usage ;;
@@ -244,6 +290,8 @@ case "$#" in
 	bisect_replay "$@" ;;
     log)
 	cat "$GIT_DIR/BISECT_LOG" ;;
+    run)
+        bisect_run "$@" ;;
     *)
         usage ;;
     esac
diff --git a/t/t6030-bisect-run.sh b/t/t6030-bisect-run.sh
new file mode 100755
index 0000000..39c7228
--- /dev/null
+++ b/t/t6030-bisect-run.sh
@@ -0,0 +1,57 @@
+#!/bin/sh
+#
+# Copyright (c) 2007 Christian Couder
+#
+test_description='Tests git-bisect run functionality'
+
+. ./test-lib.sh
+
+add_line_into_file()
+{
+    _line=$1
+    _file=$2
+
+    if [ -f "$_file" ]; then
+        echo "$_line" >> $_file || return $?
+        MSG="Add <$_line> into <$_file>."
+    else
+        echo "$_line" > $_file || return $?
+        git add $_file || return $?
+        MSG="Create file <$_file> with <$_line> inside."
+    fi
+
+    git-commit -m "$MSG" $_file
+}
+
+HASH1=
+HASH3=
+HASH4=
+
+test_expect_success \
+    'set up basic repo with 1 file (hello) and 4 commits' \
+    'add_line_into_file "1: Hello World" hello &&
+     add_line_into_file "2: A new day for git" hello &&
+     add_line_into_file "3: Another new day for git" hello &&
+     add_line_into_file "4: Ciao for now" hello &&
+     HASH1=$(git rev-list HEAD | tail -1) &&
+     HASH3=$(git rev-list HEAD | head -2 | tail -1) &&
+     HASH4=$(git rev-list HEAD | head -1)'
+
+# We want to automatically find the commit that
+# introduced "Another" into hello.
+test_expect_success \
+    'git bisect run simple case' \
+    'echo "#!/bin/sh" > test_script.sh &&
+     echo "grep Another hello > /dev/null" >> test_script.sh &&
+     echo "test \$? -ne 0" >> test_script.sh &&
+     chmod +x test_script.sh &&
+     git bisect start &&
+     git bisect good $HASH1 &&
+     git bisect bad $HASH4 &&
+     git bisect run ./test_script.sh > my_bisect_log.txt &&
+     grep "$HASH3 is first bad commit" my_bisect_log.txt'
+
+#
+#
+test_done
+
-- 
1.5.1.rc1.13.g0872-dirty

Re: [RFC/PATCH] Bisect: implement "git bisect run <cmd>..." to automatically bisect.

[Junio C Hamano]

Christian Couder <chriscool@tuxfamily.org> writes:

> This idea was suggested by Bill Lear
> (Message-ID: <17920.38942.364466.642979@lisa.zopyra.com>)
> and I think it is a very good one.

Nicely done.

People often find that during bisect they want to have a
near-constant tweaks (e.g., s/#define DEBUG 0/#define DEBUG 1/
in a header file, or "revision that does not have this commit
needs this patch applied to work around other problem this
bisection is not interested in") applied to the revision being
tested.  

To cope with such a situation, after the inner git-bisect finds
the next revision to test, with the "run" script, the user can
apply that tweak before compiling, run the real test, and after
the test decides if the revision (possibly with the needed
tweaks) passed the test, rewind the tree to the pristine state.
Finally the "run" script can exit with the status of the real
test to let "git-bisect run" command loop to know the outcome.

When you write a documentation for this patch, the above issue
should be addressed, as this is often asked on and off the list.

> @@ -220,6 +222,50 @@ bisect_replay () {
>  	bisect_auto_next
>  }
>  
> +bisect_run () {
> +    while true
> +    do
> +      echo "running $@"
> +      "$@"
> +      res=$?
> +
> +      # Check for really bad run error.
> +      if [ $res -lt 0 -o $res -ge 128 ]; then
> +	  echo >&2 "bisect run failed:"
> +	  echo >&2 "exit code $res from '$@' is < 0 or >= 128"
> +	  exit $res
> +      fi

I am not sure if this flexibility/leniency is desirable.  It
certainly allows a sloppily written shell script that exits with
any random small-positive values to report a badness, which may
be handy, but allowing sloppiness might lead to wasted time
after all.  I dunno.  A more strict convention that says the run
script should exit 1 to signal "bad, please continue", 0 for
"good, please continue" and other values for "no good, abort"
might be less error prone.

In any case, the exit status convention needs documentation.

A program that does "exit(-1)" leaves $? = 255 (see exit(3)
manual page, the value is chopped with "& 0377"), so the test
for -ge 128 is certainly good; I do not know if any system gives
negative values, but the check shouldn't hurt.  As a style, I
would have written that as:

	if test $res -lt 0 || test $res -ge 128
        then
        	...

but that is probably a bit old-fashioned.  

> +      # Use "git-bisect good" or "git-bisect bad"
> +      # depending on run success or failure.
> +      # We cannot use bisect_good or bisect_bad functions
> +      # because they can exit.
> +      if [ $res -gt 0 ]; then
> +	  next_bisect='git-bisect bad'
> +      else
> +	  next_bisect='git-bisect good'
> +      fi
> +
> +      $next_bisect > "$GIT_DIR/BISECT_RUN"


If their exiting, and possibly variable assignments, are the
only problem, you can run that in subshell, can't you?  Like:

	( $next_bisect >"$GIT_DIR/BISECT_RUN" )

> +      res=$?

Re: [RFC/PATCH] Bisect: implement "git bisect run <cmd>..." to automatically bisect.

[Christian Couder]

Junio C Hamano a écrit :

[...]

> > +      # Check for really bad run error.
> > +      if [ $res -lt 0 -o $res -ge 128 ]; then
> > +	  echo >&2 "bisect run failed:"
> > +	  echo >&2 "exit code $res from '$@' is < 0 or >= 128"
> > +	  exit $res
> > +      fi
>
> I am not sure if this flexibility/leniency is desirable.  It
> certainly allows a sloppily written shell script that exits with
> any random small-positive values to report a badness, which may
> be handy, but allowing sloppiness might lead to wasted time
> after all.  I dunno.  A more strict convention that says the run
> script should exit 1 to signal "bad, please continue", 0 for
> "good, please continue" and other values for "no good, abort"
> might be less error prone.

Perhaps, but when running "git bisect run make" to automatically find the 
commit that broke the build, it would fail because make will usually return 
2 in case of error. 

It seems that there are few standards for exit code, so whatever convention 
we choose will not work in all cases.

> In any case, the exit status convention needs documentation.

Yes, I will work on it.

[...]

> > +      # Use "git-bisect good" or "git-bisect bad"
> > +      # depending on run success or failure.
> > +      # We cannot use bisect_good or bisect_bad functions
> > +      # because they can exit.
> > +      if [ $res -gt 0 ]; then
> > +	  next_bisect='git-bisect bad'
> > +      else
> > +	  next_bisect='git-bisect good'
> > +      fi
> > +
> > +      $next_bisect > "$GIT_DIR/BISECT_RUN"
>
> If their exiting, and possibly variable assignments, are the
> only problem, you can run that in subshell, can't you?  Like:
>
> 	( $next_bisect >"$GIT_DIR/BISECT_RUN" )

You are right. I will submit an updated patch with this change and some 
documentation.

Thanks,
Christian.