[PATCH] Display change history as a diff between two dirs

[PATCH] Display change history as a diff between two dirs

[Roland Kaufmann]

Watching patches serially it can be difficult to get an overview of how
a pervasive change is distributed through-out different modules. Thus:

Extract snapshots of the files that have changed between two revisions
into temporary directories and launch a graphical tool to show the diff
between them.

Use existing functionality in git-diff to figure out which files have
changed, and to get the files themselves.

Based on a script called 'git-diffc' by Nitin Gupta.

Signed-off-by: Roland Kaufmann <rlndkfmn+git@gmail.com>
---

Requests for such a scripts surface occationally, so I believe it could
be useful to have in the distribution itself.

 Documentation/git-dirdiff.txt |   55 +++++++++++++++++++++++++++++++++++++++++
 Makefile                      |    2 +
 git-dirdiff--helper.sh        |   28 +++++++++++++++++++++
 git-dirdiff.sh                |   34 +++++++++++++++++++++++++
 4 files changed, 119 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/git-dirdiff.txt
 create mode 100755 git-dirdiff--helper.sh
 create mode 100755 git-dirdiff.sh

diff --git a/Documentation/git-dirdiff.txt b/Documentation/git-dirdiff.txt
new file mode 100644
index 0000000..bdd2581
--- /dev/null
+++ b/Documentation/git-dirdiff.txt
@@ -0,0 +1,55 @@
+git-dirdiff(1)
+==============
+
+NAME
+----
+git-dirdiff - Show changes using directory compare
+
+SYNOPSIS
+--------
+[verse]
+'git dirdiff' [<options>] [<commit> [<commit>]] [--] [<path>...]
+
+DESCRIPTION
+-----------
+'git dirdiff' is a git command that allows you to compare revisions
+as a difference between two directories. 'git dirdiff' is a frontend
+to linkgit:git-diff[1].
+
+OPTIONS
+-------
+See linkgit:git-diff[1] for the list of supported options.
+
+CONFIG VARIABLES
+----------------
+'git dirdiff' uses the same config variables as linkgit:git-difftool[1]
+to determine which difftool should be used.
+
+TEMPORARY FILES
+---------------
+'git dirdiff' creates a directory with 'mktemp' to hold snapshots of the
+files which are different in the two revisions. This directory is removed
+when the diff viewer terminates.
+
+NOTES
+-----
+The diff viewer must support being passed directories instead of files
+as its arguments.
++
+Files that are not put under version control are not included when
+viewing the difference between a revision and the working directory.
+
+SEE ALSO
+--------
+linkgit:git-diff[1]::
+	 Show changes between commits, commit and working tree, etc
+
+linkgit:git-difftool[1]::
+	Show changes using common diff tools
+
+linkgit:git-config[1]::
+	 Get and set repository or global options
+
+GIT
+---
+Part of the linkgit:git[1] suite
diff --git a/Makefile b/Makefile
index 3139c19..03771cf 100644
--- a/Makefile
+++ b/Makefile
@@ -365,6 +365,8 @@ unexport CDPATH
 SCRIPT_SH += git-am.sh
 SCRIPT_SH += git-bisect.sh
 SCRIPT_SH += git-difftool--helper.sh
+SCRIPT_SH += git-dirdiff.sh
+SCRIPT_SH += git-dirdiff--helper.sh
 SCRIPT_SH += git-filter-branch.sh
 SCRIPT_SH += git-lost-found.sh
 SCRIPT_SH += git-merge-octopus.sh
diff --git a/git-dirdiff--helper.sh b/git-dirdiff--helper.sh
new file mode 100755
index 0000000..bc0b49d
--- /dev/null
+++ b/git-dirdiff--helper.sh
@@ -0,0 +1,28 @@
+#!/bin/sh
+#
+# Accumulate files in a changeset into a pre-defined directory.
+#
+# Copyright (C) 2011 Roland Kaufmann
+# Based on a script called git-diffc by Nitin Gupta
+#
+# This file is licensed under the GPL v2, or a later version
+# at the discretion of the official Git maintainer.
+
+# bail out if there is any problems copying
+set -e
+
+# check that we are called by git-dirdiff
+if [ -z $__GIT_DIFF_DIR ]; then
+  echo Error: Do not call $(basename $0) directly 1>&2
+  exit 1
+fi
+
+# don't attempt to copy new or removed files
+if [ "$2" != "/dev/null" ]; then
+  mkdir -p $__GIT_DIFF_DIR/old/$(dirname $1)
+  cp $2 $__GIT_DIFF_DIR/old/$1
+fi
+if [ "$5" != "/dev/null" ]; then
+  mkdir -p $__GIT_DIFF_DIR/new/$(dirname $1)
+  cp $5 $__GIT_DIFF_DIR/new/$1
+fi
diff --git a/git-dirdiff.sh b/git-dirdiff.sh
new file mode 100755
index 0000000..4e75eda
--- /dev/null
+++ b/git-dirdiff.sh
@@ -0,0 +1,34 @@
+#!/bin/sh
+#
+# Display differences between two commits with a directory comparison.
+#
+# Copyright (C) 2011 Roland Kaufmann
+# Based on a script called git-diffc by Nitin Gupta
+#
+# This file is licensed under the GPL v2, or a later version
+# at the discretion of the official Git maintainer.
+
+# bail out if there is any problems in getting a diff
+set -e
+
+# create a temporary directory to hold snapshots of changed files
+__GIT_DIFF_DIR=$(mktemp --tmpdir -d git-dirdiff.XXXXXX)
+export __GIT_DIFF_DIR
+
+# cleanup after we're done
+trap 'rm -rf $__GIT_DIFF_DIR' 0
+
+# list all files that have changed. store this list in a separate
+# file so that we can test the exit status of this command. (if we had
+# bash we could use pipefail, or if we had Posix we could use mkfifo)
+git diff --raw "$@" > $__GIT_DIFF_DIR/toc
+
+# let the helper script accumulate them into the temporary directory
+cut -f 2- -s $__GIT_DIFF_DIR/toc | while read f; do
+  GIT_EXTERNAL_DIFF=git-dirdiff--helper git --no-pager diff "$@" $f
+done
+
+# run original diff program, reckoning it will understand directories
+# modes and shas does not apply to the root directories so submit dummy
+# values for those, hoping that the diff tool does not use them.
+git-difftool--helper - $__GIT_DIFF_DIR/old deadbeef 0755 $__GIT_DIFF_DIR/new babeface 0755
-- 
1.7.1

Re: [PATCH] Display change history as a diff between two dirs

[Junio C Hamano]

Roland Kaufmann <rlndkfmn+git@gmail.com> writes:

> diff --git a/git-dirdiff--helper.sh b/git-dirdiff--helper.sh
> new file mode 100755
> index 0000000..bc0b49d
> --- /dev/null
> +++ b/git-dirdiff--helper.sh
> @@ -0,0 +1,28 @@
> ...
> +# bail out if there is any problems copying
> +set -e

I do not think any of our scripted Porcelains use "set -e"; rather they
try to catch errors and produce messages that are more appropriate for
specific error sites.

> +# check that we are called by git-dirdiff
> +if [ -z $__GIT_DIFF_DIR ]; then
> +  echo Error: Do not call $(basename $0) directly 1>&2
> +  exit 1
> +fi

(style)

	if test -z "$__GIT_DIFF_DIR"
        then
        	echo >&2 "..."
		exit 1
	fi

If this were to become part of the main Porcelain set, you would probably
source in the git-sh-setup helper near the beginning with

	. git-sh-setup

and use "die" instead of "echo && exit 1".

> diff --git a/git-dirdiff.sh b/git-dirdiff.sh
> new file mode 100755
> index 0000000..4e75eda
> --- /dev/null
> +++ b/git-dirdiff.sh
> @@ -0,0 +1,34 @@
> ...
> +# create a temporary directory to hold snapshots of changed files
> +__GIT_DIFF_DIR=$(mktemp --tmpdir -d git-dirdiff.XXXXXX)
> +export __GIT_DIFF_DIR

I do not think any of our scripted Porcelains use "mktemp" especially
"mktemp -d". How portable is it?

> +git diff --raw "$@" > $__GIT_DIFF_DIR/toc
> +
> +# let the helper script accumulate them into the temporary directory
> +cut -f 2- -s $__GIT_DIFF_DIR/toc | while read f; do
> +  GIT_EXTERNAL_DIFF=git-dirdiff--helper git --no-pager diff "$@" $f
> +done

(style)

	... upstream command ... |
        while read f
	do
		...
	done

It also is not clear what could be used in "$@". Obviously you would not
want things like "-U20" and "--stat" fed to the first "list of paths"
phase, but there may be some options you may want to give to the inner
"external diff" thing.

It is not clear to me why you need to grab the list of paths in toc and
iterate over them one by one. IOW, why isn't this sufficient?

    # dirdiff--helper will copy the files in the temporary directory
    GIT_EXTERNAL_DIFF=git-dirdiff--helper git --no-pager diff "$@"

    # Now the temporary old/ and new/ are populated, compare them
    git-difftool--helper - .....

Overall, I found it interesting, but I am not convinced yet that this
should be in the set of main Porcelains.  It seems to be a hack to work
around the design of the current external diff interface that specifically
targets tools that are about comparing single pair of paths at a time.

If "compare two sets of files, each extracted in its own temporary
directory" turns out to be sufficiently useful thing to do (which I
suspect is true), we would probably want to make it an option to "git
diff" itself, and not a separate git subcommand like "dirdiff". I can see
"git diff" (and possibly even things like "git log -p") populating two
temporary directories (your old/ and new/) and running a custom program
specified by GIT_EXTERNAL_TREEDIFF environment variable, instead of doing
any textual diff generation internally.

I wouldn't mind carrying a polished version of this in contrib/ for a
cycle or two in order to let people try it out and get kinks out of its
design.

For example, how well does this approach work with -M/-C (I do not think
it would do anything useful, but I haven't thought things through).  It
would be nice if we gave the hint to the external program that compares
the populated temporary directories how paths in the preimage tree and
postimage tree correspond to each other.

Re: [PATCH] Display change history as a diff between two dirs

[Roland Kaufmann]

On 2011-10-30 07:09, Junio C Hamano wrote:
> I do not think any of our scripted Porcelains use "set -e"; rather
> they try to catch errors and produce messages that are more
> appropriate for specific error sites.

git-dirdiff being a wrapper script, I reasoned that the underlaying
command would already have printed a sufficient error message, so what
was left was just to exit. But you're right in that some of the commands
will not give sufficient context. I'll put in some more detailed error
handling.

> I do not think any of our scripted Porcelains use "mktemp"
> especially "mktemp -d". How portable is it?

Even if it is not part of Posix, I reckon since it is a part of the
coreutils, it is available in most GNU userlands, inclusive GnuWin32.
I don't have any live BSD systems available, but based on the man pages
it seems to be available there too, albeit with some different options.

--tmpdir seems to be more of a problem in than respect than -d. I'll see
if I can find a set of options that are digestable to most platforms.

I think it is unfortunate to use the current directory as scratch space;
it can be a network disk, or even read-only. mktemp can in contrast make
a directory in a place which is not spilled to disk.

> It is not clear to me why you need to grab the list of paths in toc
> and iterate over them one by one. IOW, why isn't this sufficient?

This design decision was probably appropriate in an earlier iteration,
but as you point out, it is indeed superfluous now! I apologize for not
realizing that myself.

> suspect is true), we would probably want to make it an option to
> "git diff" itself, and not a separate git subcommand like "dirdiff".
> I can see

Although being an able *user* of Git, I am not (yet) familiar enough
with the codebase to have a patch ready for `git diff` itself. It would
certainly require more rigorous testing.

> "git diff" (and possibly even things like "git log -p") populating
> two temporary directories (your old/ and new/) and running a custom
> program specified by GIT_EXTERNAL_TREEDIFF environment variable,
> instead of doing

Would it be better to have yet another configuration available for this 
option instead of reusing the existing infrastructure for `git difftool`?

> It also is not clear what could be used in "$@". Obviously you would
>  not want things like "-U20" and "--stat" fed to the first "list of
> paths" phase, but there may be some options you may want to give to
> the inner "external diff" thing.

Ideally, it should work the same way as `git difftool`.

> For example, how well does this approach work with -M/-C (I do not
> think it would do anything useful, but I haven't thought things
> through).  It would be nice if we gave the hint to the external

As of now, files that are moved will turn up a different places in the
tree without any annotations, and off the top of my head I don't see any
obvious way to propagate such hints. If the diff tool is sophisticated
can probably use the same heuristics as Git currently does, but I am
unaware of any that is able to do that yet.

> I wouldn't mind carrying a polished version of this in contrib/ for
> a cycle or two in order to let people try it out and get kinks out of
>  its design.

I would appreciate that! I'll submit a reworked proposal taking you
comments into account. It may take a few days due to unrelated
engagements, though.

-- 
   Roland.

Re: [PATCH] Display change history as a diff between two dirs

[Junio C Hamano]

Roland Kaufmann <rlndkfmn+git@gmail.com> writes:

>> "git diff" (and possibly even things like "git log -p") populating
>> two temporary directories (your old/ and new/) and running a custom
>> program specified by GIT_EXTERNAL_TREEDIFF environment variable,
>> instead of doing
>
> Would it be better to have yet another configuration available for
> this option instead of reusing the existing infrastructure for `git
> difftool`?

GIT_EXTERNAL_DIFF is designed to drive tools that can take a single pair
of files and compare, so it is entirely possible for people to have such
tool that would _not_ grok (as a tool named by the variable does not have
to) two directories specified by it. So yes, it is essential that the
variable not be reused.

It probably is OK for "git diff --dirdiff" to use GIT_EXTERNAL_TREEDIFF if
and only if GIT_EXTERNAL_DIFF is not defined, and use GIT_EXTERNAL_DIFF
otherwise. People who have GIT_EXTERNAL_DIFF set to a tool capable of
handing directory pair can just add "--dirdiff" to the command line, and
others can find such a tool and set it to GIT_EXTERNAL_TREEDIFF when they
do so.

>> It also is not clear what could be used in "$@". Obviously you would
>>  not want things like "-U20" and "--stat" fed to the first "list of
>> paths" phase, but there may be some options you may want to give to
>> the inner "external diff" thing.
>
> Ideally, it should work the same way as `git difftool`.

I am not so sure about that; difftool is another way to drive a tool one
filepair at a time. You want to drive a tool that takes them as two _sets_
of files, one in each directory.

Re: [PATCH] Display change history as a diff between two dirs

[Roland Kaufmann]

On Tue, Nov 1, 2011 at 04:49, Junio C Hamano <gitster@pobox.com> wrote:
> Roland Kaufmann <rlndkfmn+git@gmail.com> writes:
>> Would it be better to have yet another configuration available for
>> this option instead of reusing the existing infrastructure for `git
>> difftool`?

> It probably is OK for "git diff --dirdiff" to use GIT_EXTERNAL_TREEDIFF if
> and only if GIT_EXTERNAL_DIFF is not defined, and use GIT_EXTERNAL_DIFF
> otherwise. People who have GIT_EXTERNAL_DIFF set to a tool capable of
> handing directory pair can just add "--dirdiff" to the command line, and

Did you perhaps mean the other way around: GIT_EXTERNAL_TREEDIFF
if set, and GIT_EXTERNAL_DIFF otherwise?

I will revise the patch to incorporate that, and at the same time fix
the isempty
function (it can be done simpler).

>>> It also is not clear what could be used in "$@". Obviously you would
>>>  not want things like "-U20" and "--stat" fed to the first "list of
>>> paths" phase, but there may be some options you may want to give to
>>> the inner "external diff" thing.

>> Ideally, it should work the same way as `git difftool`.

> I am not so sure about that; difftool is another way to drive a tool one
> filepair at a time. You want to drive a tool that takes them as two _sets_
> of files, one in each directory.

I was thinking about the behaviour when you send options that are not
applicable to files at all, such as "--stat". In that case, it just stops after
`git diff`. Options that only applies to one file and not to a set
have I admittedly
not thought about (yet).

-- 
   Roland.

Re: [PATCH] Display change history as a diff between two dirs

[Junio C Hamano]

Roland Kaufmann <rlndkfmn+git@gmail.com> writes:

> On Tue, Nov 1, 2011 at 04:49, Junio C Hamano <gitster@pobox.com> wrote:
>> Roland Kaufmann <rlndkfmn+git@gmail.com> writes:
>>> Would it be better to have yet another configuration available for
>>> this option instead of reusing the existing infrastructure for `git
>>> difftool`?
>
>> It probably is OK for "git diff --dirdiff" to use GIT_EXTERNAL_TREEDIFF if
>> and only if GIT_EXTERNAL_DIFF is not defined, and use GIT_EXTERNAL_DIFF
>> otherwise. People who have GIT_EXTERNAL_DIFF set to a tool capable of
>> handing directory pair can just add "--dirdiff" to the command line, and
>
> Did you perhaps mean the other way around: GIT_EXTERNAL_TREEDIFF
> if set, and GIT_EXTERNAL_DIFF otherwise?

I think I misstated the precedence order.  What I had in mind was roughly
like this:

 1. "git diff" (and others like "git log") without "--dirdiff" option:
    GIT_EXTERNAL_DIFF is used just as before and no GIT_EXTERNAL_TREEDIFF
    is consulted;

 2. "git diff --dirdiff": GIT_EXTERNAL_TREEDIFF is used if set, otherwise
    GIT_EXTERNAL_DIFF is used.

That way, people who have been using GIT_EXTERNAL_DIFF that is capable of
comparing two directories do not have to change anything and can just give
"--dirdiff" when they want to operate the command with this new mode of
comparison. People whose GIT_EXTERNAL_DIFF is not capable of comparing two
directories have two choices, but in either case they first would need to
find an external tool they want to use with the "--dirdiff" mode. If they
want to use the same new external tool for non "--dirdiff" mode, then they
can set GIT_EXTERNAL_DIFF to it. If they want to keep using the command
they have been using with GIT_EXTERNAL_DIFF when not in "--dirdiff" mode,
however, they can keep their GIT_EXTERNAL_DIFF to the old "two files
comparison" tool, and set GIT_EXTERNAL_TREEDIFF to the new one.

But that is all about envisioning how a proper integration of the feature
in the main "diff/log -p" command codepath would work. I do not know if it
is worth to add that to your script, which would be a stop-gap experiment
until we find out if the "populate two temporary directories and have a
tool compare them as a whole" mode is useful.