git diff across submodules

git diff across submodules

[Kenny Simpson]

Hello,
  Is there any nice way to get a diff and/or diffstat of both a project and its submodules between two revisions of the main project?

Something like 'git diff --stat tag_a tag_b' but also including the diffstat on the submodule from the revision tied to in tag_a to the revision tied to tag_b.  A few shell backflips will do it, but this seems like I'm missing something.

thanks,
-Kenny

Re: git diff across submodules

[Jens Lehmann]

Am 18.09.2012 05:12, schrieb Kenny Simpson:
>   Is there any nice way to get a diff and/or diffstat of both a project and its submodules between two revisions of the main project?
> 
> Something like 'git diff --stat tag_a tag_b' but also including the diffstat on the submodule from the revision tied to in tag_a to the revision tied to tag_b.  A few shell backflips will do it, but this seems like I'm missing something.

Unfortunately that isn't possible yet, "git diff" still has to learn the
--recurse-submodules option. And as far as I know, nobody is currently
working on that.

Re: git diff across submodules

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> Am 18.09.2012 05:12, schrieb Kenny Simpson:
>>   Is there any nice way to get a diff and/or diffstat of both a project and its submodules between two revisions of the main project?
>> 
>> Something like 'git diff --stat tag_a tag_b' but also including the diffstat on the submodule from the revision tied to in tag_a to the revision tied to tag_b.  A few shell backflips will do it, but this seems like I'm missing something.
>
> Unfortunately that isn't possible yet, "git diff" still has to learn the
> --recurse-submodules option. And as far as I know, nobody is currently
> working on that.

I do not think it is _that_ unfortunate, at least for two reasons.

When I made something a submodule, not a plain vanilla directory, I
did want it to be treated differently from a collection of files in
random states (which is what a directory is), but as a logical unit.
If I bind another project 'frotz' at my path 'lib/frotz' and ask for
diff of two versions of my superproject, e.g.

    $ git diff v1.0 v1.1

I am more interested in finding out that we used to use v2.5 of
'frotz' back when we were at v1.0 but in our v1.1 we downgraded it
to v2.4, perhaps because we found a regression in the 'frotz'
library, rather than the whole reverse differences between v2.4 and
v2.5 of the frotz project.  That difference, when I want to, I can
get by going to that submodule and grab it myself with

    $ cd lib/frotz &7 git diff v2.4 v2.5

I also suspect that you do not have to change "git diff" at all to
show the patch recursively by using the attribute mechanism (look in
Documentation/gitattributes.text for a string GIT_EXTERNAL_DIFF).
It might be just as simple as doing this:

	echo >.gitattributes "/lib/frotz diff=subrecurse" 
	git config diff.subrecurse.command $HOME/bin/diff-subrecurse
        cat >$HOME/bin/diff-subrecurse <<\-EOF
	#!/bin/sh
        path=$1 old_hex=$3 new_hex=$6
        unset GIT_DIR
        cd "$path" || exit 1
        git diff "$old_hex" "$new_hex"        
        EOF
        chmod +x $HOME/bin/diff-subrecurse

The corner cases like "new submodule", "removed submodule" are left
as an exercise to the reader ;-)

Re: git diff across submodules

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> I also suspect that you do not have to change "git diff" at all to
> show the patch recursively by using the attribute mechanism (look in
> Documentation/gitattributes.text for a string GIT_EXTERNAL_DIFF).
> It might be just as simple as doing this:
>
> 	echo >.gitattributes "/lib/frotz diff=subrecurse" 
> 	git config diff.subrecurse.command $HOME/bin/diff-subrecurse
>       cat >$HOME/bin/diff-subrecurse <<\-EOF
> 	#!/bin/sh
>       path=$1 old_hex=$3 new_hex=$6
>       unset GIT_DIR
>       cd "$path" || exit 1
>       git diff "$old_hex" "$new_hex"        
>       EOF
>       chmod +x $HOME/bin/diff-subrecurse
>
> The corner cases like "new submodule", "removed submodule" are left
> as an exercise to the reader ;-)

It turns out that essentially the above outline I concocted in my
MUA is usable almost as-is.

Here is what I ended up with.

 * In .git/config of the superproject, I added this:

        [diff "submodule-recurse"]
		command = src/bin/diff-submodule-recurse

 * In the superproject, src/bin/diff-submodule-recurse has this
   (this is probably whitespace damaged---the lines must be indented
   by HT for the here document to correctly work):

        #!/bin/sh
        # $1   $2       $3      $4       $5       $6      $7
        # path old-file old-hex old-mode new-file new-hex new-mode

        case "$#,$4,$7" in
        7,160000,160000) ;;
        *)      echo "diff --git a/$1 b/$1"
                echo "(punt)"
                exit
                ;;
        esac
        unset GIT_DIR
        cd "$1" || {
                cat <<-\EOF
                diff --git a/$1 b/$1
                (cannot chdir to $1)
		-Subproject commit $3
		+Subproject commit $6
                EOF
        }
        git --no-pager diff --src-prefix="s/$1/" --dst-prefix="m/$1/" "$3" "$6"

 * In .gitattributes of the superproject, I have this:

        /var	diff=submodule-recurse

The superproject in this case is a repository to control what I have
in my $HOME directory (e.g. it has src/dot/Makefile that builds and
installs the appropriate dotfiles, src/bin/Makefile that builds and
installs to $HOME/bin, etc.), and one subdirectory, 'var', is a
submodule that is only cloned to some but not all machines I clone
this superproject to.

With this setting, things like

	$ git diff HEAD~20

show differences with recursion into the var/ submodule just fine.

Re: git diff across submodules

[Jens Lehmann]

Am 20.09.2012 00:31, schrieb Junio C Hamano:
> Junio C Hamano <gitster@pobox.com> writes:
> 
>> I also suspect that you do not have to change "git diff" at all to
>> show the patch recursively by using the attribute mechanism (look in
>> Documentation/gitattributes.text for a string GIT_EXTERNAL_DIFF).
>> It might be just as simple as doing this:
>>
>> 	echo >.gitattributes "/lib/frotz diff=subrecurse" 
>> 	git config diff.subrecurse.command $HOME/bin/diff-subrecurse
>>       cat >$HOME/bin/diff-subrecurse <<\-EOF
>> 	#!/bin/sh
>>       path=$1 old_hex=$3 new_hex=$6
>>       unset GIT_DIR
>>       cd "$path" || exit 1
>>       git diff "$old_hex" "$new_hex"        
>>       EOF
>>       chmod +x $HOME/bin/diff-subrecurse
>>
>> The corner cases like "new submodule", "removed submodule" are left
>> as an exercise to the reader ;-)
> 
> It turns out that essentially the above outline I concocted in my
> MUA is usable almost as-is.
> 
> Here is what I ended up with.
> 
>  * In .git/config of the superproject, I added this:
> 
>         [diff "submodule-recurse"]
> 		command = src/bin/diff-submodule-recurse
> 
>  * In the superproject, src/bin/diff-submodule-recurse has this
>    (this is probably whitespace damaged---the lines must be indented
>    by HT for the here document to correctly work):
> 
>         #!/bin/sh
>         # $1   $2       $3      $4       $5       $6      $7
>         # path old-file old-hex old-mode new-file new-hex new-mode
> 
>         case "$#,$4,$7" in
>         7,160000,160000) ;;
>         *)      echo "diff --git a/$1 b/$1"
>                 echo "(punt)"
>                 exit
>                 ;;
>         esac
>         unset GIT_DIR
>         cd "$1" || {
>                 cat <<-\EOF
>                 diff --git a/$1 b/$1
>                 (cannot chdir to $1)
> 		-Subproject commit $3
> 		+Subproject commit $6
>                 EOF
>         }
>         git --no-pager diff --src-prefix="s/$1/" --dst-prefix="m/$1/" "$3" "$6"
> 
>  * In .gitattributes of the superproject, I have this:
> 
>         /var	diff=submodule-recurse
> 
> The superproject in this case is a repository to control what I have
> in my $HOME directory (e.g. it has src/dot/Makefile that builds and
> installs the appropriate dotfiles, src/bin/Makefile that builds and
> installs to $HOME/bin, etc.), and one subdirectory, 'var', is a
> submodule that is only cloned to some but not all machines I clone
> this superproject to.
> 
> With this setting, things like
> 
> 	$ git diff HEAD~20
> 
> show differences with recursion into the var/ submodule just fine.

That's pretty cool! Even though diff options like --stat and --name-only
still won't take into account what happened inside the submodule this
approach makes it possible to see the diff recursively. Wouldn't it make
sense add this script to contrib (after teaching it new and removed
submodules and documenting its use in a few lines after the shebang)?

Re: git diff across submodules

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

> That's pretty cool! Even though diff options like --stat and --name-only
> still won't take into account what happened inside the submodule this
> approach makes it possible to see the diff recursively. Wouldn't it make
> sense add this script to contrib (after teaching it new and removed
> submodules and documenting its use in a few lines after the shebang)?

A few things somebody may want to work on while doing that "few
lines of documentation" I know about are:

 * From the core side, pass options that are releavant when
   generating patch (i.e. with p) in environment variables to the
   external diff script;

 * Not using "s/$1" and "m/$1" as prefix; instead, pass src/dst
   prefix values (i.e. s/ and m/) from the core side in environment
   variables, and make the external diff script itself aware of
   possibly nested submodules, e.g.

    SUBMODULE_PATH="${SUBMODULE_PATH}$1"
    export SUBMODULE_PATH
    exec git --no-pager diff -p \
    	--src-prefix="$SRC_PREFIX/$SUBMODULE_PATH" \
    	--dst-prefix="$DST_PREFIX/$SUBMODULE_PATH" "$3" "$6"

After people gain sufficient experience with it, as the next step,
we can think about how to handle --stat and other options when we
are run without -p (currently the attribute mechanism would not
trigger) and then we can call the result a native "diff that
recurses into submodules" that people can use without setting up the
attributes based mechanism.