Can git tell me which uncommitted files clash with the incoming changes?

Can git tell me which uncommitted files clash with the incoming changes?

[Mark Kharitonov]

Hi,
I have asked this question on SO
(https://stackoverflow.com/questions/53679167/can-git-tell-me-which-uncommitted-files-clash-with-the-incoming-changes)
and usually there are tons of responses on Git questions, but not on
this one.

Allow me to quote it now.

Please, observe:

    C:\Dayforce\test [master ↓2 +0 ~2 -0 !]> git pull
    error: Your local changes to the following files would be
overwritten by merge:
            2.txt
    Please commit your changes or stash them before you merge.
    Aborting
    Updating 2dc8bd0..ea343f8
    C:\Dayforce\test [master ↓2 +0 ~2 -0 !]>

Does git have a command that can tell me which uncommitted files cause
the this error? I can see them displayed by git pull, but I really do
not want to parse git pull output.

I am fully aware of `pull.rebase` and `rebase.autostash` config
options, please do not bring them up.

**EDIT 1**

It is OK to execute `git pull` first. In fact, the logic to identify
the problematic files will be done after `git pull` fails with this
reason. The way I recognize it in Powershell is:

    git pull
    # Possible exit codes:
    # 1 - either local changes or pull merge conflict (but the merge
has not been started yet)
    # 128 - a merge is in progress
    if ($LASTEXITCODE)
    {
        git merge HEAD 2> $null                      # Disambiguate
the exit code
        if ($LASTEXITCODE -eq 128)
        {
            # Two options:
            #  - pull merge conflict
            #  - a merge is in progress
            git mergetool
        }
        else
        {
            throw "Cannot pull due to uncommitted changes"
        }
    }

So, instead of aborting I would like to identify the problematic files
and essentially replicate the `rebase.autostash`, but without
`rebase`.

**EDIT 2**

I used to think that git pull outputs something like this in case of
clashes with uncommitted changes:

    C:\xyz\test [master ↓4 ↑1 +0 ~3 -0 !]> git pull
    error: Your local changes to the following files would be
overwritten by merge:
            2.txt
            a.txt
    Please commit your changes or stash them before you merge.
    Aborting
    C:\xyz\test [master ↓4 ↑1 +0 ~3 -0 !]>

Which is easy to parse. But today, I got something different:

    C:\xyz\test [master ↓4 ↑1 +0 ~2 -0 | +0 ~1 -0 !]> git pull
    error: Your local changes to the following files would be
overwritten by merge:
      1.txt a.txt
    C:\xyz\test [master ↓4 ↑1 +0 ~2 -0 | +0 ~1 -0 !]>

I do not know if this has something to do with my Powershell console
having gotten botched somehow or with some recent git update, which I
had installed automatically without noticing it.

-- 
Be well and prosper.
==============================
"There are two kinds of people.Those whose guns are loaded and those who dig."
   ("The good, the bad and the ugly")
So let us drink for our guns always be loaded.

Re: Can git tell me which uncommitted files clash with the incoming changes?

[Jeff King]

On Mon, Dec 17, 2018 at 08:08:49AM -0500, Mark Kharitonov wrote:

>     C:\Dayforce\test [master ↓2 +0 ~2 -0 !]> git pull
>     error: Your local changes to the following files would be
> overwritten by merge:
>             2.txt
>     Please commit your changes or stash them before you merge.
>     Aborting
>     Updating 2dc8bd0..ea343f8
>     C:\Dayforce\test [master ↓2 +0 ~2 -0 !]>
> 
> Does git have a command that can tell me which uncommitted files cause
> the this error? I can see them displayed by git pull, but I really do
> not want to parse git pull output.

That message is generated by merge-recursive, I believe after it's
figured out which files would need to be touched.

I don't offhand know of a way to get that _exact_ answer from another
plumbing command. But in practice it would probably be reasonable to ask
for the diff between your current branch and what you plan to merge, and
cross-reference that with the list of files with local changes.

Something like:

  git pull ;# this fails, but FETCH_HEAD is left over

  git diff-tree -z --name-only HEAD FETCH_HEAD >one
  git diff-index -z --name-only HEAD >two
  comm -z -12 one two

would work on Linux, but "comm -z" is not portable (and I suspect you
may not have comm at all on Windows). You can probably find a way to
show the common elements of the two lists using the scripting language
of your choice.

The answer that gives will be overly broad (e.g., in a case where our
local branch had touched file "foo" but other side had not, we'd
consider "foo" as a difference the two-point diff-tree, whereas a real
3-way merge would realize that we'd keep our version of "foo"). But it
might be good enough for your purposes.

-Peff

Re: Can git tell me which uncommitted files clash with the incoming changes?

[Ævar Arnfjörð Bjarmason]

On Mon, Dec 17 2018, Jeff King wrote:

> On Mon, Dec 17, 2018 at 08:08:49AM -0500, Mark Kharitonov wrote:
>
>>     C:\Dayforce\test [master ↓2 +0 ~2 -0 !]> git pull
>>     error: Your local changes to the following files would be
>> overwritten by merge:
>>             2.txt
>>     Please commit your changes or stash them before you merge.
>>     Aborting
>>     Updating 2dc8bd0..ea343f8
>>     C:\Dayforce\test [master ↓2 +0 ~2 -0 !]>
>>
>> Does git have a command that can tell me which uncommitted files cause
>> the this error? I can see them displayed by git pull, but I really do
>> not want to parse git pull output.
>
> That message is generated by merge-recursive, I believe after it's
> figured out which files would need to be touched.
>
> I don't offhand know of a way to get that _exact_ answer from another
> plumbing command. But in practice it would probably be reasonable to ask
> for the diff between your current branch and what you plan to merge, and
> cross-reference that with the list of files with local changes.
>
> Something like:
>
>   git pull ;# this fails, but FETCH_HEAD is left over
>
>   git diff-tree -z --name-only HEAD FETCH_HEAD >one
>   git diff-index -z --name-only HEAD >two
>   comm -z -12 one two
>
> would work on Linux, but "comm -z" is not portable (and I suspect you
> may not have comm at all on Windows). You can probably find a way to
> show the common elements of the two lists using the scripting language
> of your choice.
>
> The answer that gives will be overly broad (e.g., in a case where our
> local branch had touched file "foo" but other side had not, we'd
> consider "foo" as a difference the two-point diff-tree, whereas a real
> 3-way merge would realize that we'd keep our version of "foo"). But it
> might be good enough for your purposes.

Isn't this done more simply with just running the merge with
git-merge-tree? Maybe I'm missing something. E.g. earlier I had a
conflict between a WIP series of mine in next in
parse-options-cb.c. Just using git-merge-tree and grepping for conflict
markers gives me what conflicted:

    $ git merge-tree origin/master unconditional-abbrev-2 origin/next|grep -E -e '^(merged|changed in both|  (base|our|their|result))' -e '^\+======='
    [...]
    merged
      result 100644 d70c6d9afb94c77c285fe8ee3237f7a40867157a packfile.h
      our    100644 6c4037605d0dfee59a084c440506f1af11708d63 packfile.h
    changed in both
      base   100644 8c9edce52f63bcb1085b119b3a2264a97b1fb374 parse-options-cb.c
      our    100644 1afc11a9901dba25dc0f6151e5d9a7654b6e3192 parse-options-cb.c
      their  100644 e2f3eaed072f77d63890ec814d810199f57248d5 parse-options-cb.c
    +=======
    [...]

Or more simply, you can "grep -q" for '^\+=======' to ask "does this
conflict?".

Re: Can git tell me which uncommitted files clash with the incoming changes?

[Jeff King]

On Mon, Dec 17, 2018 at 07:49:00PM +0100, Ævar Arnfjörð Bjarmason wrote:

> > The answer that gives will be overly broad (e.g., in a case where our
> > local branch had touched file "foo" but other side had not, we'd
> > consider "foo" as a difference the two-point diff-tree, whereas a real
> > 3-way merge would realize that we'd keep our version of "foo"). But it
> > might be good enough for your purposes.
> 
> Isn't this done more simply with just running the merge with
> git-merge-tree? Maybe I'm missing something. E.g. earlier I had a
> conflict between a WIP series of mine in next in
> parse-options-cb.c. Just using git-merge-tree and grepping for conflict
> markers gives me what conflicted:

I forgot about the existence of merge-tree (though TBH I don't have a
huge amount of faith in antique plumbing tools like that that very few
people actually run these days).

It won't look at the working tree at all, but it could be used instead
of diff-tree to find the set of touched paths, and then that can be
correlated with the diff-files output. We'd want to see all paths, not
just conflicted ones, so you'd have to be a little fancy with the
parsing.

It's also not _quite_ the same as what git-pull is doing to merge, since
merge-recursive does fancy stuff like renames. But the distinction would
probably be OK for casual use.

-Peff

Re: Can git tell me which uncommitted files clash with the incoming changes?

[Duy Nguyen]

On Mon, Dec 17, 2018 at 2:11 PM Mark Kharitonov
<mark.kharitonov@gmail.com> wrote:
>
> Hi,
> I have asked this question on SO
> (https://stackoverflow.com/questions/53679167/can-git-tell-me-which-uncommitted-files-clash-with-the-incoming-changes)
> and usually there are tons of responses on Git questions, but not on
> this one.
>
> Allow me to quote it now.
>
> Please, observe:
>
>     C:\Dayforce\test [master ↓2 +0 ~2 -0 !]> git pull
>     error: Your local changes to the following files would be
> overwritten by merge:
>             2.txt
>     Please commit your changes or stash them before you merge.
>     Aborting
>     Updating 2dc8bd0..ea343f8
>     C:\Dayforce\test [master ↓2 +0 ~2 -0 !]>
>
> Does git have a command that can tell me which uncommitted files cause
> the this error? I can see them displayed by git pull, but I really do
> not want to parse git pull output.

Assume that you have done "git fetch origin" (or whatever master's
upstream is). Do

git diff --name-only HEAD origin/master

You get the list of files that will need to be updated. Do

git diff --name-only

to get the list of files that have local changes. If this list shares
some paths with the first list, these paths will very likely cause
"git pull" to abort.

For a better check, I think you need to do "git read-tree -m" by
yourself (to a temporary index file with --index-output) then you can
examine that file and determine what file has changed compared to HEAD
(and if the same file has local changes, git-pull will be aborted).
You may need to read more in read-tree man page.

Ideally though, git-read-tree should be able to tell what paths are
updated in "--dry-run -u" mode. But I don't think it's supported yet.
-- 
Duy

Re: Can git tell me which uncommitted files clash with the incoming changes?

[Elijah Newren]

On Mon, Dec 17, 2018 at 8:26 AM Duy Nguyen <pclouds@gmail.com> wrote:
>
> On Mon, Dec 17, 2018 at 2:11 PM Mark Kharitonov
> <mark.kharitonov@gmail.com> wrote:
> >
> > Hi,
> > I have asked this question on SO
> > (https://stackoverflow.com/questions/53679167/can-git-tell-me-which-uncommitted-files-clash-with-the-incoming-changes)
> > and usually there are tons of responses on Git questions, but not on
> > this one.
> >
> > Allow me to quote it now.
> >
> > Please, observe:
> >
> >     C:\Dayforce\test [master ↓2 +0 ~2 -0 !]> git pull
> >     error: Your local changes to the following files would be
> > overwritten by merge:
> >             2.txt
> >     Please commit your changes or stash them before you merge.
> >     Aborting
> >     Updating 2dc8bd0..ea343f8
> >     C:\Dayforce\test [master ↓2 +0 ~2 -0 !]>
> >
> > Does git have a command that can tell me which uncommitted files cause
> > the this error? I can see them displayed by git pull, but I really do
> > not want to parse git pull output.
>
> Assume that you have done "git fetch origin" (or whatever master's
> upstream is). Do
>
> git diff --name-only HEAD origin/master
>
> You get the list of files that will need to be updated. Do
>
> git diff --name-only

Are you assuming that `git diff --cached --name-only` is empty?  If it
isn't, that alone will trigger a failure (unless using an esoteric
merge strategy or an older version of git), so this assumption is
fairly reasonable to make.  But it may be worth being explicit about
for external readers.

> to get the list of files that have local changes. If this list shares
> some paths with the first list, these paths will very likely cause
> "git pull" to abort.
>
> For a better check, I think you need to do "git read-tree -m" by
> yourself (to a temporary index file with --index-output) then you can
> examine that file and determine what file has changed compared to HEAD
> (and if the same file has local changes, git-pull will be aborted).
> You may need to read more in read-tree man page.
>
> Ideally though, git-read-tree should be able to tell what paths are
> updated in "--dry-run -u" mode. But I don't think it's supported yet.

merge-recursive currently uses unpack_trees to do this "files would be
overwritten by merge" checking, so the suggestion of read-tree (which
also uses unpack_trees) makes sense.  BUT ... the error checking in
unpack_trees has both false positives and false negatives due to not
understanding renames, and it is somewhat of a nightmarish mess.  See
[1] for details.  Further, I think it warns in cases that shouldn't be
needed (both sides of history modified the same file, with the
modifications on HEAD's side being a superset of the changes on the
other side, in such a way that 3-way content merge happens to match
what is in HEAD already).  So, while the suggestions made so far give
some useful approximations, it's an approximation that will get worse
over time.  I don't have a better approximation to provide at this
time, though.


Elijah

[1] https://public-inbox.org/git/20171124195901.2581-1-newren@gmail.com/
, starting at "Note that unpack_trees() doesn't understand renames"
and running until "4-way merges simply cause the complexity to
increase with every new capability."

Re: Can git tell me which uncommitted files clash with the incoming changes?

[Duy Nguyen]

On Mon, Dec 17, 2018 at 6:17 PM Elijah Newren <newren@gmail.com> wrote:
>
> On Mon, Dec 17, 2018 at 8:26 AM Duy Nguyen <pclouds@gmail.com> wrote:
> >
> > On Mon, Dec 17, 2018 at 2:11 PM Mark Kharitonov
> > <mark.kharitonov@gmail.com> wrote:
> > >
> > > Hi,
> > > I have asked this question on SO
> > > (https://stackoverflow.com/questions/53679167/can-git-tell-me-which-uncommitted-files-clash-with-the-incoming-changes)
> > > and usually there are tons of responses on Git questions, but not on
> > > this one.
> > >
> > > Allow me to quote it now.
> > >
> > > Please, observe:
> > >
> > >     C:\Dayforce\test [master ↓2 +0 ~2 -0 !]> git pull
> > >     error: Your local changes to the following files would be
> > > overwritten by merge:
> > >             2.txt
> > >     Please commit your changes or stash them before you merge.
> > >     Aborting
> > >     Updating 2dc8bd0..ea343f8
> > >     C:\Dayforce\test [master ↓2 +0 ~2 -0 !]>
> > >
> > > Does git have a command that can tell me which uncommitted files cause
> > > the this error? I can see them displayed by git pull, but I really do
> > > not want to parse git pull output.
> >
> > Assume that you have done "git fetch origin" (or whatever master's
> > upstream is). Do
> >
> > git diff --name-only HEAD origin/master
> >
> > You get the list of files that will need to be updated. Do
> >
> > git diff --name-only
>
> Are you assuming that `git diff --cached --name-only` is empty?  If it
> isn't, that alone will trigger a failure (unless using an esoteric
> merge strategy or an older version of git), so this assumption is
> fairly reasonable to make.  But it may be worth being explicit about
> for external readers.

Actually I think Jeff's suggestion may be better since he compares
worktree with HEAD and should catch everything.

> > to get the list of files that have local changes. If this list shares
> > some paths with the first list, these paths will very likely cause
> > "git pull" to abort.
> >
> > For a better check, I think you need to do "git read-tree -m" by
> > yourself (to a temporary index file with --index-output) then you can
> > examine that file and determine what file has changed compared to HEAD
> > (and if the same file has local changes, git-pull will be aborted).
> > You may need to read more in read-tree man page.
> >
> > Ideally though, git-read-tree should be able to tell what paths are
> > updated in "--dry-run -u" mode. But I don't think it's supported yet.
>
> merge-recursive currently uses unpack_trees to do this "files would be
> overwritten by merge" checking, so the suggestion of read-tree (which
> also uses unpack_trees) makes sense.  BUT ... the error checking in
> unpack_trees has both false positives and false negatives due to not
> understanding renames, and it is somewhat of a nightmarish mess.  See
> [1] for details.  Further, I think it warns in cases that shouldn't be
> needed (both sides of history modified the same file, with the
> modifications on HEAD's side being a superset of the changes on the
> other side, in such a way that 3-way content merge happens to match
> what is in HEAD already).  So, while the suggestions made so far give
> some useful approximations, it's an approximation that will get worse
> over time.

Ah.. dang. I guess we need "git merge --dry-run" then :)

> I don't have a better approximation to provide at this
> time, though.
>
>
> Elijah
>
> [1] https://public-inbox.org/git/20171124195901.2581-1-newren@gmail.com/
> , starting at "Note that unpack_trees() doesn't understand renames"
> and running until "4-way merges simply cause the complexity to
> increase with every new capability."



-- 
Duy

Re: Can git tell me which uncommitted files clash with the incoming changes?

[Mark Kharitonov]

Guys, having git merge --dry-run would be great, but I am OK with git
merge for real as long as its output is parseable.

However, somewhere in between git 2.18 and git 2.20 the output of
merge changed and now I do not know how to parse it.
it used to be something like that:

bla bla bla
<tab>file name 1
<tab>file name 2
...
bla bla bla

But now, the files are output in one line and given that some files
may have spaces in the name I do not see how this can be parsed. If we
could have easily parseable output of merge, it would be enough for
me.


Le lun. 17 déc. 2018 à 14:37, Duy Nguyen <pclouds@gmail.com> a écrit :
>
> On Mon, Dec 17, 2018 at 6:17 PM Elijah Newren <newren@gmail.com> wrote:
> >
> > On Mon, Dec 17, 2018 at 8:26 AM Duy Nguyen <pclouds@gmail.com> wrote:
> > >
> > > On Mon, Dec 17, 2018 at 2:11 PM Mark Kharitonov
> > > <mark.kharitonov@gmail.com> wrote:
> > > >
> > > > Hi,
> > > > I have asked this question on SO
> > > > (https://stackoverflow.com/questions/53679167/can-git-tell-me-which-uncommitted-files-clash-with-the-incoming-changes)
> > > > and usually there are tons of responses on Git questions, but not on
> > > > this one.
> > > >
> > > > Allow me to quote it now.
> > > >
> > > > Please, observe:
> > > >
> > > >     C:\Dayforce\test [master ↓2 +0 ~2 -0 !]> git pull
> > > >     error: Your local changes to the following files would be
> > > > overwritten by merge:
> > > >             2.txt
> > > >     Please commit your changes or stash them before you merge.
> > > >     Aborting
> > > >     Updating 2dc8bd0..ea343f8
> > > >     C:\Dayforce\test [master ↓2 +0 ~2 -0 !]>
> > > >
> > > > Does git have a command that can tell me which uncommitted files cause
> > > > the this error? I can see them displayed by git pull, but I really do
> > > > not want to parse git pull output.
> > >
> > > Assume that you have done "git fetch origin" (or whatever master's
> > > upstream is). Do
> > >
> > > git diff --name-only HEAD origin/master
> > >
> > > You get the list of files that will need to be updated. Do
> > >
> > > git diff --name-only
> >
> > Are you assuming that `git diff --cached --name-only` is empty?  If it
> > isn't, that alone will trigger a failure (unless using an esoteric
> > merge strategy or an older version of git), so this assumption is
> > fairly reasonable to make.  But it may be worth being explicit about
> > for external readers.
>
> Actually I think Jeff's suggestion may be better since he compares
> worktree with HEAD and should catch everything.
>
> > > to get the list of files that have local changes. If this list shares
> > > some paths with the first list, these paths will very likely cause
> > > "git pull" to abort.
> > >
> > > For a better check, I think you need to do "git read-tree -m" by
> > > yourself (to a temporary index file with --index-output) then you can
> > > examine that file and determine what file has changed compared to HEAD
> > > (and if the same file has local changes, git-pull will be aborted).
> > > You may need to read more in read-tree man page.
> > >
> > > Ideally though, git-read-tree should be able to tell what paths are
> > > updated in "--dry-run -u" mode. But I don't think it's supported yet.
> >
> > merge-recursive currently uses unpack_trees to do this "files would be
> > overwritten by merge" checking, so the suggestion of read-tree (which
> > also uses unpack_trees) makes sense.  BUT ... the error checking in
> > unpack_trees has both false positives and false negatives due to not
> > understanding renames, and it is somewhat of a nightmarish mess.  See
> > [1] for details.  Further, I think it warns in cases that shouldn't be
> > needed (both sides of history modified the same file, with the
> > modifications on HEAD's side being a superset of the changes on the
> > other side, in such a way that 3-way content merge happens to match
> > what is in HEAD already).  So, while the suggestions made so far give
> > some useful approximations, it's an approximation that will get worse
> > over time.
>
> Ah.. dang. I guess we need "git merge --dry-run" then :)
>
> > I don't have a better approximation to provide at this
> > time, though.
> >
> >
> > Elijah
> >
> > [1] https://public-inbox.org/git/20171124195901.2581-1-newren@gmail.com/
> > , starting at "Note that unpack_trees() doesn't understand renames"
> > and running until "4-way merges simply cause the complexity to
> > increase with every new capability."
>
>
>
> --
> Duy



-- 
Be well and prosper.
==============================
"There are two kinds of people.Those whose guns are loaded and those who dig."
   ("The good, the bad and the ugly")
So let us drink for our guns always be loaded.

Re: Can git tell me which uncommitted files clash with the incoming changes?

[Jeff King]

On Mon, Dec 17, 2018 at 05:50:31PM -0500, Mark Kharitonov wrote:

> Guys, having git merge --dry-run would be great, but I am OK with git
> merge for real as long as its output is parseable.
> 
> However, somewhere in between git 2.18 and git 2.20 the output of
> merge changed and now I do not know how to parse it.
> it used to be something like that:
> 
> bla bla bla
> <tab>file name 1
> <tab>file name 2
> ...
> bla bla bla
> 
> But now, the files are output in one line and given that some files
> may have spaces in the name I do not see how this can be parsed. If we
> could have easily parseable output of merge, it would be enough for
> me.

Interesting. I don't see that behavior at all. E.g., running this:

-- >8 --
git init repo
cd repo

echo base >base; git add base; git commit -m base
for i in one two three; do
	echo $i >$i
done
git add .
git commit -m master

git checkout -b other HEAD^
echo other >other; git add other; git commit -m other

for i in one two three; do
	echo working-tree >$i
done

git pull . master
-- 8< --

I see:

  error: The following untracked working tree files would be overwritten by merge:
  	one
  	three
  	two
  Please move or remove them before you merge.

I wonder if it has to do with Windows.

If you can reproduce it at will, can you try bisecting between v2.18 and
v2.20 to see which commit introduced the change?

-Peff

Re: Can git tell me which uncommitted files clash with the incoming changes?

[Elijah Newren]

On Tue, Dec 18, 2018 at 5:14 AM Jeff King <peff@peff.net> wrote:
>
> On Mon, Dec 17, 2018 at 05:50:31PM -0500, Mark Kharitonov wrote:
>
> > Guys, having git merge --dry-run would be great, but I am OK with git
> > merge for real as long as its output is parseable.

Don't rely on that.  merge output has changed occasionally and will
likely change again in the future multiple times.  The other solutions
provided by Peff and Duy are much better.  If we need to add more
options to provide you with what you need, then that's a route we can
take, but I'll make no guarantees about merge output being stable and
parseable.

All that said...

> > However, somewhere in between git 2.18 and git 2.20 the output of
> > merge changed and now I do not know how to parse it.
> > it used to be something like that:
> >
> > bla bla bla
> > <tab>file name 1
> > <tab>file name 2
> > ...
> > bla bla bla
> >
> > But now, the files are output in one line and given that some files
> > may have spaces in the name I do not see how this can be parsed. If we
> > could have easily parseable output of merge, it would be enough for
> > me.
>
> Interesting. I don't see that behavior at all. E.g., running this:
>
...
> I see:
>
>   error: The following untracked working tree files would be overwritten by merge:
>         one
>         three
>         two
>   Please move or remove them before you merge.
>
> I wonder if it has to do with Windows.
>
> If you can reproduce it at will, can you try bisecting between v2.18 and
> v2.20 to see which commit introduced the change?
>
> -Peff

I see the same as Peff, and I see no changes to
unpack_trees.c:display_error_msgs() since git 2.11 (not even in a
clone of git-for-windows), and as far as I can tell that function is
the place that prints all the files and adds the newlines, so I'm kind
of perplexed how you're seeing things print multiple files on a line.
Bisecting as Peff suggests would help but I'm curious if there's
something special about your setup needed to reproduce and which
changed since you were using 2.18 (e.g. always forcing output through
a pager and having a pager that doesn't understand LF and requires
CRLF to display things correctly??)