sporadic git failures on interactive rebase

sporadic git failures on interactive rebase

[Henning Moll]

Hi,

(git version 2.2.0)

I am currently developing/testing a script for a "history surgery" on a quite big repository (~30000 commits). The script always runs against exactly the same copy of a git repository. So things should be reproducable, but sometimes i get failures for the following sequence of commands:  

$ git checkout some_branch
$ GIT_SEQUENCE_EDITOR="sed -i '1s/^pick /edit /'" git rebase -i $MERGETARGET~1
$ git rm -rf some_files
$ git commit --amend --no-edit -c $MERGETARGET
$ git rebase --continue

(where MERGETARGET is a valid commit id in the history of some_branch)

Here is an example output where things went wrong (MERGETARGET is 6185ac39299a740dc9bc6c5906dd1f229b3f046b). The interesting parts are the messages "c4095c1: not a commit that can be picked" and "error: short SHA1 c4095c1 is ambiguous.":

. Switched to branch 'master-flat'
. c4095c1: not a commit that can be picked
. 80f99bd: not a commit that can be picked
. Stopped at 6185ac39299a740dc9bc6c5906dd1f229b3f046b... some_comment
. You can amend the commit now, with
. 
.         git commit --amend
. 
. Once you are satisfied with your changes, run
. 
.         git rebase --continue
. rm 'some_files'
. [detached HEAD 56675a06316345ac121997dde2b9eddb649d0539] some_comment
.  Author: user <foo.bar@com.com>
.  Date: Wed Sep 26 09:11:17 2012 +0000
. [more info about that commit]
. error: short SHA1 c4095c1 is ambiguous.
. fatal: Needed a single revision
. Invalid commit name: c4095c1

Now that the command failed, i checked for ambigous c4095c1. But there is only one:
$ git log -1 c4095c1
. commit c4095c1f5e7c126accf93ba68e2fa72bb055f567
. ...

Just as a test:
$ git log -1 c409
. error: short SHA1 c409 is ambiguous.
. error: short SHA1 c409 is ambiguous.

As i said above this issue is not strictly reproducable. I have a full backup of the working dir including the .git folder from just before the command sequence above. If i rollback and execute the commands again, everything works fine. Up to now i didn't had the situation the the sequence failed twice two in a row.

I also have the backup of the directory from directly after the sequence failure. So i can still test some commands if you need more information.

Does anybody have an idea of what's going on here?

Best regards
Henning

Re: sporadic git failures on interactive rebase

[Jeff King]

On Tue, Jan 13, 2015 at 11:54:32AM +0100, Henning Moll wrote:

> . error: short SHA1 c4095c1 is ambiguous.
> . fatal: Needed a single revision
> . Invalid commit name: c4095c1
> 
> Now that the command failed, i checked for ambigous c4095c1. But there is only one:
> $ git log -1 c4095c1
> . commit c4095c1f5e7c126accf93ba68e2fa72bb055f567
> . ...
> 
> Just as a test:
> $ git log -1 c409
> . error: short SHA1 c409 is ambiguous.
> . error: short SHA1 c409 is ambiguous.

Hmm. There are some instances in git where we know we are looking for an
object of a particular type, and we can disambiguate a short-sha1 based
on the type. And "git log" is just such a place, whereas a generic "git
rev-parse" used by the git-rebase script would not be.

Try:

  git rev-list --objects --all |
  grep ^c4095c1 |
  cut -d' ' -f1 |
  git cat-file --batch-check

If my hunch is right, then you may find multiple objects, only one of
which is a commit.

And if so, the solution is likely one of:

  1. Teach rev-parse a "prefer committish" option and use it in the
     appropriate spot in git-rebase.

  2. Teach whatever is generating the shortended sha1s to report a truly
     unambiguous result, not one that depends on the type. I don't think
     we _have_ a codepath to do type-dependent shortening, though, which
     is maybe an indication that this is a red herring.

It's also possible that part of the rebase sequence generates a new
object that creates the ambiguity. That would be consistent with your
statement that rolling back and running the commands again makes it
work. I'm not sure how to defend against that, aside from adding a
character or two to the shortened ids, which reduces the likelihood of a
collision.

-Peff

Re: sporadic git failures on interactive rebase

[Jeff King]

On Wed, Jan 14, 2015 at 07:19:15AM -0500, Jeff King wrote:

> Hmm. There are some instances in git where we know we are looking for an
> object of a particular type, and we can disambiguate a short-sha1 based
> on the type. And "git log" is just such a place, whereas a generic "git
> rev-parse" used by the git-rebase script would not be.
> [...]
>      I don't think
>      we _have_ a codepath to do type-dependent shortening, though, which
>      is maybe an indication that this is a red herring.

Yeah, I think this cannot be it. There is a 7-character commit/blob
ambiguity in git. You can find it yourself with:

  git rev-list --objects --all |
  cut -d' ' -f1 |
  sort | uniq -w 7 -D |
  git cat-file --batch-check |
  head -2

which produces:

  01319837c53050109c60e6740dfa9462327161f0 commit 649
  0131983dfbc143ce5dae77e067663bb2e7d5f126 blob 20638

And it behaves as expected. Running "git rev-parse 0131983" complains of
the ambiguity, but "git log 0131983" shows the commit.

What happens if we rebase with it?

  $ git checkout 01319837
  $ git rebase -i HEAD^

will yield a todo file with the 8-character unambiguous abbreviation.

So I guess all is working as intended there. Perhaps you really were
just very unlucky and an earlier step of the rebase created a
conflicting sha1.

-Peff

Re: sporadic git failures on interactive rebase

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> What happens if we rebase with it?
>
>   $ git checkout 01319837
>   $ git rebase -i HEAD^
>
> will yield a todo file with the 8-character unambiguous abbreviation.
>
> So I guess all is working as intended there. Perhaps you really were
> just very unlucky and an earlier step of the rebase created a
> conflicting sha1.

That would mean 75c69766 (rebase -i: fix short SHA-1 collision,
2013-08-23) did not fix what it intended to fix, no?  Is the symptom
coming from pre-1.8.4.2 version of Git?

Re: sporadic git failures on interactive rebase

[Jeff King]

On Wed, Jan 14, 2015 at 09:12:46AM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > What happens if we rebase with it?
> >
> >   $ git checkout 01319837
> >   $ git rebase -i HEAD^
> >
> > will yield a todo file with the 8-character unambiguous abbreviation.
> >
> > So I guess all is working as intended there. Perhaps you really were
> > just very unlucky and an earlier step of the rebase created a
> > conflicting sha1.
> 
> That would mean 75c69766 (rebase -i: fix short SHA-1 collision,
> 2013-08-23) did not fix what it intended to fix, no?  Is the symptom
> coming from pre-1.8.4.2 version of Git?

Yeah, you're right. I didn't even remember that commit at all. On the
off chance that the abbreviation code was different in that earlier
version, I also checked rebasing 01319837 with an older version, but it
does work fine.

So yeah, the most plausible theory to me so far is unluckiness combined
with pre-1.8.4.2. That should be easy to disprove if Henning tells us
his git version.

-Peff

Re: sporadic git failures on interactive rebase

[Eric Sunshine]

On Wed, Jan 14, 2015 at 3:54 PM, Jeff King <peff@peff.net> wrote:
> On Wed, Jan 14, 2015 at 09:12:46AM -0800, Junio C Hamano wrote:
>
>> Jeff King <peff@peff.net> writes:
>>
>> > What happens if we rebase with it?
>> >
>> >   $ git checkout 01319837
>> >   $ git rebase -i HEAD^
>> >
>> > will yield a todo file with the 8-character unambiguous abbreviation.
>> >
>> > So I guess all is working as intended there. Perhaps you really were
>> > just very unlucky and an earlier step of the rebase created a
>> > conflicting sha1.
>>
>> That would mean 75c69766 (rebase -i: fix short SHA-1 collision,
>> 2013-08-23) did not fix what it intended to fix, no?  Is the symptom
>> coming from pre-1.8.4.2 version of Git?
>
> Yeah, you're right. I didn't even remember that commit at all. On the
> off chance that the abbreviation code was different in that earlier
> version, I also checked rebasing 01319837 with an older version, but it
> does work fine.
>
> So yeah, the most plausible theory to me so far is unluckiness combined
> with pre-1.8.4.2. That should be easy to disprove if Henning tells us
> his git version.

Henning mentioned it at the very top of his original problem report:

    "(git version 2.2.0)"

Re: sporadic git failures on interactive rebase

[Eric Sunshine]

On Wed, Jan 14, 2015 at 4:00 PM, Eric Sunshine <sunshine@sunshineco.com> wrote:
> On Wed, Jan 14, 2015 at 3:54 PM, Jeff King <peff@peff.net> wrote:
>> On Wed, Jan 14, 2015 at 09:12:46AM -0800, Junio C Hamano wrote:
>>
>>> Jeff King <peff@peff.net> writes:
>>>
>>> > What happens if we rebase with it?
>>> >
>>> >   $ git checkout 01319837
>>> >   $ git rebase -i HEAD^
>>> >
>>> > will yield a todo file with the 8-character unambiguous abbreviation.
>>> >
>>> > So I guess all is working as intended there. Perhaps you really were
>>> > just very unlucky and an earlier step of the rebase created a
>>> > conflicting sha1.
>>>
>>> That would mean 75c69766 (rebase -i: fix short SHA-1 collision,
>>> 2013-08-23) did not fix what it intended to fix, no?  Is the symptom
>>> coming from pre-1.8.4.2 version of Git?
>>
>> Yeah, you're right. I didn't even remember that commit at all. On the
>> off chance that the abbreviation code was different in that earlier
>> version, I also checked rebasing 01319837 with an older version, but it
>> does work fine.
>>
>> So yeah, the most plausible theory to me so far is unluckiness combined
>> with pre-1.8.4.2. That should be easy to disprove if Henning tells us
>> his git version.
>
> Henning mentioned it at the very top of his original problem report:
>
>     "(git version 2.2.0)"

For completeness: http://article.gmane.org/gmane.comp.version-control.git/262334

Re: sporadic git failures on interactive rebase

[Jeff King]

On Wed, Jan 14, 2015 at 04:00:37PM -0500, Eric Sunshine wrote:

> > So yeah, the most plausible theory to me so far is unluckiness combined
> > with pre-1.8.4.2. That should be easy to disprove if Henning tells us
> > his git version.
> 
> Henning mentioned it at the very top of his original problem report:
> 
>     "(git version 2.2.0)"

Ah, reading comprehension. It pays off.

I'm stumped, then.

-Peff

Re: sporadic git failures on interactive rebase

[Eric Sunshine]

On Wed, Jan 14, 2015 at 4:02 PM, Jeff King <peff@peff.net> wrote:
> On Wed, Jan 14, 2015 at 04:00:37PM -0500, Eric Sunshine wrote:
>
>> > So yeah, the most plausible theory to me so far is unluckiness combined
>> > with pre-1.8.4.2. That should be easy to disprove if Henning tells us
>> > his git version.
>>
>> Henning mentioned it at the very top of his original problem report:
>>
>>     "(git version 2.2.0)"
>
> Ah, reading comprehension. It pays off.
>
> I'm stumped, then.

Perhaps some new code been added to git-rebase--interactive.sh since
75c69766 which neglects to invoke expand_todo_ids()?

Or, possibly some older version of git is being invoked somehow during
rebase despite his "front end" use of 2.2.0?

Re: sporadic git failures on interactive rebase

[Junio C Hamano]

Eric Sunshine <sunshine@sunshineco.com> writes:

> On Wed, Jan 14, 2015 at 4:02 PM, Jeff King <peff@peff.net> wrote:
>> On Wed, Jan 14, 2015 at 04:00:37PM -0500, Eric Sunshine wrote:
>>
>>> > So yeah, the most plausible theory to me so far is unluckiness combined
>>> > with pre-1.8.4.2. That should be easy to disprove if Henning tells us
>>> > his git version.
>>>
>>> Henning mentioned it at the very top of his original problem report:
>>>
>>>     "(git version 2.2.0)"
>>
>> Ah, reading comprehension. It pays off.
>>
>> I'm stumped, then.
>
> Perhaps some new code been added to git-rebase--interactive.sh since
> 75c69766 which neglects to invoke expand_todo_ids()?
>
> Or, possibly some older version of git is being invoked somehow during
> rebase despite his "front end" use of 2.2.0?

Or git_sequence_editor creates conflicting objects?