git reset --hard isn't resetting

git reset --hard isn't resetting

[Matt Graham]

Hi,
I'm using a git svn tree in Cygwin.  I tried doing an svn rebase and
got in some weird state with local changes I can't get rid of.  It's
not an issue w/ the same repository on my linux machine.

git reset --hard
toggles 4 files between capitalization.  The files don't appear to
have changed case in svn, but it's a huge repository and not easy to
determine with certainty.

I first encountered this during a rebase, so I rebranched master from
git-svn HEAD and the problem followed me there.

Any tips about how to work around it or investigate further would be
appreciated.

scrollback of toggling filenames is pasted below.

thanks


mgraham@mgraham-wks /src/project.git
$ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#
#       modified:   dim_grade.dsx
#       modified:   dim_institution.dsx
#       modified:   dim_section.dsx
#       modified:   dim_term.dsx
#
no changes added to commit (use "git add" and/or "git commit -a")

mgraham@mgraham-wks /src/project.git
$ git reset --hard
HEAD is now at 6170b8f RPrasad - For the time being switched to
MockAuthenticator; while working on resolving the LDAP issue.

mgraham@mgraham-wks /src/project.git
$ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#
#       modified:   DIM_GRADE.dsx
#       modified:   DIM_INSTITUTION.dsx
#       modified:   DIM_SECTION.dsx
#       modified:   DIM_TERM.dsx
#
no changes added to commit (use "git add" and/or "git commit -a")

Re: git reset --hard isn't resetting

[Dmitry Potapov]

Hi,

On Wed, Aug 6, 2008 at 8:41 PM, Matt Graham <mdg149@gmail.com> wrote:
> I'm using a git svn tree in Cygwin.  I tried doing an svn rebase and
> got in some weird state with local changes I can't get rid of.  It's
> not an issue w/ the same repository on my linux machine.
>
> git reset --hard
> toggles 4 files between capitalization.  The files don't appear to
> have changed case in svn, but it's a huge repository and not easy to
> determine with certainty.

What version of Git do you use?
Was this repo created with Git prior 1.5.6?
Do you have core.ignorecase set to true in .git/config?

What "git ls-files" says for these files?
What "ls" says for these files?

Dmitry

Re: git reset --hard isn't resetting

[Avery Pennarun]

On 8/6/08, Matt Graham <mdg149@gmail.com> wrote:
>  I'm using a git svn tree in Cygwin.  I tried doing an svn rebase and
>  got in some weird state with local changes I can't get rid of.  It's
>  not an issue w/ the same repository on my linux machine.
>
>  git reset --hard
>  toggles 4 files between capitalization.  The files don't appear to
>  have changed case in svn, but it's a huge repository and not easy to
>  determine with certainty.

Try:
   git log --name-only
to see which patches change which files.  It's a virtual certainty
that they were renamed in svn at some point.

git doesn't handle case-munging filesystems perfectly, and gets into
the situation you describe.  First, you need to figure out whether you
have files with *both* cases accidentally added to your index (if git
reset toggles the capitalization, this is almost certainly the case):

    git ls-tree HEAD

If you see the same files with different case, that's your problem.

Now just 'git rm' the ones with the case you don't want, and commit
the result.  (Do *not* use commit -a!)  'git status' will give you
some funny messages indicating that files you *didn't* 'git rm' have
gone away in the filesystem; it's true, of course, but don't worry
about that.  Now 'git reset --hard HEAD' and you should be okay.

I'm not really sure what git should do better in this case, although
the current behaviour is obviously a bit confusing.

Have fun,

Avery

Re: git reset --hard isn't resetting

[Matt Graham]

On Wed, Aug 6, 2008 at 2:02 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On 8/6/08, Matt Graham <mdg149@gmail.com> wrote:
>>  I'm using a git svn tree in Cygwin.  I tried doing an svn rebase and
>>  got in some weird state with local changes I can't get rid of.  It's
>>  not an issue w/ the same repository on my linux machine.
>>
>>  git reset --hard
>>  toggles 4 files between capitalization.  The files don't appear to
>>  have changed case in svn, but it's a huge repository and not easy to
>>  determine with certainty.
>
> Try:
>   git log --name-only
> to see which patches change which files.  It's a virtual certainty
> that they were renamed in svn at some point.

They weren't "renamed".  Further investigation w/ the hated svn tools
showed that the upper case was removed, then many commits later, the
lowercase was added.

> git doesn't handle case-munging filesystems perfectly, and gets into
> the situation you describe.  First, you need to figure out whether you
> have files with *both* cases accidentally added to your index (if git
> reset toggles the capitalization, this is almost certainly the case):
>
>    git ls-tree HEAD
>
> If you see the same files with different case, that's your problem.

Indeed that was the problem.  In fact, l now noticed that my linux
machine has both versions as well.  Being case sensitive, it didn't
mind and the problem wasn't obvious.

> Now just 'git rm' the ones with the case you don't want, and commit
> the result.  (Do *not* use commit -a!)  'git status' will give you
> some funny messages indicating that files you *didn't* 'git rm' have
> gone away in the filesystem; it's true, of course, but don't worry
> about that.  Now 'git reset --hard HEAD' and you should be okay.

This worked fine exactly as you said.  I'm curious what will happen when I do
   git svn dcommit
These aren't my files and I'm sort of using git svn on the sly.  I'd
prefer to not have something weird happen to the svn repository due to
this.  Due to the schedule, our tolerance for screwing things up b/c I
want to use git will be low.  And my argument that we should have used
git from the outset probably won't help any.

> I'm not really sure what git should do better in this case, although
> the current behaviour is obviously a bit confusing.

Yes, if SVN is going to have both versions, it's understandable that
git wouldn't know what to do.  Unfortunately, it looks like SVN only
had one version at a time.  So it seems git somehow revived the
uppercase version when the lowercase one was readded through git svn.

This happened on both the cygwin and linux versions, although it only
caused an obvious problem on the cygwin version.  I don't know git
well enough to speculate why this happened, but it looks like it's a
real bug that shouldn't have happened in this case.

On cygwin I'm using 1.5.5.1 and the repository only used that version.
On linux, I currently have 1.6.0.rc0.79.gb0320 but the repo may have
been originally cloned w/ earlier versions.

Re: git reset --hard isn't resetting

[Avery Pennarun]

On Thu, Aug 7, 2008 at 9:06 PM, Matt Graham <mdg149@gmail.com> wrote:
> On Wed, Aug 6, 2008 at 2:02 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
>> Try:
>>   git log --name-only
>> to see which patches change which files.  It's a virtual certainty
>> that they were renamed in svn at some point.
>
> They weren't "renamed".  Further investigation w/ the hated svn tools
> showed that the upper case was removed, then many commits later, the
> lowercase was added.

Hmm.  Well, one possibly important thing is that if you take a diff
between the version before the old files were removed, and the version
after the new files were added, it will *look* like a rename because
git doesn't look at the intermediate revisions.  And note that this
sort of thing will be happen if you "git checkout" the before and
after versions.

> Indeed that was the problem.  In fact, l now noticed that my linux
> machine has both versions as well.  Being case sensitive, it didn't
> mind and the problem wasn't obvious.

Did your Linux machine import the data using git-svn, or did it clone
a repo from Windows that imported using git-svn?

I can imagine a situation where git-svn on Windows could get confused
and add the wrong filenames (although it would be kind of unlikely if
they really were removed in one revision, then readded in another; why
would git-svn even think about the old names in that case?).  However,
there's no explanation for a Linux system introducing such a mistake,
since the two files are just unrelated as far as Linux is concerned.

> This worked fine exactly as you said.  I'm curious what will happen when I do
>   git svn dcommit
> These aren't my files and I'm sort of using git svn on the sly.  I'd
> prefer to not have something weird happen to the svn repository due to
> this.  Due to the schedule, our tolerance for screwing things up b/c I
> want to use git will be low.  And my argument that we should have used
> git from the outset probably won't help any.

If your git-svn repo doesn't reflect *exactly* the set of files in
your real svn repo, then you've hit a pretty bad bug and you're almost
certainly going to have problems with dcommit.  On the other hand,
you're unlikely to manage to screw up your svn repo, assuming the
files you deleted were the ones that weren't supposed to be there;
"extra deleting" them from svn wouldn't be dangerous.  I'd expect git
svn dcommit to just fail with a weird error.

>> I'm not really sure what git should do better in this case, although
>> the current behaviour is obviously a bit confusing.
>
> Yes, if SVN is going to have both versions, it's understandable that
> git wouldn't know what to do.  Unfortunately, it looks like SVN only
> had one version at a time.  So it seems git somehow revived the
> uppercase version when the lowercase one was readded through git svn.

Since this seems virtually impossible, it would be nice if you could
double check your SVN repo to make sure the problem really doesn't
exist there in *any* version.  It just doesn't seem likely that git
would have had this problem if the files were cleanly removed in one
revision, then added in a later one.  I could imagine it if they were
renamed all in one revision, though, or if there was *ever* an svn
revision where both files existed at once.  In all those cases we
effectively have a bug in git-svn, but at least in the latter cases
it's an explainable one :)

Beware that svn doesn't reliably sort its filename lists, so you might
find that two different files in the *same* directory are in totally
different places in the list; perhaps you missed a filename that way.

Good luck,

Avery

Re: git reset --hard isn't resetting

[Eric Wong]

Avery Pennarun <apenwarr@gmail.com> wrote:
> On Thu, Aug 7, 2008 at 9:06 PM, Matt Graham <mdg149@gmail.com> wrote:
> > On Wed, Aug 6, 2008 at 2:02 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> > Indeed that was the problem.  In fact, l now noticed that my linux
> > machine has both versions as well.  Being case sensitive, it didn't
> > mind and the problem wasn't obvious.
> 
> Did your Linux machine import the data using git-svn, or did it clone
> a repo from Windows that imported using git-svn?
> 
> I can imagine a situation where git-svn on Windows could get confused
> and add the wrong filenames (although it would be kind of unlikely if
> they really were removed in one revision, then readded in another; why
> would git-svn even think about the old names in that case?).  However,
> there's no explanation for a Linux system introducing such a mistake,
> since the two files are just unrelated as far as Linux is concerned.

git-svn *never* touches the working tree on the filesystem directly.
It only does so via git-rebase or git-reset when dcommiting.

That said, I have no idea (nor interest in knowing the gory details) as
to how/if git works on case-insensitive filesystems.  git-svn certainly
has never done any special with them; git-svn itself will always take
path names that SVN provides as-is.

> > This worked fine exactly as you said.  I'm curious what will happen when I do
> >   git svn dcommit
> > These aren't my files and I'm sort of using git svn on the sly.  I'd
> > prefer to not have something weird happen to the svn repository due to
> > this.  Due to the schedule, our tolerance for screwing things up b/c I
> > want to use git will be low.  And my argument that we should have used
> > git from the outset probably won't help any.

Matt: try using --dry-run with dcommit to figure out what it's doing.

Whenever git-svn dcommits to SVN, it reads all of its pathnames from
already-committed history in git, so it's unlikely to be affected
by issues on the local filesystem.  However the rebase/reset after
dcommit could be problematic.  --no-rebase can probably be used with
dcommit here to avoid issues with rebase.

That said, I take no responsibility for any screwups that may happen.
(especially since Windows is involved).

> If your git-svn repo doesn't reflect *exactly* the set of files in
> your real svn repo, then you've hit a pretty bad bug and you're almost
> certainly going to have problems with dcommit.  On the other hand,
> you're unlikely to manage to screw up your svn repo, assuming the
> files you deleted were the ones that weren't supposed to be there;
> "extra deleting" them from svn wouldn't be dangerous.  I'd expect git
> svn dcommit to just fail with a weird error.

git-svn should always die/croak immediately if it notices anything
wrong.  Again, there is no guarantee nor warranty :)

> >> I'm not really sure what git should do better in this case, although
> >> the current behaviour is obviously a bit confusing.
> >
> > Yes, if SVN is going to have both versions, it's understandable that
> > git wouldn't know what to do.  Unfortunately, it looks like SVN only
> > had one version at a time.  So it seems git somehow revived the
> > uppercase version when the lowercase one was readded through git svn.
> 
> Since this seems virtually impossible, it would be nice if you could
> double check your SVN repo to make sure the problem really doesn't
> exist there in *any* version.  It just doesn't seem likely that git
> would have had this problem if the files were cleanly removed in one
> revision, then added in a later one.  I could imagine it if they were
> renamed all in one revision, though, or if there was *ever* an svn
> revision where both files existed at once.  In all those cases we
> effectively have a bug in git-svn, but at least in the latter cases
> it's an explainable one :)

One possibility is that the SVN libraries themselves fail to report
case-changing renames on Windows when git-svn is fetching.  And then
(hypothetically) git on cygwin tries to do something smart somewhere
with case-insensitive paths.

The above is purely a hunch, anybody else want to investigate that
possibility?

-- 
Eric Wong