git reset --hard not removing some files

git reset --hard not removing some files

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]

hoi :)

I have the following problem:


nbg1l001:~/src/git > git branch
* master
  next
  origin
nbg1l001:~/src/git > git checkout -b test
nbg1l001:~/src/git > git status
# On branch refs/heads/test
nothing to commit
zsh: exit 1     git status
nbg1l001:~/src/git > git reset --hard v1.3.3
nbg1l001:~/src/git > git status
# On branch refs/heads/test
#
# Untracked files:
#   (use "git add" to add to commit)
#
#       git-quiltimport
#       git-upload-tar
nothing to commit
zsh: exit 1     git status
nbg1l001:~/src/git > git reset --hard master
nbg1l001:~/src/git > git status
# On branch refs/heads/test
nothing to commit
zsh: exit 1     git status

nbg1l001:~/src/git > git --version
git version 1.3.3.g0825d


However, the complete test suite and especially t7101-reset.sh succeed.
Any ideas?

-- 
Martin Waitz

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: git reset --hard not removing some files

[Sean]

On Thu, 1 Jun 2006 18:00:52 +0200
Martin Waitz <tali@admingilde.org> wrote:

Removed your prompt just to make it a bit more readable:

> $ git branch
> * master
>   next
>   origin
>
> $ git checkout -b test
>
> $ git status
> # On branch refs/heads/test
> nothing to commit

The generated files "git-quiltimport" and "git-upload-tar" exist at
this point.  They are both untracked files and aren't listed because
they have entries in the .gitignore file.

> $ git reset --hard v1.3.3
>
> $ git status
> # On branch refs/heads/test
> #
> # Untracked files:
> #   (use "git add" to add to commit)
> #
> #       git-quiltimport
> #       git-upload-tar
> nothing to commit

Resetting to version 1.3.3 gets you an old version of the .gitignore
file which doesn't ignore these two untracked files.  Reset --hard
doesn't remove them because it only deals with tracked files.  Thus,
they show up in your status report.

> $ git reset --hard master
> $ git status
> # On branch refs/heads/test
> nothing to commit

Returning to the current version gets you an updated .gitignore and the
two files are no longer listed even though they still exist.

> $ git --version
> git version 1.3.3.g0825d

This is the expected behavior regardless of version.

Sean

Re: git reset --hard not removing some files

[Sean]

On Thu, 1 Jun 2006 18:00:52 +0200
Martin Waitz <tali@admingilde.org> wrote:

Removed your prompt just to make it a bit more readable:

> $ git branch
> * master
>   next
>   origin
>
> $ git checkout -b test
>
> $ git status
> # On branch refs/heads/test
> nothing to commit

The generated files "git-quiltimport" and "git-upload-tar" exist at
this point.  They are both untracked files and aren't listed because
they have entries in the .gitignore file.

> $ git reset --hard v1.3.3
>
> $ git status
> # On branch refs/heads/test
> #
> # Untracked files:
> #   (use "git add" to add to commit)
> #
> #       git-quiltimport
> #       git-upload-tar
> nothing to commit

Resetting to version 1.3.3 gets you an old version of the .gitignore
file which doesn't ignore these two untracked files.  Reset --hard
doesn't remove them because it only deals with tracked files.  Thus,
they show up in your status report.

> $ git reset --hard master
> $ git status
> # On branch refs/heads/test
> nothing to commit

Returning to the current version gets you an updated .gitignore and the
two files are no longer listed even though they still exist.

> $ git --version
> git version 1.3.3.g0825d

This is the expected behavior regardless of version.

Sean

Re: git reset --hard not removing some files

[Jakub Narebski]

Sean wrote:

> On Thu, 1 Jun 2006 18:00:52 +0200
> Martin Waitz <tali@admingilde.org> wrote:
> 
>> $ git reset --hard v1.3.3
>>
>> $ git status
>> # On branch refs/heads/test
>> #
>> # Untracked files:
>> #   (use "git add" to add to commit)
>> #
>> #       git-quiltimport
>> #       git-upload-tar
>> nothing to commit
> 
> Resetting to version 1.3.3 gets you an old version of the .gitignore
> file which doesn't ignore these two untracked files.  Reset --hard
> doesn't remove them because it only deals with tracked files.  Thus,
> they show up in your status report.

Do you think it is _frequently_ asked question, worth adding
to http://git.or.cz/gitwiki/GitFaq ?

-- 
Jakub Narebski
Warsaw, Poland

Re: git reset --hard not removing some files

[Sean]

On Thu, 01 Jun 2006 19:21:19 +0200
Jakub Narebski <jnareb@gmail.com> wrote:

> Do you think it is _frequently_ asked question, worth adding
> to http://git.or.cz/gitwiki/GitFaq ?

Hi Jakub,

Not sure if it needs to be in the FAQ at this point, but i guess it couldn't
hurt either.  The man page actually mentions that --hard deals with tracked
files.  An extra example or a more explicit notice about untracked files
might be helpful though.

Sean

Re: git reset --hard not removing some files

[Sean]

On Thu, 01 Jun 2006 19:21:19 +0200
Jakub Narebski <jnareb@gmail.com> wrote:

> Do you think it is _frequently_ asked question, worth adding
> to http://git.or.cz/gitwiki/GitFaq ?

Hi Jakub,

Not sure if it needs to be in the FAQ at this point, but i guess it couldn't
hurt either.  The man page actually mentions that --hard deals with tracked
files.  An extra example or a more explicit notice about untracked files
might be helpful though.

Sean

Re: git reset --hard not removing some files

[Jakub Narebski]

Sean wrote:

> On Thu, 01 Jun 2006 19:21:19 +0200
> Jakub Narebski <jnareb@gmail.com> wrote:
> 
>> Do you think it is _frequently_ asked question, worth adding
>> to http://git.or.cz/gitwiki/GitFaq ?
> 
> Not sure if it needs to be in the FAQ at this point, but i guess it couldn't
> hurt either.  

Added. See: http://git.or.cz/gitwiki/GitFaq#reset-hard-leaving-files
Feel free to add examples, correct information, rework it etc.

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: git reset --hard not removing some files

[Linus Torvalds]

On Thu, 1 Jun 2006, Martin Waitz wrote:
> 
> I have the following problem:

It's not a problem, it's a feature.

> nbg1l001:~/src/git > git branch
> * master
>   next
>   origin
> nbg1l001:~/src/git > git checkout -b test
> nbg1l001:~/src/git > git status
> # On branch refs/heads/test
> nothing to commit
> zsh: exit 1     git status
> nbg1l001:~/src/git > git reset --hard v1.3.3
> nbg1l001:~/src/git > git status
> # On branch refs/heads/test
> #
> # Untracked files:
> #   (use "git add" to add to commit)
> #
> #       git-quiltimport
> #       git-upload-tar

Those files were _never_ tracked.

What happened is that when you switched to an earlier version, you _also_ 
switched the ".gitignore" file to the earlier version, and that older 
.gitignore file doesn't have those two (newer) binaries listed as being 
ignored.

And "git reset" won't be deleting files it doesn't track (it had _better_ 
not touch them), even more so when it has been told to ignore them, so it 
makes total sense to _not_ delete them when doing that reset.

			Linus

Re: git reset --hard not removing some files

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 1294 bytes --]

hoi :)

On Thu, Jun 01, 2006 at 09:21:38AM -0700, Linus Torvalds wrote:
> Those files were _never_ tracked.

You are right, I used the wrong demonstration to show my problem.
I had one problem like this in the linux-kernel and tried to
reproduce them in the git repository but did it wrong...

GIT reset seems to have a problem when a file is tracked and ignored
at the same time.

This fails:

diff --git a/t/t7101-reset.sh b/t/t7101-reset.sh
index a919140..865e0f6 100755
--- a/t/t7101-reset.sh
+++ b/t/t7101-reset.sh
@@ -21,10 +21,12 @@ test_expect_success \
      cp ../../COPYING path1/COPYING &&
      cp ../../COPYING COPYING &&
      cp ../../COPYING path0/COPYING-TOO &&
+     echo COPYING > .gitignore &&
      git-add path1/path2/COPYING &&
      git-add path1/COPYING &&
      git-add COPYING &&
      git-add path0/COPYING-TOO &&
+     git-add .gitignore &&
      git-commit -m change -a'

 test_expect_success \


This hit me as the Linux kernel .gitignore includes ".*" which matches
all the .gitignore files and so they are not removed when you go
back to 2.6.13.  But with the new git checks regarding files in
the working dir, git then refuses to pull in newer Linux versions
because it would overwrite ".gitignore".

-- 
Martin Waitz

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: git reset --hard not removing some files

[Sean]

On Fri, 2 Jun 2006 11:37:36 +0200
Martin Waitz <tali@admingilde.org> wrote:

Hi Martin,

> GIT reset seems to have a problem when a file is tracked and ignored
> at the same time.

I tried to reproduce this problem, but couldn't.

> This fails:
> 
> diff --git a/t/t7101-reset.sh b/t/t7101-reset.sh
> index a919140..865e0f6 100755
> --- a/t/t7101-reset.sh
> +++ b/t/t7101-reset.sh
> @@ -21,10 +21,12 @@ test_expect_success \
>       cp ../../COPYING path1/COPYING &&
>       cp ../../COPYING COPYING &&
>       cp ../../COPYING path0/COPYING-TOO &&
> +     echo COPYING > .gitignore &&
>       git-add path1/path2/COPYING &&
>       git-add path1/COPYING &&
>       git-add COPYING &&
>       git-add path0/COPYING-TOO &&
> +     git-add .gitignore &&
>       git-commit -m change -a'
> 
>  test_expect_success \

This fails because git-add refuses to mark an ignored file as tracked.
So in your altered test above the COPYING file is never tracked.
git-reset then does the proper thing and refuses to remove it.  So it's
still not demonstrating the case you mention below.
 
> This hit me as the Linux kernel .gitignore includes ".*" which matches
> all the .gitignore files and so they are not removed when you go
> back to 2.6.13.  But with the new git checks regarding files in
> the working dir, git then refuses to pull in newer Linux versions
> because it would overwrite ".gitignore".

Strange.. this little test shows that tracked files that are ignored
do get removed when resetting --hard...

$ mkdir tryit ; cd tryit
$ git init-db
defaulting to local storage area

$ touch one ; git add one
$ git commit -a -m"Initial"
Committing initial tree 5fcffbd6e4c5c5b8d81f5e9314b20e338e3ffff5

$ touch .gitignore
$ git add .gitignore
$ git commit -a -m"now we have gitignore"

$ echo '.*' >> .gitignore
$ git commit -a -m'now git ignores all .* files'

$ ls -l .gitignore
-rw-r--r-- 1 sean sean 3 Jun  2 06:01 .gitignore

$ git reset --hard master~2
$ ls -l .gitignore
ls: .gitignore: No such file or directory


Sean

Re: git reset --hard not removing some files

[Sean]

On Fri, 2 Jun 2006 11:37:36 +0200
Martin Waitz <tali@admingilde.org> wrote:

Hi Martin,

> GIT reset seems to have a problem when a file is tracked and ignored
> at the same time.

I tried to reproduce this problem, but couldn't.

> This fails:
> 
> diff --git a/t/t7101-reset.sh b/t/t7101-reset.sh
> index a919140..865e0f6 100755
> --- a/t/t7101-reset.sh
> +++ b/t/t7101-reset.sh
> @@ -21,10 +21,12 @@ test_expect_success \
>       cp ../../COPYING path1/COPYING &&
>       cp ../../COPYING COPYING &&
>       cp ../../COPYING path0/COPYING-TOO &&
> +     echo COPYING > .gitignore &&
>       git-add path1/path2/COPYING &&
>       git-add path1/COPYING &&
>       git-add COPYING &&
>       git-add path0/COPYING-TOO &&
> +     git-add .gitignore &&
>       git-commit -m change -a'
> 
>  test_expect_success \

This fails because git-add refuses to mark an ignored file as tracked.
So in your altered test above the COPYING file is never tracked.
git-reset then does the proper thing and refuses to remove it.  So it's
still not demonstrating the case you mention below.
 
> This hit me as the Linux kernel .gitignore includes ".*" which matches
> all the .gitignore files and so they are not removed when you go
> back to 2.6.13.  But with the new git checks regarding files in
> the working dir, git then refuses to pull in newer Linux versions
> because it would overwrite ".gitignore".

Strange.. this little test shows that tracked files that are ignored
do get removed when resetting --hard...

$ mkdir tryit ; cd tryit
$ git init-db
defaulting to local storage area

$ touch one ; git add one
$ git commit -a -m"Initial"
Committing initial tree 5fcffbd6e4c5c5b8d81f5e9314b20e338e3ffff5

$ touch .gitignore
$ git add .gitignore
$ git commit -a -m"now we have gitignore"

$ echo '.*' >> .gitignore
$ git commit -a -m'now git ignores all .* files'

$ ls -l .gitignore
-rw-r--r-- 1 sean sean 3 Jun  2 06:01 .gitignore

$ git reset --hard master~2
$ ls -l .gitignore
ls: .gitignore: No such file or directory


Sean

Re: git reset --hard not removing some files

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 326 bytes --]

hoi :)

On Fri, Jun 02, 2006 at 06:08:20AM -0400, Sean wrote:
> Strange.. this little test shows that tracked files that are ignored
> do get removed when resetting --hard...

ok, git must be behaving correctly then, I don't know what problem I ran into.
Must have been some kind of user error...

-- 
Martin Waitz

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: git reset --hard not removing some files

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> On Thu, 1 Jun 2006, Martin Waitz wrote:
>> 
>> I have the following problem:
>
> It's not a problem, it's a feature.
>...
> Those files were _never_ tracked.

I would agree in the reproduction recipe Martin gave there is no
problem but feature, but at the same time I suspect the recent
"reset --hard simplification" has introduced a true regression.

        $ mkdir test || exit
        $ cd test || exit
        $ git init-db
        defaulting to local storage area
        $ echo init >file0
        $ echo init >file1
        $ git add file0 file1
        $ git commit -m initial
        Committing initial tree de84dc367842fdbbb3acad3b3ed252f8c984296f
        $ git branch side
        $ rm -f file1
        $ echo second >file2
        $ git add file2
        $ git commit -a -m 'master adds file2 and deletes file1'
        $ git checkout side
        $ echo modified >file1
        $ git commit -a -m 'side edits file1'
        $ git checkout master
        $ ls
        file0  file2
        $ git pull . side
        Trying really trivial in-index merge...
        fatal: Merge requires file-level merging
        Nope.
        Merging HEAD with 7934c9c383f611cf2b9895a46cf95b815569beef
        Merging: 
        b684570dc1141552d0da950a18f2d84a3ffadbc1 master adds file2 and deletes file1 
        7934c9c383f611cf2b9895a46cf95b815569beef side edits file1 
        found 1 common ancestor(s): 
        598d6491f72b6057ca87683d43cf64b08d90ddaf initial 
        CONFLICT (delete/modify): file1 deleted in HEAD and modified in 7934c9c383f611cf2b9895a46cf95b815569beef. Version 7934c9c383f611cf2b9895a46cf95b815569beef of file1 left in tree. 

        Automatic merge failed; fix conflicts and then commit the result.
        $ git ls-files -u
        100644 b1b716105590454bfc4c0247f193a04088f39c7f 1	file1
        100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 3	file1
        $ ls
        file0  file1  file2
        $ git reset --hard
        $ ls
        file0  file1  file2

We used to remove file1 from the working tree in this case.  One
of the most important reason to use "git reset --hard" is to
recover from a conflicted, failed merge.  

Leaving file1 in the working tree around in this case has
unpleasant consequences.  After the above:

	$ git checkout side
	fatal: Untracked working tree file 'file1' would be overwritten by merge.

Re: git reset --hard not removing some files

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> I would agree in the reproduction recipe Martin gave there is no
> problem but feature, but at the same time I suspect the recent
> "reset --hard simplification" has introduced a true regression.
> ...
>         $ git ls-files -u
>         100644 b1b716105590454bfc4c0247f193a04088f39c7f 1	file1
>         100644 2e0996000b7e9019eabcad29391bf0f5c7702f0b 3	file1
>         $ ls
>         file0  file1  file2
>         $ git reset --hard
>         $ ls
>         file0  file1  file2
>
> We used to remove file1 from the working tree in this case.  One
> of the most important reason to use "git reset --hard" is to
> recover from a conflicted, failed merge.  

I think this patch fixes the regression.  Comments?

-- >8 --
read-tree --reset: update working tree file for conflicted paths.

The earlier "git reset --hard" simplification stopped removing
leftover working tree files from a failed automerge, when
switching back to the HEAD version that does not have the
paths.

This patch, instead of removing the unmerged paths from the
index, drops them down to stage#0 but marks them with mode=0
(the same "to be deleted" marker we internally use for paths
deleted by the merge).  one_way_merge() function and the
functions it calls already know what to do with them -- if the
tree we are reading has the path the working tree file is
overwritten, and if it doesn't the working tree file is
removed.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

 builtin-read-tree.c |   19 +++++++++++++++----
 1 files changed, 15 insertions(+), 4 deletions(-)

diff --git a/builtin-read-tree.c b/builtin-read-tree.c
index 716f792..71edaf6 100644
--- a/builtin-read-tree.c
+++ b/builtin-read-tree.c
@@ -753,6 +753,8 @@ static int read_cache_unmerged(void)
 {
 	int i, deleted;
 	struct cache_entry **dst;
+	int unmerged = 0;
+	struct cache_entry *last = NULL;
 
 	read_cache();
 	dst = active_cache;
@@ -760,16 +762,22 @@ static int read_cache_unmerged(void)
 	for (i = 0; i < active_nr; i++) {
 		struct cache_entry *ce = active_cache[i];
 		if (ce_stage(ce)) {
-			deleted++;
+			unmerged++;
+			if (last && !strcmp(ce->name, last->name)) {
+				deleted++;
+				continue;
+			}
 			invalidate_ce_path(ce);
-			continue;
+			last = ce;
+			ce->ce_mode = 0;
+			ce->ce_flags &= ~htons(CE_STAGEMASK);
 		}
 		if (deleted)
 			*dst = ce;
 		dst++;
 	}
 	active_nr -= deleted;
-	return deleted;
+	return unmerged;
 }
 
 static void prime_cache_tree_rec(struct cache_tree *it, struct tree *tree)
@@ -850,7 +858,10 @@ int cmd_read_tree(int argc, const char *
 			continue;
 		}
 
-		/* This differs from "-m" in that we'll silently ignore unmerged entries */
+		/* This differs from "-m" in that we'll silently ignore
+		 * unmerged entries and overwrite working tree files that
+		 * correspond to them.
+		 */
 		if (!strcmp(arg, "--reset")) {
 			if (stage || merge)
 				usage(read_tree_usage);

Re: git reset --hard not removing some files

[Linus Torvalds]

On Sat, 3 Jun 2006, Junio C Hamano wrote:
> 
> I think this patch fixes the regression.  Comments?

The approach looks fine, but with your changes, the "deleted" flag makes 
no sense any more and looks redundant. It's not actually used for anything 
except deciding if we need to re-write the active_cache[] thing, and you 
migth as well use either "last" or "unmerged" for that.

(yeah, "last" and "unmerged" becomes non-null one entry earlier than 
"deleted", but that's actually just confusing, I think, and now depends on 
the fact that we don't have to re-write the first entry).

The whole "*dst++ = ce" _could_ be unconditional, the only reason I made 
it conditional at all is to avoid dirtying the mmap when not necessary (ie 
it's a performance optimization, and it's not one where that off-by-one 
matters).

So I think you could make it clearer by dropping "deleted" entirely, and 
just making the store be conditional on "last".

			Linus

Re: git reset --hard not removing some files

[Martin Waitz]

[-- Attachment #1: Type: text/plain, Size: 411 bytes --]

hoi :)

On Fri, Jun 02, 2006 at 07:57:57AM -0700, Junio C Hamano wrote:
> I would agree in the reproduction recipe Martin gave there is no
> problem but feature, but at the same time I suspect the recent
> "reset --hard simplification" has introduced a true regression.

This may have been the bug that bit me.
Thanks for finding it although I was not able to reproduce it myself!

-- 
Martin Waitz

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: git reset --hard not removing some files

[Junio C Hamano]

Martin Waitz <tali@admingilde.org> writes:

> hoi :)
>
> On Fri, Jun 02, 2006 at 07:57:57AM -0700, Junio C Hamano wrote:
>> I would agree in the reproduction recipe Martin gave there is no
>> problem but feature, but at the same time I suspect the recent
>> "reset --hard simplification" has introduced a true regression.
>
> This may have been the bug that bit me.
> Thanks for finding it although I was not able to reproduce it myself!

I found this somewhat the hard way myself.  I have:

        [pull]
                twohead = resolve

in my .git/config -- IOW, I usually do not use recursive
strategy myself.  When a merge with resolve strategy fails (and
with recent trend to busyboxify git commands, many merges
between my topics and "next" and/or "master" do), I relied on
"reset --hard" followed by the same merge of the topic using
"pull -s recursive" to recover things, but it didn't.  When
pulling a branch with builtin-*.c names into another branch with
older names, regressed "reset --hard" left builtin-*.c files
behind, and then the next merge attempt complained by saying my
untracked working tree files would be overwritten X-<.