Re: Untracked working tree files

[Andrew Morton <akpm@linux-foundation.org>]

On Wed, 15 Oct 2008 13:08:36 -0700 (PDT)
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> 
> 
> On Wed, 15 Oct 2008, Andrew Morton wrote:
> > 
> > I treat my git directory as a read-only thing.  I only ever modify it
> > with git commands.
> 
> Ok. 
> 
> > > (By the filename, I realize it's a file that doesn't exist in one tree or 
> > > the other, and which doesn't get removed at some point. But have you had 
> > > merge failures, for example? Is it perhaps a file that was created during 
> > > a non-clean merge, and then got left behind due to the merge being 
> > > aborted? It would be interesting to know what led up to this..)
> > 
> > That's certainly a possibility - I get a lot of merge failures.  A real
> > lot.  And then quite a bit of rebasing goes on, especially in
> > linux-next.  And then there's all the other stuff which Stephen does on
> > top of the underlying trees to get something releasable happening.
> 
> Is "git checkout -f" part of the scripting? Or "git reset --hard"?

well, this script has been hacked on so many times I'm not sure what
it does any more.

Presently the main generate-a-diff function is

doit()
{
	tree=$1
	upstream=$2

	cd $GIT_TREE
	git checkout "$upstream"
	git reset --hard "$upstream"
	git fetch "$tree" || exit 1
	git merge --no-commit 'test merge' HEAD FETCH_HEAD > /dev/null

	{
		git_header "$tree"
		git log --no-merges ORIG_HEAD..FETCH_HEAD
		git diff --patch-with-stat ORIG_HEAD
	} >$PULL/$tree.patch
	{
		echo DESC
		echo $tree.patch
		echo EDESC
		git_header "$tree"
		git log --no-merges ORIG_HEAD..FETCH_HEAD
	} >$PULL/$tree.txt
	git reset --hard "$upstream"
}

usually invoked as

doit origin v2.6.27
doit origin linux-next

etc.

the above seemed fairly busted, so I'm now using

        git checkout -f "$upstream"
        git reset --hard "$upstream"
        git fetch "$tree" || exit 1

which seems a bit more sensible.  Perhaps I should do the reset before
the checkout, dunno.

That function has been through sooooooo many revisions and each time
some scenario get fixed (more like "improved"), some other scenario
gets busted (more like "worsened").  The above sorta mostly works,
although it presently generates thirty-odd rejects against
git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-tip.git#auto-latest,
which is way above my fix-it-manually threshold.  linux-next is still
dead because it's taking Stephen over two days to fix the mess he's
been fed so I'm madly rebasing everything on mainline over here.


> So what I could imagine is happening is:
> 
>  - you have a lot of automated merging
> 
>  - a merge goes south with a data conflict, and since it's all automated, 
>    you just want to throw it away. So you do "git reset --force" to do 
>    that.

didn't know about --force.

>  - but what "git reset --hard" means is to basically ignore all error 
>    cases, including any unmerged entries that it just basically ignores.
> 
>  - so it did set the tree back, but the whole point of "--hard" is that it 
>    ignores error cases, and doesn't really touch them.
> 
> Now, I don't think we ever really deeply thought about what the error 
> cases should do when they are ignored. Should the file that is in some 
> state we don't like be removed? Or should we just ignore the error and 
> return without removing the file? Generally git tries to avoid touching 
> things it doesn't understand, but I do think this may explain some pain 
> for you, and it may not be the right thing in this case.

Yeah, there's no easy solution here, and I suspect the real solution is
"read programmer's mind".  Providing a reliable override (like -f) is a
sensible solution.

> (And when I say "this case", I don't really know whether you use "git 
> checkout -f" or "git reset --hard" or something else, so I'm not even 
> going to say I'm sure exactly _which_ case "this case" actually us :)
> 
> Of course, the cheesy way for you to fix this may be to just add a
> 
> 	git clean -dqfx
> 
> to directly after whatever point where you decide to reset and revert to 
> an earlier stage. That just says "force remove all files I don't know 
> about, including any I might ignore". IOW, "git reset --hard" will 
> guarantee that all _tracked_ files are reset, but if you worry about some 
> other crud that could have happened due to a failed merge, that additional 
> "git clean" may be called for.

OK, I'll try git clean -dqfx if it blows up again.

> Of course, it's going to read the whole directory tree and that's not 
> really cheap, but especially if you only do this for error cases, it's 
> probably not going to be any worse. And I'm assuming you're not compiling 
> in that tree, so you probably don't want to save object files (you can 
> remove the "x" part, but then you could still at least in theory get a 
> filename clash with something that is ignored and thus didn't get cleaned 
> up).