Re: git-name-rev off-by-one bug

[linux@horizon.com]

> Thanks for the complaints.  No sarcasm intended.  Yours is
> exactly the kind of message we (people who've been around here
> for too long) need to hear.

Thanks for taking it so well!  I'm trying to really *understand* git,
so I can predict its behaviour, but I've been coming to the conclusion
that the only way to do that is by re-reading the mailing list from day 1.

And to understand git at all, you have to understand merging, since
doing merging fast and well is the central reason for git's entire
existence.  Single-developer porcelain like "cvs annotate" is
noticeably lacking, but branching and merging is great.

(In particular, and unlike other SCMs, "push" and "pull" are
based on merging!  So I can't even understand what pulling from
Linus's tree does until I understand merging.)


I'm working on some notes to explain git to myself and people
I work with, which I'll post when they're vaguely complete.

> To the git barebone Porcelain layer (things that start with
> git-*, not with cg-*) [*2*], a merge is always between the
> current HEAD and one or more remote branch heads, and the index
> file must exactly match the tree of HEAD commit (i.e. the
> contents of the last commit) when it happens.  In other words,
> "git-diff --cached HEAD" must report no changes [*3*].  So
> HEAD+1 must be HEAD in your above notation, or merge will refuse
> to do any harm to your repository (that is, it may fetch the
> objects from remote, and it may even update the local branch
> used to keep track of the remote branch with "git pull remote
> rbranch:lbranch", but your working tree, .git/HEAD pointer and
> index file are left intact).

Right!  Since the object database is strictly append-only, it's easy to
see how such changes are quite harmless, and updating a tracking branch
is hardly a big nasty surprise.

It's the index and working directory that are volatile, and that I was
worried about.

BUT... what's the second argument to git-read-tree for, if it
always has to be HEAD?

BTW, I'd change the description of git-read-tree from

>     Reads the tree information  given  by  <tree-ish>  into  the directory
>     cache, but does not actually update any of the files it "caches". (see:
>     git-checkout-index)

to

      Reads the tree information given by <tree-ish> into the index.
      The working directory is not modified in any way (unless -u
      is used).  Use git-checkout-index to do that.

      In addition to the simple one-tree case, this can (with the
      -m flag) merge 2 or 3 trees into the index.  When used with
      -m, the -u flag causes it to also update the files in the
      working directory.

      Trivial merges are done by "git-read-tree" itself.  Conflicts
      are left in an unmerged state for git-merge-index to resolve.

> You may have local modifications in the working tree files.  In
> other words, "git-diff" is allowed to report changes (the
> difference between HEAD+2 and HEAD+1 in your notation).
> However, the merge uses your working tree as the working area,
> and in order to prevent the merge operation from losing such
> changes, it makes sure that they do not interfere with the
> merge.  Those complex tables in read-tree documentation define
> what it means for a path to "interfere with the merge".  And if
> your local modifications interfere with the merge, again, it
> stops before touching anything.

THANK YOU for finally making this clear!  I was wondering why the
hell a 2-way merge looked more complex than a 3-way.  (Although,
admittedly, I'm *still* not clear on what the difference is.  It
seems like a 2-way just picks the origin commit automatically.)
So it goes like this:

- git-merge will refuse to do anything if there are any uncommitted changes
  in the index.  [footnote about harmless exceptions]
- git-merge will refuse to do anything if there are changes in the
  working directory to a file that would be affected by the merge.
  You CAN, however, have unindexed changes to files that are unchanged
  by the merge.


The description of a 1-way merge in git-read tree is quite confusing.
Here's a rephrasing; do I understand it?

READING

    Without -m, git-read-tree pulls a specified tree into the index.
    Any file whose underlying blob is changed by the update will have
    its cached stat data invalidated, so it will appear in the output
    of git-diff-files, and git-checkout-index -f will overwrite it.

    A single-tree merge is a slightly nicer variant on this.

MERGING
    ...


Single Tree Merge
    If only one tree is specfied, any file whose blob changes as a
    result of the update is compared with the working directory file,
    and the cached stat data updated if the working directory file
    matches the new blob.  Thus, working directory files which happen to
    match the contents of the new tree will be excluded from the outout
    of git-diff-files, and git-checkout-index -f will not update their
    file modification times.

    This is basically the effect of git-update-index --refresh.

    This is actually usually preferable to the non-merge case, but
    does do extra work.

I haven't found the code yet, but obviously if the working directory
file is clean relative to the previous blob, it's dirty relative to
the changed blob, so there's no need to actually read the file.

Um... I just tried it, and it appears that things do NOT work as
I just described.  Time to read the code...


BTW, an even cuter way of writing same() in read_tree.c would be:

static int same(struct cache_entry *a, struct cache_entry *b)
{
	if (!a || !b)
		return a == b;
        return a->ce_mode == b->ce_mode &&
                !memcmp(a->sha1, b->sha1, 20);
}

(I presume that
return a&&b ? a->ce_mode == b->ce_mode && !memcmp(a->sha1, b->sha1, 20) : a==b;
would be taking it too far...)

Oh, and the git-read-tree man page fails to mention --reset and --trivial.
And the usage message should have <sha> changed to <tree>.

> So in the above two "failed merge" case, you do not have to
> worry about lossage of data --- you simply were not ready to do
> a merge, so no merge happened at all.  You may want to finish
> whatever you were in the middle of doing, and retry the same
> pull after you are done and ready.

I feel about a thousand percent better about git already.

> When things cleanly merge, these things happen:
>
> (1) the results are updated both in the index file and in your
>     working tree,
> (2) index file is written out as a tree,
> (3) the tree gets committed, and 
> (4) the HEAD pointer gets advanced.

This is git-merge, as opposed to the more primitive git-read-tree -m
plus git-merge-index, right?

(Aside, you should document that got-merge --no-commit saves the <msg>
in .git/MERGE_MSG, and git-commit uses it as the default message.
Otherwise, users wonder why the hell it asks for a commit message it
knows won't be used.)

> Because of (2), we require that the original state of the index
> file to match exactly the current HEAD commit; otherwise we will
> write out your local changes already registered in your index
> file (the difference between HEAD+1 and HEAD in your notation)
> along with the merge result, which is not good.

> *3* This is a bit of lie.  In certain special cases, your index
> are allowed to be different from the tree of HEAD commit;
> basically your index entries are allowed to match the result of
> trivial merge already (e.g. you received the same patch from
> external source to produce the same result as what you are
> merging).  For example, if a path did not exist in the common
> ancestor and your head commit but exists in the tree you are
> merging into your repository, and if you already happen to have
> that path exactly in your index, the merge does not have to
> fail.  This is case #2 in the 3-way read-tree table in t/t1000.

Ah, yes, the light dawns!  So it *is* okay if you have an uncommitted
change in the index which exactly matches what git-read-tree -m would
have done anyway.  It doesn't make a difference to the final state
of the index, and forcing the poor user to undo it just so the merge
can redo it is simply annoying.

But it's NOT okay if you have any other change in the index, even in a
file unaffected by the merge, because then the final state would not
be the result of the merge, and that could be confusing.

> Because (1)
> involves only the paths different between your branch and the
> remote branch you are pulling from during the merge (which is
> typically a fraction of the whole tree), you can have local
> modifications in your working tree as long as they do not
> overlap with what the merge updates.

Wonderfully clear.  Such changes must not exist at the git-read-tree
level, since there's not even a way to represent them to
git-merge-index.  Still, this can save considerable bother when
trying to track someone else's tree.

> When there are conflicts, these things happen:
>
>  (0) HEAD stays the same.
>
>  (1) Cleanly merged paths are updated both in the index file and
>      in your working tree.
> 
>  (2) For conflicting paths, the index file records the version
>      from HEAD. The working tree files have the result of
>      "merge" program; i.e. 3-way merge result with familiar
>      conflict markers <<< === >>>.
> 
>  (3) No other changes are done.  In particular, the local
>      modifications you had before you started merge will stay the
>      same and the index entries for them stay as they were,
>      i.e. matching HEAD.

So this is a lot like what CVS does, but neater because all the
merge sources are available unmodified in the index.  Excellent.

> After seeing a conflict, you can do two things:
>
> * Decide not to merge.  The only clean-up you need are to reset
>   the index file to the HEAD commit to reverse (1) and to clean
>   up working tree changes made by (1) and (2); "git-reset" can
>   be used for this.

Cool.  Two minor questions:

- Doesn't any non-trivial merge or invocation of git-update-index
  produce blob objects in the database that become garbage if you
  do this?  Or are they somehow kept separate until a tree object
  is created to point to them?

  (You could have an "unreferenced" bit in the index, indicating that
  the blobs in question could be found in .git/pending rather than
  .git/objects, until git-write-tree moved them into the database.
  But I don't see mention of any such scheme.)

- Is there any difference between "git-reset --hard" and "git-checkout -f"?

> * Resolve the conflicts.  "git-diff" would report only the
>   conflicting paths because of the above (1) and (2).  Edit the
>   working tree files into a desirable shape, git-update-index
>   them, to make the index file contain what the merge result
>   should be, and run "git-commit" to commit the result.

Okay, so git-update-index will overwrite a staged file with a
fresh stage-0 copy.  And git-commit will refuse to commit
(to be precise, it'll stop at the git-write-tree stage) if there
are unresolved conflicts.

If you want to see the unmodified input files, you can find their
IDs with "git-ls-files -u" and then get a copy with "git-cat-file blob"
or "git-unpack-file".  git-merge-index is basically a different way to
process the output of git-ls-files -u.


> *1* It is a shame that the most comprehensive definition of
> 3-way read-tree semantics is in t/t1000-read-tree-m-3way.sh test
> script.

Thanks for the pointer; I'll go and read it!

> *2* Cogito (things that start with cg-*) seems to try to be
> cleverer.  Pasky might want to brag about the rules in Cogito
> land.

In fact, he might want to explain what the difference is between cogito
and git.  Most particularly, are there any restrictions on mixing cg-*
and git-* operations from within the same directory?

I've been assuming that cogito is just series of friendlier
utilities, in much the same way that a text editor is friendlier
than "cat > kernel/sched.c", so I'll study it after I understand
core git.


One more question to be sure I understand merging... AFAICT, it would
be theoretically possible to stop distinguishing "stage 2" and "stage 0".
If there is only a stage 2 file, then in-index will immediately
collapse it to stage 0, and if there are any other stage files,
you know there's an incomplete merge.

(Alternatively, you could collapse "stage 3" and "stage 0", since
stages 2 and 3 are treated identically, but traditionally, stage 2 is
the "trunk" and state 3 is the "branch" being merged in.)