FETCH_HEAD question

FETCH_HEAD question

[Jay Soffian]

I did this the other day out of mild curiosity:

$ git fetch
$ git merge FETCH_HEAD

Which did something, but not something that was at all useful. It
merges in the first ref listed in FETCH_HEAD. It does not appear to be
an accident that it does this, as git merge has special treatment for
FETCH_HEAD to generate the merge message.

Why does this behavior exist? Historical?

j.

Re: FETCH_HEAD question

[Jay Soffian]

On Mon, Feb 16, 2009 at 11:43 PM, Jay Soffian <jaysoffian@gmail.com> wrote:
> I did this the other day out of mild curiosity:
>
> $ git fetch
> $ git merge FETCH_HEAD
>
> Which did something, but not something that was at all useful. It
> merges in the first ref listed in FETCH_HEAD. It does not appear to be
> an accident that it does this, as git merge has special treatment for
> FETCH_HEAD to generate the merge message.
>
> Why does this behavior exist? Historical?

To be clear, this seems only to be useful if you are only fetching a
single branch from remote. Otherwise the branch which you end up
merging (the first alphabetically) may very well not be what the
current checked-out branch is based on.

So to work correctly in the case of pulling multiple branches,
dwim_ref() would have to return the sha1 corresponding to the single
line in FETCH_HEAD that is not marked "not-for-merge" and git merge
would similarly need to use that line for the merge message. Or git
fetch would need to place the non-not-for-merge ref first in the file.

I found this in the archives, but it didn't really answer my question
about why it got implemented the current way:

http://thread.gmane.org/gmane.comp.version-control.git/42788/focus=42850

j.

Re: FETCH_HEAD question

[Junio C Hamano]

Jay Soffian <jaysoffian@gmail.com> writes:

> I did this the other day out of mild curiosity:
>
> $ git fetch
> $ git merge FETCH_HEAD
>
> Which did something, but not something that was at all useful. It
> merges in the first ref listed in FETCH_HEAD. It does not appear to be
> an accident that it does this, as git merge has special treatment for
> FETCH_HEAD to generate the merge message.

"git pull" internally invokes "git fetch" to learn the tips of branches
the invoking user is interested in.  "git fetch" downloads the necessary
objects to complete the ancestry chain from local refs to these tips of
branches, and then uses FETCH_HEAD to communicate which commit object
these branches point at, and at the same time which lines in the file
describe the commits that need to be merged into the current HEAD by
marking lines with not-for-merge markers ("cat .git/FETCH_HEAD" to see how
it is done), back to the invoking "git pull".  "git pull" then inspects
this file to decide which commit object to pass to "git merge".

This division of labor makes sense because "git fetch" knows more about
the user preference ("remotes.*.fetch" and "branch.*.merge" configuration
items, .git/remotes/* files, and .git/branches/* files are inspected by
the command).  

The rule initially was that the first non-wildcard Pull: rule in remotes/*
file specifies the commits to be merged or something, and over time, the
system learned the more elaborate per-branch settings via branch.*.merge
variables, but the idea has always been that you cannot just grab the
first line of FETCH_HEAD and expect that is what you want to merge.  At
least you need to omit the ones that are marked not-for-merge (see how it
is done by "git-pull.sh" with a simple sed script), even though
"FETCH_HEAD" can be used as an extended SHA-1 to name the commit object
recorded on the first line in the file.

Having said that, being able to directly use FETCH_HEAD as an extended
SHA-1 is often useful when somebody asks an integrator to pull one-shot
with git-request-pull.  Upon receiving such a request, the integrator can
say:

	$ git fetch git://repo.or.cz/his.git for-linus
        $ git log -p ..FETCH_HEAD ;# to inspect
        $ git merge FETCH_HEAD

to take advantage of the fact that the commit object name that appears at
the beginning of resulting FETCH_HEAD file is actually the commit we would
want, because by definition FETCH_HEAD records only one commit when we are
fetching only one branch.

Incidentally, such a command line to explicitly fetch one single branch is
interpreted by "git fetch" to mean that you are interested in merging the
tip of the fetched branch, and the first line of the resulting FETCH_HEAD
file is not marked with not-for-merge.  Therefore,

	$ git pull git://repo.or.cz/his.git for-linus

would also work regardless of what you have in .git/config file.

Re: FETCH_HEAD question

[Jay Soffian]

On Tue, Feb 17, 2009 at 12:36 AM, Junio C Hamano <gitster@pobox.com> wrote:
> [... explanation of how git pull and git fetch communicate via FETCH_HEAD...]

I'm aware of all that, and I apologize for not making that clear,
since I made you do lots of extra typing. :-(

> Having said that, being able to directly use FETCH_HEAD as an extended
> SHA-1 is often useful when somebody asks an integrator to pull one-shot
> with git-request-pull.  Upon receiving such a request, the integrator can
> say:
>
>        $ git fetch git://repo.or.cz/his.git for-linus
>        $ git log -p ..FETCH_HEAD ;# to inspect
>        $ git merge FETCH_HEAD
>
> to take advantage of the fact that the commit object name that appears at
> the beginning of resulting FETCH_HEAD file is actually the commit we would
> want, because by definition FETCH_HEAD records only one commit when we are
> fetching only one branch.

That makes sense, but I was confused why git merge goes through the
trouble of stripping out the not-for-merge tag which in the above use
case wouldn't be there.

j.

Re: FETCH_HEAD question

[Junio C Hamano]

Jay Soffian <jaysoffian@gmail.com> writes:

> On Tue, Feb 17, 2009 at 12:36 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> [... explanation of how git pull and git fetch communicate via FETCH_HEAD...]
>
> I'm aware of all that, and I apologize for not making that clear,
> since I made you do lots of extra typing. :-(

Your "git merge FETCH_HEAD" after fetching random set of refs by the
default wildcard refspec in .git/config made me suspect you aren't.

>>        $ git fetch git://repo.or.cz/his.git for-linus
>>        $ git log -p ..FETCH_HEAD ;# to inspect
>>        $ git merge FETCH_HEAD
>> ...
> That makes sense, but I was confused why git merge goes through the
> trouble of stripping out the not-for-merge tag which in the above use
> case wouldn't be there.

Because it is designed to handle a lot more general case of fetching all
remote branches into your remote tracking branches with wildcard refspecs,
and most of the entries need to be marked with not-for-merge marker.  If
you left only the for-merge branches, you would not have any sane way to
learn what refs were fetched after you said "git fetch" (and no, looking
at timestamp of files in .git/refs/remotes/origin/ is not a sensible
answer).

Re: FETCH_HEAD question

[Jay Soffian]

On Tue, Feb 17, 2009 at 2:18 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Because it is designed to handle a lot more general case of fetching all
> remote branches into your remote tracking branches with wildcard refspecs,
> and most of the entries need to be marked with not-for-merge marker.  If
> you left only the for-merge branches, you would not have any sane way to
> learn what refs were fetched after you said "git fetch" (and no, looking
> at timestamp of files in .git/refs/remotes/origin/ is not a sensible
> answer).

We are talking past each other.

I understand the not-for-merge marker, why git fetch puts it there,
and how it is used by git pull.

I was asking why "git merge FETCH_HEAD" is useful, and you gave an
example of fetching a *single* branch using "git fetch <url>
<branch>". But in that case, as you said, git fetch does *not* leave a
"not-for-merge" marker, since only one branch was fetched.

So then, why does "git merge FETCH_HEAD" bother to strip out a
not-for-merge marker that wouldn't be there?

I can only guess there is a use case where one fetches multiple
branches and then uses "git merge FETCH_HEAD".

j.

Re: FETCH_HEAD question

[Junio C Hamano]

Jay Soffian <jaysoffian@gmail.com> writes:

> So then, why does "git merge FETCH_HEAD" bother to strip out a
> not-for-merge marker that wouldn't be there?

In general, FETCH_HEAD is usable as an extended SHA-1, in that any file
that is in .git/ and begins with [0-9a-f]{40} names that object.  That is
how "git log -p ..FETCH_HEAD" in my example worked.

The reason stripping of not-for-merge is done is to support a usage that
is not so sane as well.

You can do a wildcard fetch that leaves full of not-for-merge entries in
FETCH_HEAD, and then use FETCH_HEAD as an extended SHA-1 to merge the one
that happens to be on the first line.  "git merge" *could* in such a case
produce:

	Merge commit $ugly_40_hexadecimal_name

to say that it merged that commit named by a random extended SHA-1, but it
knows that FETCH_HEAD has better information than that.  The line tells
where you got that commit from.  And that is recorded past not-for-merge
marker (if exists, otherwise it is an empty string) that is enclosed by a
HT on each side.  This is a direct translation from the scripted version
that did:

	sed -e 's/	not-for-merge	/		/' -e 1q

It is being overly nice to produce a sensible merge message, even when you
asked it to do something not very sensible.

Now to something totally useless.

After reading the builtin-merge.c and original git-merge.sh (now in
contrib/examples) script, I think it could have done something entirely
different.

It could have done this instead.

	sed -e '/	not-for-merge	/d'

to learn the commits and their human-readable origins, and it could have
tried to reproduce what "git pull" did when it invoked git-merge using
that information.  Then you could use this workflow:

	$ git pull <possibly with arguments>
        ... oops, conflicted and is very messy.
        ... I tried to resolve, but failed and made the mess even worse.
        ... Let's start over.
        $ git reset --hard
        ... FETCH_HEAD knows which refs are for merging
        $ git merge FETCH_HEAD

That is, no matter what the arguments were for the initial "git pull",
what should be merged is recorded in FETCH_HEAD, and that is how you can
retry the merge without refetching over the network.

But such a change makes FETCH_HEAD different from what it traditionally
meant, and does that only to "git merge", making the result very
inconsistent.  For example, "git log ..FETCH_HEAD" will still use the
object name on the first line, and it won't be a way to convince yourself
that the changes are sensible and it is Ok to run "git merge FETCH_HEAD"
anymore.  So I do not think such a change will be an improvement.

Re: FETCH_HEAD question

[Jay Soffian]

On Tue, Feb 17, 2009 at 3:35 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Now to something totally useless.
>
> After reading the builtin-merge.c and original git-merge.sh (now in
> contrib/examples) script, I think it could have done something entirely
> different.
>
> It could have done this instead.
>
>        sed -e '/       not-for-merge   /d'
>
> to learn the commits and their human-readable origins, and it could have
> tried to reproduce what "git pull" did when it invoked git-merge using
> that information.  Then you could use this workflow:
>
>        $ git pull <possibly with arguments>
>        ... oops, conflicted and is very messy.
>        ... I tried to resolve, but failed and made the mess even worse.
>        ... Let's start over.
>        $ git reset --hard
>        ... FETCH_HEAD knows which refs are for merging
>        $ git merge FETCH_HEAD
>
> That is, no matter what the arguments were for the initial "git pull",
> what should be merged is recorded in FETCH_HEAD, and that is how you can
> retry the merge without refetching over the network.
>
> But such a change makes FETCH_HEAD different from what it traditionally
> meant, and does that only to "git merge", making the result very
> inconsistent.  For example, "git log ..FETCH_HEAD" will still use the
> object name on the first line, and it won't be a way to convince yourself
> that the changes are sensible and it is Ok to run "git merge FETCH_HEAD"
> anymore.  So I do not think such a change will be an improvement.

Unless dwim_ref() is updated to handle FETCH_HEAD specially, and
return not the first SHA1, but the one not marked "not-for-merge".
Then the UI would at least be consistent, but this would not be
backward compatible.

j.

Re: FETCH_HEAD question

[Johannes Schindelin]

Hi,

On Tue, 17 Feb 2009, Jay Soffian wrote:

> On Tue, Feb 17, 2009 at 3:35 AM, Junio C Hamano <gitster@pobox.com> wrote:
> > Now to something totally useless.
> >
> > After reading the builtin-merge.c and original git-merge.sh (now in
> > contrib/examples) script, I think it could have done something entirely
> > different.
> >
> > It could have done this instead.
> >
> >        sed -e '/       not-for-merge   /d'
> >
> > to learn the commits and their human-readable origins, and it could have
> > tried to reproduce what "git pull" did when it invoked git-merge using
> > that information.  Then you could use this workflow:
> >
> >        $ git pull <possibly with arguments>
> >        ... oops, conflicted and is very messy.
> >        ... I tried to resolve, but failed and made the mess even worse.
> >        ... Let's start over.
> >        $ git reset --hard
> >        ... FETCH_HEAD knows which refs are for merging
> >        $ git merge FETCH_HEAD
> >
> > That is, no matter what the arguments were for the initial "git pull",
> > what should be merged is recorded in FETCH_HEAD, and that is how you can
> > retry the merge without refetching over the network.
> >
> > But such a change makes FETCH_HEAD different from what it traditionally
> > meant, and does that only to "git merge", making the result very
> > inconsistent.  For example, "git log ..FETCH_HEAD" will still use the
> > object name on the first line, and it won't be a way to convince yourself
> > that the changes are sensible and it is Ok to run "git merge FETCH_HEAD"
> > anymore.  So I do not think such a change will be an improvement.
> 
> Unless dwim_ref() is updated to handle FETCH_HEAD specially, and
> return not the first SHA1, but the one not marked "not-for-merge".
> Then the UI would at least be consistent, but this would not be
> backward compatible.

You cannot fix parsing FETCH_HEAD as a ref (and neither will you be able 
to do with PUSH_HEAD), as it can contain _more_ than one SHA-1s.  This 
still holds true when ignoring the not-for-merge lines, as an octopus is
a quite real possibility.

Ciao,
Dscho

Re: FETCH_HEAD question

[Jay Soffian]

On Tue, Feb 17, 2009 at 12:22 PM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>> Unless dwim_ref() is updated to handle FETCH_HEAD specially, and
>> return not the first SHA1, but the one not marked "not-for-merge".
>> Then the UI would at least be consistent, but this would not be
>> backward compatible.
>
> You cannot fix parsing FETCH_HEAD as a ref (and neither will you be able
> to do with PUSH_HEAD), as it can contain _more_ than one SHA-1s.  This
> still holds true when ignoring the not-for-merge lines, as an octopus is
> a quite real possibility.

An even better argument against than it not being backward compatible. :-)

And there is no PUSH_HEAD.

j.

PUSH_HEAD, was Re: FETCH_HEAD question

[Johannes Schindelin]

Hi,

On Tue, 17 Feb 2009, Jay Soffian wrote:

> On Tue, Feb 17, 2009 at 12:22 PM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> >> Unless dwim_ref() is updated to handle FETCH_HEAD specially, and
> >> return not the first SHA1, but the one not marked "not-for-merge".
> >> Then the UI would at least be consistent, but this would not be
> >> backward compatible.
> >
> > You cannot fix parsing FETCH_HEAD as a ref (and neither will you be able
> > to do with PUSH_HEAD), as it can contain _more_ than one SHA-1s.  This
> > still holds true when ignoring the not-for-merge lines, as an octopus is
> > a quite real possibility.
> 
> An even better argument against than it not being backward compatible. :-)
> 
> And there is no PUSH_HEAD.

Heh, that's why I used the future tense :-)

Ciao,
Dscho

Re: PUSH_HEAD, was Re: FETCH_HEAD question

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> On Tue, 17 Feb 2009, Jay Soffian wrote:
>
>> On Tue, Feb 17, 2009 at 12:22 PM, Johannes Schindelin
>> <Johannes.Schindelin@gmx.de> wrote:
>> >> Unless dwim_ref() is updated to handle FETCH_HEAD specially, and
>> >> return not the first SHA1, but the one not marked "not-for-merge".
>> >> Then the UI would at least be consistent, but this would not be
>> >> backward compatible.
>> >
>> > You cannot fix parsing FETCH_HEAD as a ref (and neither will you be able
>> > to do with PUSH_HEAD), as it can contain _more_ than one SHA-1s.  This
>> > still holds true when ignoring the not-for-merge lines, as an octopus is
>> > a quite real possibility.
>> 
>> An even better argument against than it not being backward compatible. :-)
>> 
>> And there is no PUSH_HEAD.
>
> Heh, that's why I used the future tense :-)

FETCH_HEAD is something the user who is sitting in front of the repository
controls with "git fetch" into it.  The objects referenced from it is not
part of the "reachable" set, and expectation is that anything that
requires the objects will happen before that user decides to actively go
prune unreachable objects.

PUSH_HEAD is quite different, and it needs to be handled a lot more
carefully.

We have been looking at denyCurrentBranch as a way to capture gremlin
updates and prevent it from confusing the end user, but PUSH_HEAD is more
like a pull request in reverse.  The necessary object transfer part has
been already done, and the tip commit waits for the final merge step to
pick it up.  You might want a loud notification that tells you that there
are outstanding reverse pull requests waiting to be picked up.  Its need
to protect the objects pushed from pruning is stronger than that of
FETCH_HEAD.

When PUSH_HEAD is in use for a receiving repository, we would intercept
pushes to certain refs, and instead of updating them, append a record in
the PUSH_HEAD file.  Each entry of the file would record the object name
and who pushed it from where and for which branch.  A new front-end
command to deal with PUSH_HEAD would allow the user to:

 * browse them,

 * choose one of them and comply, by checking out the destination branch
   and performing the merge,

 * choose one of them and discard, or

 * leave undecided.

Earlier I've considered introducing the notion of (per-ref) split head.
Forgetting the presense of packed-refs for now, when a gremlin push tries
to update a ref that is the current branch, you would instead of
overwriting the ref (that is 40-byte hexadecimal SHA-1 plus LF), append it
(making it contain two such lines).  Bash completion and friends can
notice the situation and say "Your HEAD is split by a gremlin push", and
the the user can choose to "git merge" the two (or more) commits recorded
in the ref to get the split head merged again (optionally, if the work
tree and the index is clean, and if the object the push is trying to
overwrite the HEAD with is a descendant of the HEAD, then you can allow it
to overwrite and run "reset --hard" afterwards, instead of splitting the
head).  I think Hg does something like that, and I think PUSH_HEAD is
nicer than outright "you are not allowed to push into this branch",
because the users already understand how a pull in response to a pull
request works as a workflow.

Having said that, there need to be clear and sane semantics defined to
tell which push should go to PUSH_HEAD file and which should directly go
to the underlying ref.  It is probably the easiest to explain if all
pushes to local refs are redirected to PUSH_HEAD when the repository is
non-bare, but it means a non-bare repository cannot be used as the central
shared repository for everybody to meet and swap with push-pull.  We could
only redirect push to the checked-out branch to PUSH_HEAD, but then that
would introduce an inconsistency between checked-out branches and others
from fetchers' point of view; pushes to some branches are observable from
the outside while pushes to one special branch (the checked-out one) are
not.

Re: PUSH_HEAD, was Re: FETCH_HEAD question

[Jay Soffian]

On Tue, Feb 17, 2009 at 6:43 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Having said that, there need to be clear and sane semantics defined to
> tell which push should go to PUSH_HEAD file and which should directly go
> to the underlying ref.  It is probably the easiest to explain if all
> pushes to local refs are redirected to PUSH_HEAD when the repository is
> non-bare, but it means a non-bare repository cannot be used as the central
> shared repository for everybody to meet and swap with push-pull.  We could
> only redirect push to the checked-out branch to PUSH_HEAD, but then that
> would introduce an inconsistency between checked-out branches and others
> from fetchers' point of view; pushes to some branches are observable from
> the outside while pushes to one special branch (the checked-out one) are
> not.

This is along the lines of what I was thinking. However, I have also
been thinking of an alternative. It would go something like this:

$ git init --hybrid

This would basically setup a repository that is a combination of bare
and non-bare and would act like having a pair or repositories. i.e.
Given,

$ mkdir project
$ cd project
$ git init
$ touch file && git add file && git commit -m initial
$ git clone --bare file://$(pwd) ../project-public.git
$ git remote add public ../project-public.git
$ git config branch.master.remote public
$ git config branch.master.merge refs/heads/master

I don't mean to suggest that git init --hybrid would literally create
two repositories. Rather, it would create a single repo that acts like
the above two. You'd "git push" to publish a branch and "git pull" to
integrate any pushed-in changes. refs/heads in such a repo would never
receive outside changes directly, nor would changes to refs/heads be
externally visible until published via push.

This would typically be used in the mothership/satellite scenario,
where the mothership repo is setup with --hybrid and the satellite is
just a normal clone.

But maybe this is a lot of work w/little reward.

j.

Re: PUSH_HEAD, was Re: FETCH_HEAD question

[Johannes Schindelin]

Hi,

On Tue, 17 Feb 2009, Junio C Hamano wrote:

> FETCH_HEAD is something the user who is sitting in front of the 
> repository controls with "git fetch" into it.  The objects referenced 
> from it is not part of the "reachable" set, and expectation is that 
> anything that requires the objects will happen before that user decides 
> to actively go prune unreachable objects.
> 
> PUSH_HEAD is quite different, and it needs to be handled a lot more 
> carefully.

Well, when I talked about the push equivalent of FETCH_HEAD I really meant 
that it be exactly that.  No additional taking care, that's the user's 
responsibility.

I got carried away with the "2nd push will not need to push the objects 
again", and in the meantime I think that should not be the case.

The main use for it I had in mind was to have some obviously temporary 
"ref" that you could push to.

However, I think that would not bode well, as receive.denyCurrentBranch 
will error out early, and rightfully so.

The way out I _could_ think of right now is to have a REFUSED_PUSH_HEAD 
_just_ for the denyCurrentBranch case, but it's bedtime for lil' Dscho, so 
I cannot think clearly about that today.

Ciao,
Dscho "who will be dreaming of detached, split and pushed heads all night long"

Re: FETCH_HEAD question

[Sitaram Chamarty]

On 2009-02-17, Jay Soffian <jaysoffian@gmail.com> wrote:
> On Tue, Feb 17, 2009 at 12:36 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> [... explanation of how git pull and git fetch communicate via FETCH_HEAD...]
>
> I'm aware of all that, and I apologize for not making that clear,
> since I made you do lots of extra typing. :-(

oy -- ease up on that apology; I'm thankful he did --
cleared up a bunch of stuff for me!

Junio: I hope you put it on your blog.  These detailed,
descriptive, explanations should all be in one place for
easy reference.