Multi-head fetches, pulls, and a King Ghidorah

Multi-head fetches, pulls, and a King Ghidorah

[Junio C Hamano]

The current "pu" branch has most of the necessary plumbing for
multi-head fetching, pulling and creating Octopus merges based
on multiple heads.  I have made "git pull" multi-head aware but
did not make it multi-head capable.

I have been trying out Tony Luck's excellent topic-branch
workflow (see Documentation/using-topic-branches.txt), and the
head of the "pu" branch is an Octopus on top of the "master" and
my two private topic branches [*1*] made with "git octopus".

I have been thinking about how the multi-head fetch and merge
should interact with each other.  It appears to me that coming
up with a reasonably semantics for "git pull" that would fit
everybody is very difficult, because there are at least two
different valid workflows that involve multi-head pulling, and
they need to do different things.  I'll talk about "git pull"
that can deal with more than one remote heads, which does not
exist yet, in the next couple of paragraphs.

Suppose I were the Emperor Penguin [*2*], and Jeff asks "Please
pull from my netdev-2.6.git repository, sis190 and e100
branches."  It is very reasonable to expect the following to
fetch heads of those two branches and attempt to make an Octopus
[*3*] on top of a copy of the Penguin head:

    $ cat .git/branches/jgarzik
    kernel.org:/pub/linux/kernel/git/jgarzik/
    $ git checkout -b try-jeffs-update master
    $ git pull jgarzik/netdev-2.6.git/ sis190 e100
    $ git diff -p master..HEAD | git apply --stat --summary

The arguments to pull, especially the refspec parameters
("sis190" and "e100" in the above example) are given to the
underlying "git fetch", and because neither of them have a
colon, they are not stored in corresponding places under the
local refs/heads/.  Instead, fetched heads would be fed directly
to "git octopus" inside "git pull" (just like the current "git
pull" invokes "git resolve" internally after fetching just one
head).

However, suppose then I were Joe Random, an individual netdev
contributor who is interested in these two netdev branches.
Upon seeing the pull request, I might decide it is a good time
to get changes from there, my upstream.

    $ git pull jgarzik/netdev-2.6.git/ sis190:sis190 e100:e100

Because I am keeping track of copies of these two branches, I
use "sis190:sis190 e100:e100" to update my local heads.

I might have some local changes in these two branches.  The old
"git fetch" unconditionally overwrote local heads when told to,
but lately it acquired the "reverse push" semantics Johannes
Schindelin proposed to make it safer.  The fetch process only
overwrites local heads when the upstream change results in a
fast-forward merge; practically, that happens only when I have
not worked on that branch since I pulled from the upstream the
last time.  So my refs/heads/sis190 and refs/heads/e100 may be
copies of Jeff's heads, or they may be the same heads as I had
before starting the fetch.

At this point, it might be reasonable to expect that the above
"git pull" command would behave as if I pulled (i.e. fetched and
merged) these heads separately, using traditional single-head
pull:

    $ git checkout sis190
    $ git pull jgarzik/netdev-2.6.git/ sis190:sis190
    $ git checkout e100
    $ git pull jgarzik/netdev-2.6.git/ e100:e100

That is, fetch and resolve them independently and individually.

Back in the Emperor Penguin example, he _could_ also have been
interested in keeping copies of Jeff's branch heads, so he could
have written refspecs on the command line the same way as Joe
Random did.  I.e. instead of:

    $ git pull jgarzik/netdev-2.6.git/ sis190 e100

he could have said:

    $ git pull jgarzik/netdev-2.6.git/ sis190:sis190 e100:e100

Both "Octopus" and "Multiple independent pull" semantics are
valid to support different workflows, and there is no way to
differenciate the two from the command line, without giving an
extra flag and making the implementation more complicated.

Currently, I am inclined to leave the current "not more than one
remote head" implementation, and possibly extend it to support
the "Octopus" semantics later, for three very simple reasons.

 (1) What the latter "git pull" is buying us compared to two
     traditional single-head pulls is very little; that the
     underlying "git fetch" _could_ obtain packs more
     efficiently than two independent fetches.

 (2) The netdev example happened to involve multiple heads from
     a single repository, but the pull request could as well
     have been "jgarzik/netdev-2.6.git#sis190 and
     jgarzik/libata-dev.git#sil24", in which case I wouldn't
     have to be worrying about multi-head pull at all; the user
     would just have used two independent traditional "git
     pull" --- there is no other option.

 (3) Or course, because I am lazy ;-).  Seriously, multiple
     independent merges is a nightmere when you start thinking
     about what to do when you get a conflict and need to have
     the user hand merge in the middle of the first one.

Since "git pull" tentatively would not do multi-head natively,
the Emperor Penguin example needs to be done this way:

    $ git checkout -b try-jeffs-update master
    $ git fetch jgarzik/netdev-2.6.git/ sis190 e100
    $ git octopus
    $ git diff -p master..HEAD | git apply --stat --summary

I think this should already work with the current "pu" branch
head.


[Footnotes]

*1* I should probably write a bit about how I do things in a
separate message as a how-to.

*2* I am not a penguin.  Figuring out what kind of animal I am
is left as an easter-egg hunt.  I have had the answer somewhere
in the current git.git archive for some time ;-).

*3* This one has only three heads, so that would be a Tripus,
but what X-pus counts is not heads but legs, so we should really
be calling this a King Ghidorah who has three heads.

*4* What's currently in "pu" branch dies when a head cannot be
fast forwarded, but I think that is simply a thinko.  It should
just refuse to fast-forward and warn, leave the obtained remote
head somewhere other tools can find later, just as the original
"reverse push" script by Johannes did.

Re: Multi-head fetches, pulls, and a King Ghidorah

[Johannes Schindelin]

Hi,

If I understand correctly, the multi-head fetch would not write any ref if 
used like this:

	git fetch remote:repository/ head tail

but it would try a fast-forward when used like this:

	git fetch remote:repository/ head:head tail:tail

Correct? If yes: This is fantastic! It obsoletes my dumb script.

On Fri, 19 Aug 2005, Junio C Hamano wrote:

> However, suppose then I were Joe Random, an individual netdev
> contributor who is interested in these two netdev branches.
> Upon seeing the pull request, I might decide it is a good time
> to get changes from there, my upstream.
> 
>     $ git pull jgarzik/netdev-2.6.git/ sis190:sis190 e100:e100
> 
> Because I am keeping track of copies of these two branches, I
> use "sis190:sis190 e100:e100" to update my local heads.

I propose a "--separate" flag to git pull. This would do exactly the same 
as a plain git pull, but for each fetched branch which could not be 
fast-forwarded

	- try to switch to the branch (dying if it is not the current,
	  and the working tree is dirty)

	- try a merge

	- if the merge fails, reset the branch to original state,
	  write out a temporary head and output a warning

After that, it would switch back to the original branch and check that 
out.

For all failed merges, the user needs to "git resolve" (the exact command 
line could be output by "git pull --separate").

Ciao,
Dscho

Re: Multi-head fetches, pulls, and a King Ghidorah

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Hi,
>
> If I understand correctly, the multi-head fetch would not write any ref if 
> used like this:
>
> 	git fetch remote:repository/ head tail
>
> but it would try a fast-forward when used like this:
>
> 	git fetch remote:repository/ head:head tail:tail
>
> Correct? If yes: This is fantastic! It obsoletes my dumb script.

That is the intent, but what is currently in "pu" is somewhat
broken.

My GIT Day

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> *1* I should probably write a bit about how I do things in a
> separate message as a how-to.

So here it is.

Note that the version of git on my $PATH is usually the one from
the proposed updates branch, so some of the commands I use in
the following text may not work for you unless you also have
built "pu" one yourself.

I am planning to finish updating, testing and documenting what's
in the current proposed updates branch, and have most of them
graduate to the master branch over the weekend.  I am aiming for
doing the 0.99.5 on Wednesday next week.

------------

I have the following heads all the time in my private
repository:

    master	- the one to be pushed to public "master" branch
    pu		- master plus various proposed changes
    rc		- master plus minimum release engineering

    ko-master	- a copy of public "master" branch head
    ko-rc	- a copy of public "rc" branch head

My GIT day always starts with this command:

    $ git fetch ko

I have this in .git/remotes/ko:

    $ cat .git/remotes/ko
    URL: master.kernel.org:/pub/scm/git/git.git/
    Pull: master:ko-master rc:ko-rc
    Push: master pu rc

The Pull: line gives me the default set of <refspec>s to give to
the "git fetch" command.  I slurp "master" and "rc" heads from
the public repository at master.kernel.org, and fast-forward my
ko-master and ko-rc branches with them.  I do not touch these
two branches in any way other than this "git fetch" fast
forwarding.

I have a few "topic branches" in addition to the above; they
change from time to time.  Recently I've been looking at
multi-head fetches, and that work is done in "mhf" branch.
There also is a catch-all topic called "misc".  I started them
like this:

    $ git branch mhf master
    $ git branch misc master

The first thing I do during my GIT day is to process the patches
I received via e-mail.  I store them one topic per file in my
working tree, like this:

    $ ls +*.txt
    +js-glossary.txt  +mc-mailinfo.txt

Depending on the quality of the patch, seriousness of the bug
they fix, and the area of the code they touch, they either go
directly to "master", "misc", or sent back to the sender, but
the last one, luckily for me, rarely happens:

    $ git checkout master
    $ git applymbox -q ./+js-glossary.txt .git/info/signoff

    $ git checkout misc
    $ git applymbox -q ./+mc-mailinfo.txt please

The last parameter to the applymbox command is the name of a
file that has my signoff message.  The latest applymbox in the
"pu" branch has a bit more useful extension to do the same thing
as what "git commit" does.

At this point, I may push out the "master" branch (and nothing
else), like this:

    $ git push ko master

This pushes only "master", ignoring the default <refspec>s
defined in the .git/remotes/ko file you saw earlier.

Once I am done with the outside patches, I go back to where I
left off the previous day:

    $ git checkout mhf

And I check where my head is relative to the master:

    $ git show-branches master mhf
    ! [master] Yet another tweak
     * [mhf] Make git-fetch-script a bit more chatty.
    +  Yet another tweak
    +  Another tweak in Makefile
     + Make git-fetch-script a bit more chatty.
     + Update git-ls-remote-script
     + ...
     + Start adding the $GIT_DIR/remotes/ support.
    ++ [PATCH] Allow file removal when "git commit --all" is used.

The output from show-branches is a poor-man's gitk.  The named
branches are shown, and '+' sign in each column shows whether
the commit is contained in each branch, and the output stops
where all branches converge, or you hit ^C ;-).

If the mhf branch is way behind, I may choose to first rebase
it, to clean up my history:

    $ git rebase master
    $ git show-branches master mhf
    ! [master] Yet another tweak
     * [mhf] Make git-fetch-script a bit more chatty.
     + Make git-fetch-script a bit more chatty.
     + Update git-ls-remote-script
     + ...
     + Start adding the $GIT_DIR/remotes/ support.
    ++ Yet another tweak

I keep working in my topic branches.  I may make some other
changes in "misc" topic branch.  It's a simple cycle of:

    $ edit-and-test
    $ git commit -s -a -v

Eventually I get to a good point where it makes sense to push
things to the public repository.

    $ git show-branches master mhf misc
    ! [master] Yet another tweak
     * [mhf] Make git-fetch-script a bit more chatty.
      ! [misc] Add hooks to tools/git-applypatch.
     +  Make git-fetch-script a bit more chatty.
     +  Update git-ls-remote-script
     +  ...
     +  Start adding the $GIT_DIR/remotes/ support.
    ++  Yet another tweak
    ++  Another tweak in Makefile
      + Add hooks to tools/git-applypatch.
      + Add commit hook and make the verification customizable.
    +++ [PATCH] Allow file removal when "git commit --all" is used.

As you may have noticed, my topic branches are private and not
pushed to the public repository.  Instead, I make a grand total
merge of them into "pu".  The proposed update branch is always
rewound and made from the head of the master:

    $ git checkout pu
    $ git reset master

This checks out the head of "pu" branch, and then resets the
index file to match "master" and updates .git/refs/heads/pu.

What it does _not_ do is to update my working tree to match the
index file.  Linus recommends to do "git checkout -f" at this
point, but I typically do this instead:

    $ git diff -R -p | git apply

Note.  This is an embarrassingly expensive way; the only thing
it buys me over "git checkout -f" is that it removes the files
that were introduced in "pu" branch that did not exist in the
"master" head.  I have to come up with a not so expensive way to
do this.

Then before doing the grand total merge, check again where those
heads are:

    $ git show-branches master mhf misc pu 
    ! [master] Yet another tweak
     ! [mhf] Make git-fetch-script a bit more chatty.
      ! [misc] Add hooks to tools/git-applypatch.
       * [pu] Yet another tweak
     +   Make git-fetch-script a bit more chatty.
     +   Update git-ls-remote-script
     +   ...
     +   Start adding the $GIT_DIR/remotes/ support.
    ++ + Yet another tweak
    ++ + Another tweak in Makefile
      +  Add hooks to tools/git-applypatch.
      +  Add commit hook and make the verification customizable.
    ++++ [PATCH] Allow file removal when "git commit --all" is used.

Notice that I did not rebase "misc" above, but that is OK.  What
I want to do here is to make new "pu" an Octopus over "master",
merging in all my topic branches (currently "mhf" and "misc").

    $ git fetch . mhf misc
    Packing 0 objects
    Unpacking 0 objects
    * committish: a101f32...e5580	refs/heads/mhf from .
    * committish: 4426ac7...0c5bc	refs/heads/misc from .

This fetches two heads from the current repository (!).  The
only reason I do it is that the tentative implementation of "git
octopus" always reads from $GIT_DIR/FETCH_HEAD, and "git fetch"
is the way to populate that file.

    $ git octopus
    Removing git-parse-remote
    Removing git-parse-remote
    Committed octopus merge fe1899156bffa4be6722b2ca0b74ff17b76523da
     Makefile                        |    3 -
     git-commit-script               |   75 +++++-------------
     ...
     tools/git-applypatch            |   87 ++++++++++++++++----
     15 files changed, 551 insertions(+), 224 deletions(-)

This makes an Octopus out of the master and other two topic
branches.  I can make sure that resulting "pu" contains all the
necessary commits from the branches involved:

    $ git show-branches master mhf misc pu
    ! [master] Yet another tweak
     ! [mhf] Make git-fetch-script a bit more chatty.
      ! [misc] Add hooks to tools/git-applypatch.
       * [pu] Octopus merge of the following: 
       + Octopus merge of the following:
     + + Make git-fetch-script a bit more chatty.
     + + Update git-ls-remote-script
     + + ...
     + + Start adding the $GIT_DIR/remotes/ support.
    ++ + Yet another tweak
    ++ + Another tweak in Makefile
      ++ Add hooks to tools/git-applypatch.
      ++ Add commit hook and make the verification customizable.
    ++++ [PATCH] Allow file removal when "git commit --all" is used.

And things are now ready to be pushed out.  First I sanity check
the differences between ko-master and master (the earlier "git
fetch ko" was done only for this step):

    $ git show-branches master ko-master

... and then run "git push":

    $ git push ko

This will push "master" and "rc" but would fail to push "pu",
because that is rebased and not based on what is on the public
repository.  So I push once more, this time with --force, like
this:

    $ git push --force ko pu

This pushes only "pu", ignoring the default <refspec>s defined
in the .git/remotes/ko file you saw earlier.  After that, I go
back to reading the mail and wait until the kernel.org mirror
network catches up.

Re: My GIT Day

[Johannes Schindelin]

Hi,

On Fri, 19 Aug 2005, Junio C Hamano wrote:

>     $ git checkout pu
>     $ git reset master
> 
> This checks out the head of "pu" branch, and then resets the
> index file to match "master" and updates .git/refs/heads/pu.
> 
> What it does _not_ do is to update my working tree to match the
> index file.  Linus recommends to do "git checkout -f" at this
> point, but I typically do this instead:
> 
>     $ git diff -R -p | git apply
> 
> Note.  This is an embarrassingly expensive way; the only thing
> it buys me over "git checkout -f" is that it removes the files
> that were introduced in "pu" branch that did not exist in the
> "master" head.  I have to come up with a not so expensive way to
> do this.

How about this?

  $ git reset master
  $ rm -i $(git-diff-tree ORIG_HEAD HEAD | grep "^.\{97\}D" | cut -c 100-)
  $ git checkout -f

Ciao,
Dscho