Re: Trying to Update All Heads of a Repository

Re: Trying to Update All Heads of a Repository

[Jon Loeliger]

Junio says:
    My "pu" is somewhat special; it is rewound and rebased all the time,
    so merging with its older self would conflict with it.  That's why
    my example remotes/origin file has '+' in front of it.  It tells
    git-fetch that the other side _might_ rebase and fetch would not
    result in a fast-forward merge when that happens.

So, from the git-pull man page:

    For "git push", the local ref that matches <src> is used to fast
    forward the remote ref that matches <dst>. If the optional plus + is
    used, the remote ref is updated even if it does not result in a fast
    forward update.

Ah-ha!  Wait.  But here's the conceptual missing piece: When might I
_know_ I have the situation where a fast-forward update might not
happen?  And as a remote puller, that would be "never" -- unless I know
something about the nature of the remote end.  That is, like you said,
"pu" is subject to wild fluctuation and non-linear behavior.  But any
random puller can't know that a priori.  So far, that is out-of-band
information about a branch that needs to be "available".

I think my previous "Ah ha!" paragraph should be massaged and added to
the git-pull man page as part of the above (man page) quoted material.
That's one.


Junio explained:
>  Alternatively, with your original remotes/origin
> file, you should be able to do:
> 
>	$ git checkout master
>	$ git fetch origin master:origin +pu:pu maint:maint
>       $ git pull . origin

What an excellent example for the git-pull man page!  That's two.



Junio also exampled:

    My "guinea pig" repository has this in $GIT_DIR/remotes/origin:

	    URL: git://git.kernel.org/pub/scm/git/git.git
	    Pull: master:origin
	    Pull: +pu:pu
	    Pull: maint:maint

    This means that my "master" is copied to the "origin" branch of
    the guinea pig repository and "pu" and "maint" are copies of my
    "pu" and "maint" branches.  You never do your own development on
    branches that appear on the right hand side of colon on "Pull"
    lines (i.e. origin, pu and maint) in this repository.  They are
    to be updated by git-fetch.

The notion that multiple pull lines can be placed in the .git/remotes
file should be added to the git-pull man page.  That's three.

Furthermore, there is one very important guideline you just
stated in that paragraph:

    You never do your own development on branches that appear on the
    right hand side of colon on "Pull" lines in a repository.  They are
    to be updated by git-fetch.

And it might bear stating the corollary:

    The reason for a "Pull: master:origin" line is to side-step
    the above maxim and provide a unique place where the local
    developer can do her own work.

I'll consider that concept item four for the git-pull man page . :-)


> Sorry, my "pu" does not fast forward.  The branch is to showcase what
> I've received or picked up from the list, and what changes are under
> consideration for inclusion.

Heh.  I understand it now! :-)  No problem.


And finally:

> The "hold/draw" topic branch (thanks for your ASCII art) is fully
> merged into my "master"; in fact it's head is the master branch head.

That's five-ish.  I'm on deck for some git-pull man page and
drawing documentation patches.  Coming up Real Soon Now.

Thanks for the explanations!

jdl

Trying to Update All Heads of a Repository

[Jon Loeliger]

Guys and Gals,

*sigh*

This is the sort of thing that mystifies me still.
I think there is (should be?) an easy command and
I just missed that day in class.  Feel free to
smack me with the obvious as needed...

I re-cloned (rsync) by git.git tree yesteday.
Today is a new day, and I want to start with
a fresh update of the entire git.git tree.
I have not changed anything in my copy yet.
My goal is a simple update of the entire repository.

    % git --version
    git version 0.99.9.GIT

Uh, foo.  That's a version from 02-Nov-2005.  I guess I could
dig out a SHA1 and we'd know exactly what version it is.

(Request:  Does it make sense to have a hard SHA1 as
           part of git's output version string?)

First, I have this, because it came with a git-clone:

    % cat .git/remotes/origin
    URL: rsync://rsync.kernel.org/pub/scm/git/git.git
    Pull: master:origin

I don't know how to say "Grab all the updates for
all the heads for which I have heads in my .git/refs".

So I hope "git pull --help" will tell me:

    % git pull --help
    usage: git-fetch-pack [-q] [-v] [--exec=upload-pack] [host:]directory <refs>...
    Fetch failure: --help

That would be no.  So I hit the man page, and I can't
quite decipher it.  I'm dense, so I have to re-read it
a few more times until I fully grok it.  Dinner interferes.

Like could I have just said?:

    % git fetch rsync://rsync.kernel.org/pub/scm/git/git.git
or
    % git fetch rsync://rsync.kernel.org/pub/scm/git/git.git master:origin
    % git fetch rsync://rsync.kernel.org/pub/scm/git/git.git pu:pu
    % git fetch rsync://rsync.kernel.org/pub/scm/git/git.git todo:todo

Dunno yet.


In the meantime, I figure I can effectively achieve my
goal by:

    for each head $h
        git checkout -f $h
        git pull rsync://rsync.kernel.org/pub/scm/git/git.git $h:$h

So I start with master.  I know I can cheat here and rely
on the "Pull: master:origin":

    % git pull origin
    receiving file list ... done

    sent 102 bytes  received 12339 bytes  8294.00 bytes/sec
    total size is 4708148  speedup is 378.44
    * committish: a3114b3428595710d4719dd668531210af5993c9
      branch 'master' of rsync://rsync.kernel.org/pub/scm/git/git
    * refs/heads/origin: same as branch 'master' of rsync://rsync.kernel.org/pub/scm/git/git
    Already up-to-date.

That worked fine:

    % git show-branch | head -15
    ! [maint] GIT 0.99.9b
     * [master] Document --since and --until options to rev-parse.
      ! [origin] Document --since and --until options to rev-parse.
       ! [pu] Merge branch 'master'
	! [todo] GIT 0.99.9
    -----
	+ [todo] GIT 0.99.9
	+ [todo^] 2005-10-26 (prepare for 0.99.9)
	+ [todo~2] Update 2005-10-19 afternoon.
	+ [todo~3] 2005-10-18 midnight
	+ [todo~4] 2005-10-16 early afternoon.
	+ [todo~5] 2005-10-16 midnight.
	+ [todo~6] Updates 2005-10-14 night
	+ [todo~7] Updates 2005-10-12 night.

On to the pu branch:

    % git checkout -f pu
    % git show-branch | head -10
    ! [maint] GIT 0.99.9b
     ! [master] Document --since and --until options to rev-parse.
      ! [origin] Document --since and --until options to rev-parse.
       * [pu] Merge branch 'master'
	! [todo] GIT 0.99.9
    -----
	+ [todo] GIT 0.99.9
	+ [todo^] 2005-10-26 (prepare for 0.99.9)
	+ [todo~2] Update 2005-10-19 afternoon.
	+ [todo~3] 2005-10-18 midnight

And pull it now:

    % git pull rsync://rsync.kernel.org/pub/scm/git/git.git pu:pu

    receiving file list ... done

    sent 102 bytes  received 12325 bytes  8284.67 bytes/sec
    total size is 4708148  speedup is 378.86
    * committish: 6c86914344ea64e3a7b287721bdeaa7d41f8a1c3
      branch 'pu' of rsync://rsync.kernel.org/pub/scm/git/git
    * refs/heads/pu: does not fast forward to branch 'pu' of rsync://rsync.kernel.org/pub/scm/git/git;
      not updating.
    Trying really trivial in-index merge...
    fatal: Merge requires file-level merging
    Nope.
    Trying simple merge.
    Simple merge failed, trying Automatic merge.
    Auto-merging Makefile.
    ERROR: git-merge-ours.sh: Not handling case  -> a64704f3e52be43b427792ef9c1bbe42af057c87 -> 4f3d053889de4a5ba8e6e5d519c014a51220accd
    fatal: merge program failed
    Automatic merge failed; fix up by hand

Rats.  But why did this fail?  I've changed nothing in my tree.
I'm pulling remote pu into local pu directly.  I think this
should Just Work...

Uh, now what?

    % git status

    # On branch refs/heads/pu
    #
    # Updated but not checked in:
    #   (will commit)
    #
    #       new file: Documentation/git-http-push.txt
    #       modified: Documentation/git-rev-parse.txt
    #       modified: Makefile
    #       modified: commit.c
    #       modified: config.c
    #       modified: fetch-pack.c
    #       modified: git-branch.sh
    #       modified: git-checkout.sh
    #       unmerged: git-merge-ours.sh
    #       modified: git-push.sh
    #       new file: git-shallow-pack.sh
    #       modified: git-svnimport.perl
    #       new file: http-push.c
    #       modified: ls-files.c
    #       modified: name-rev.c
    #       modified: send-pack.c
    #       modified: server-info.c
    #       modified: sha1_name.c
    #       modified: t/t3001-ls-files-others-exclude.sh
    #       modified: tag.c
    #       modified: tag.h
    #       modified: upload-pack.c
    #
    #
    # Changed but not updated:
    #   (use git-update-index to mark for commit)
    #
    #       unmerged: git-merge-ours.sh
    #
    #
    # Untracked files:
    #   (use "git add" to add to commit)
    #
    #       git-merge-ours.sh

    % git diff
    * Unmerged path git-merge-ours.sh

    % cat .git/HEAD
    0e6a92bd0b33b3461fb7b7f7eb34647ad93d451d

    % cat .git/FETCH_HEAD
    6c86914344ea64e3a7b287721bdeaa7d41f8a1c3                branch 'pu' of rsync://rsync.kernel.org/pub/scm/git/git

So.  Yes, I know I can "git-add git-merge-ours.sh" and then
do a "git-update-index git-merge-ours.sh" and get a spiffy
new index.  And then I can "git-commit" it all, and that
should be everything.

But I really thought that this should Just Work?
Shouldn't this be a fast forward, nice-n-clean-n-easy?

Do I need to add .git/remote entries for each head?
Can I place multiple "Pull: pu:pu" type refspecs into
the one .git/remote/origin?  Is that what people do?
Even if that I do that, I still have to manually iterate:

    for each head $h
        git checkout -f $h
        git pull origin $h

More tequilla.

Thanks,
jdl

Re: Trying to Update All Heads of a Repository

[Linus Torvalds]

On Thu, 3 Nov 2005, Jon Loeliger wrote:
> 
> I don't know how to say "Grab all the updates for
> all the heads for which I have heads in my .git/refs".

Well, you can't per se - there's no guarantee that the heads on the other 
side match, and in fact usually they won't (at the very minimum, the 
remote "master" tends to be the local "origin", although if you never do 
any development at _all_, you can make even that match).

I sent out a patch some time ago that added "--all" and "--tags", but only 
the "--tags" part was accepted. Arguably for good reasons.

But "git fetch" will take a list of heads. So you can do

	git fetch repo master:origin pu todo

and it will update the three local heads (origin, pu and todo) from the 
remote heads master, pu and todo respectively).

IF you have a 100% match between heads names locally and remotely, you 
could also do something like

	git fetch repo $(cd .git/refs ; echo heads/*)

which is pretty cheesy, but should work ;)

As to your experiences with "git pull" - it should work fine too, but you 
really do need to make sure that the branch you are in always matches the 
branch you're pulling. Otherwise you'll be doing a merge between two 
different branches - which is a valid operation, but not what you're 
trying to achieve here.

As we saw earlier, the way to undo an unsuccessful "git pull" is to do a 
"git reset --hard". Undoing a _successful_ git pull is actually slightly 
harder in the general case (it's really undoing a series of commit), but 
if you do it right after the pull, you can do

	git reset --hard ORIG_HEAD

because "git pull" will save the original head before pulling.

			Linus

Re: Trying to Update All Heads of a Repository

[Junio C Hamano]

Jon Loeliger <jdl@freescale.com> writes:

> First, I have this, because it came with a git-clone:
>
>     % cat .git/remotes/origin
>     URL: rsync://rsync.kernel.org/pub/scm/git/git.git
>     Pull: master:origin
>
> I don't know how to say "Grab all the updates for
> all the heads for which I have heads in my .git/refs".

My "guinea pig" repository has this in $GIT_DIR/remotes/origin:

        URL: git://git.kernel.org/pub/scm/git/git.git
        Pull: master:origin
        Pull: +pu:pu
        Pull: maint:maint

This means that my "master" is copied to the "origin" branch of
the guinea pig repository and "pu" and "maint" are copies of my
"pu" and "maint" branches.  You never do your own development on
branches that appear on the right hand side of colon on "Pull"
lines (i.e. origin, pu and maint) in this repository.  They are
to be updated by git-fetch.

And the regular workflow over there is:

	$ cd over-there
        $ git checkout master
        $ git pull
        ... build, test, install

> So I hope "git pull --help" will tell me:

You're right.  We need to add --help to that command.

> Like could I have just said?:
>
>     % git fetch rsync://rsync.kernel.org/pub/scm/git/git.git
> or
>     % git fetch rsync://rsync.kernel.org/pub/scm/git/git.git master:origin
>     % git fetch rsync://rsync.kernel.org/pub/scm/git/git.git pu:pu
>     % git fetch rsync://rsync.kernel.org/pub/scm/git/git.git todo:todo

You could.  Alternatively, with your original remotes/origin
file, you should be able to do:

	$ git checkout master
	$ git fetch origin master:origin +pu:pu maint:maint
        $ git pull . origin

> On to the pu branch:
>
> Rats.  But why did this fail?  I've changed nothing in my tree.
> I'm pulling remote pu into local pu directly.  I think this
> should Just Work...

My "pu" is somewhat special; it is rewound and rebased all the
time, so merging with its older self would conflict with it.
That's why my example remotes/origin file has '+' in front of
it.  It tells git-fetch that the other side _might_ rebase and
fetch would not result in a fast-forward merge when that happens.

I should advertise that fact more often, for people who came
here after I did the last such advertisement.

> But I really thought that this should Just Work?
> Shouldn't this be a fast forward, nice-n-clean-n-easy?

Sorry, my "pu" does not fast forward.  The branch is to showcase
what I've received or picked up from the list, and what changes
are under consideration for inclusion.

The "pu" head is built by pulling a handful topic branches on
top of the "master" branch.  My private repository right now
looks like this:

    ! [hold/draw] Illustration: "Commit DAG Revision Naming"
     ! [hold/http] [PATCH 2/2] Add support for git-http-push to git-pu...
      ! [hold/svn] Several fixes to import mono's svn tree
       ! [jcmu] git-tag: Do not assume the working tree root is writab...
        ! [maint] GIT 0.99.9c
         * [master] Illustration: "Commit DAG Revision Naming"
          ! [pu] Merge branch 'toys'
           ! [toys] git-shallow-pack
    --------
          +  [pu] Merge branch 'toys'
          ++ [toys] git-shallow-pack
          +  [pu^] Merge branches 'hold/svn' and 'hold/http'
     +    +  [hold/http] [PATCH 2/2] Add support for git-http-push to ...
     +    +  [hold/http^] [PATCH 1/2] Add support for pushing to a rem...
      +   +  [hold/svn] Several fixes to import mono's svn tree
       +  +  [jcmu] git-tag: Do not assume the working tree root is wr...
        +    [maint] GIT 0.99.9c
    +  ++++  [hold/draw] Illustration: "Commit DAG Revision Naming"
    +  ++++  [hold/draw^] Illustration: "Git Diff Types"
    +  ++++  [hold/draw~2] Illustration: "Fundamental Git Index Operat...
    +  ++++  [hold/draw~3] git-merge-ours: make sure our index matches...
    +  ++++  [hold/draw~4] Add 'ours' merge strategy.
    +  ++++  [hold/draw~5] Add --no-commit to git-merge/git-pull.
    +  ++++  [hold/draw~6] Document --since and --until options to rev...
    ++++++++ [hold/draw~7] Be careful when dereferencing tags.

The "hold/draw" topic branch (thanks for your ASCII art) is
fully merged into my "master"; in fact it's head is the master
branch head.  Nick's http-push and Yaacov's svnimport fixes are
kept in separate topic branches, which originated at the
previous "master" (now master~7 == hold/draw~7), and they have
not been rebased, partly I am lazy and partly there are no
changes that would cause merge conflicts with other branches.

On the other hand, the commit structure of my public repository
right now looks like this:

    ! [maint] GIT 0.99.9c
     * [master] Illustration: "Commit DAG Revision Naming"
      ! [pu] Merge branch 'toys'
    ---
      + [pu] Merge branch 'toys'
      + [pu^2] git-shallow-pack
      + [pu^] Merge branches 'hold/svn' and 'hold/http'
      + [pu^^3] [PATCH 2/2] Add support for git-http-push to git-push script
      + [pu^^3^] [PATCH 1/2] Add support for pushing to a remote reposito...
      + [pu^^2] Several fixes to import mono's svn tree
      + [pu~2] git-tag: Do not assume the working tree root is writable.
    +   [maint] GIT 0.99.9c
    +++ [master] Illustration: "Commit DAG Revision Naming"

That is, I do not publish the heads of my topic branches, but
you can get them yourselves by looking at the merge logs in the
"pu" branch.  "toys" head is pu^2, "hold/svn" head is pu^^2, etc.

I think this "bundling together into one pu" head policy
somewhat discourages contributor developments on top of "pu"
material, but on the other hand the topic branches in my
repository tend to have relatively short lifespan, and I'd
rather not clutter people's cloned repositories with these
transient branches.

If you wanted to try out Nick's http-push from "hold/http"
branch, for example, but prefer not to worry about bugs in other
things in "pu", you need to do this:

1. Find out if I keep the change as a separate topic branch.
   Either run "gitk --all" and inspect the commit structure
   between "master" and "pu", or look at the "pu" merge log from
   "git-show-branch" output above.

2. Isolate the topic branch from "pu", discarding other stuff
   you are not interested in.  After step 1, you figured out
   that pu^^3 is the tip of hold/http branch, so make a local
   branch for that, and merge the tip of "origin" on top of it,
   because hold/http branch may not have been rebased to the
   current master:

    $ git checkout -b http-push pu^^3
    $ git pull -n . origin
    $ git show-branch origin http-push http-push^
    ! [origin] Illustration: "Commit DAG Revision Naming"
     * [http-push] Merge branch 'master'
      ! [http-push^] [PATCH 2/2] Add support for git-http-push to ...
    ---
     +  [http-push] Merge branch 'master'
    ++  [origin] Illustration: "Commit DAG Revision Naming"
    ++  [origin^] Illustration: "Git Diff Types"
    ++  [origin~2] Illustration: "Fundamental Git Index Operations"
    ++  [origin~3] git-merge-ours: make sure our index matches HEAD
    ++  [origin~4] Add 'ours' merge strategy.
    ++  [origin~5] Add --no-commit to git-merge/git-pull.
    ++  [origin~6] Document --since and --until options to rev-parse.
     ++ [http-push^] [PATCH 2/2] Add support for git-http-push to ...
     ++ [http-push^^] [PATCH 1/2] Add support for pushing to a rem...
    +++ [http-push^~2] Be careful when dereferencing tags.

3. Build, test, install and have fun.

4. Suppose you came up with a neat enhancement idea or two to
   Nick's http-push.  Do your development and make commits on
   top of the current branch.  Your commit structure would look
   like this after a couple of commits:

    $ git show-branch origin http-push
    ! [origin] Illustration: "Commit DAG Revision Naming"
     * [http-push] Add feature nitfol.
    --
     + [http-push] Add feature nitfol.
     + [http-push^] Add feature frotz.
     + [http-push~2] Merge branch 'master'
    ++ [origin] Illustration: "Commit DAG Revision Naming"

5. Then feed me your changes, perhaps CC:ing the original author:

    $ git format-patch -k -m http-push~2
    * 0001-Add-feature-frotz.txt
    * 0002-Add-feature-nitfol.txt

You could publish the head of this branch and tell me to pull
from there, but I have a slight preference to receiving e-mail
patches if the changes are about the material still in "pu".

Re: Trying to Update All Heads of a Repository

[Daniel Barkalow]

On Thu, 3 Nov 2005, Junio C Hamano wrote:

> Jon Loeliger <jdl@freescale.com> writes:
> 
> > First, I have this, because it came with a git-clone:
> >
> >     % cat .git/remotes/origin
> >     URL: rsync://rsync.kernel.org/pub/scm/git/git.git
> >     Pull: master:origin
> >
> > I don't know how to say "Grab all the updates for
> > all the heads for which I have heads in my .git/refs".
> 
> My "guinea pig" repository has this in $GIT_DIR/remotes/origin:
> 
>         URL: git://git.kernel.org/pub/scm/git/git.git
>         Pull: master:origin
>         Pull: +pu:pu
>         Pull: maint:maint

Shouldn't approximately this be what git-clone sets up for you? If it's 
getting those heads initially, it should keep the information for future 
use. The only thing it wouldn't know would be the "+" on pu.

On the "pu" issue, maybe "pu" shouldn't be a branch, but should be 
something different, like a tag. I think, actually, that a tag would 
describe "pu" pretty well: no development happens on its line; rather, 
development happens elsewhere (topic branches), and a merge of some of 
them is then released as being interesting. What is interesting as "pu" 
today isn't necessarily related to yesterday or tomorrow. Then you 
wouldn't exactly be rebasing things all the time, but you'd instead be 
marking different and unrelated commits in the same way at different 
times. (Probably need to do more to handle the case where a tag ref 
changes and define the semantics properly)

	-Daniel
*This .sig left intentionally blank*

Re: Trying to Update All Heads of a Repository

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> Shouldn't approximately this be what git-clone sets up for you? If it's 
> getting those heads initially, it should keep the information for future 
> use. The only thing it wouldn't know would be the "+" on pu.

Yes, and I am actually thinking along the lines of:

         URL: git://git.kernel.org/pub/scm/git/git.git
         Pull: master:origin/master
         Pull: +pu:origin/pu
         Pull: maint:origin/maint

to further prevent the puller's namespace being cluttered.  This
has an added benefit that it gives a convention that
"remotes/origin" file describes the "refs/heads/origin"
hierarchy.  In the past, somebody complained that branches/*
files are configuration per local branch, but remotes/* files
are not.

> On the "pu" issue, maybe "pu" shouldn't be a branch, but should be 
> something different, like a tag. I think, actually, that a tag would 
> describe "pu" pretty well: no development happens on its line; rather, 
> development happens elsewhere (topic branches), and a merge of some of 
> them is then released as being interesting.

Hmph.  That is a thought.

Another thing that bothers me somewhat about git.git/ repository
is the todo branch.  The "todo" branch does not have any common
commit from the mainline and it is deliberately so --- I'd want
to maintain a unified TODO list even after we go 1.0, at that
time I plan to have master that continues post 1.0 development,
maint which corresponds the current maint (i.e. maintenance for
post 1.0 releases) and maint1.0 which is for backporting
important fixes to frozen 1.0.  I could maintain separate
TODO-for-post-1.0 and TODO-for-1.0-maint, but that is rather a
waste --- items on the two list both end up requirign attention
and time from the same brain.  Also merging "todo" into master
and maint1.0 the way "gitk" project was merged into git.git
would not make much sense, because that merge is pretty much
one-way street.

The first thing I clone from git.git/ repository is almost
always "rm -f .git/refs/heads/todo", because it clutters
git-show-branch output otherwise ;-).  In fact, it comes from
totally separate repository on my private end, and just happens
to be pushed into the same git.git/ repository (partly because I
do not have any other repository there at kernel.org; i.e.
/pub/scm/git/ is not writable by me).