Question regarding git fetch

Question regarding git fetch

[Tom Lambda]

I noticed that git-fetch seems smarter when it is run without a <refspec>
argument than when one specifies a branch name. I use a simple setup where a
remote central repository is defined when it is cloned the first time (clone
-o central ...). This leads to these default parameters:

remote.central.fetch=+refs/heads/*:refs/remotes/central/*
branch.master.remote=central
branch.master.merge=refs/heads/master

When I use "git fetch central" each branch in central's refs/heads/ is
automatically fetched to my refs/remotes/central/ as expected.

What was a little bit surprising to me is that running "git fetch central
master" does not update refs/remotes/central/master but simply updates
FETCH_HEAD.

I read the manual and I know that updating FETCH_HEAD is the expected
default behavior for git-fetch. However, I really like the fact that git
fetch (without <refspec>) knows that ANY branch in refs/heads/ corresponds
to refs/remotes/central/. Is there a way to change the configuration to have
"git fetch central branch" always updating refs/remotes/central/branch
whatever the specified branch.

I would prefer not to have to specify the <dst> each time:
git fetch central branch:remotes/central/branch

Thank you,
Tom

-- 
View this message in context: http://n2.nabble.com/Question-regarding-git-fetch-tp3527289p3527289.html
Sent from the git mailing list archive at Nabble.com.

Re: Question regarding git fetch

[Avery Pennarun]

On Thu, Aug 27, 2009 at 3:30 PM, Tom Lambda<tom.lambda@gmail.com> wrote:
> What was a little bit surprising to me is that running "git fetch central
> master" does not update refs/remotes/central/master but simply updates
> FETCH_HEAD.

I've often wanted this myself, especially when doing things like "git
pull origin master".  However, I know the current behaviour is also
useful sometimes, and changing it would introduce an unexpected side
effect.  Git currently promises that your refs/remotes/* branches will
never be updated unless you explicitly request it, even if you're
fetching, merging, and pulling other stuff.  This means you can write
scripts to do complicated things without triggering unexpected
user-visible side effects.

So basically, I agree that it would often be much more user-friendly
to do what you're asking.  But it would be less scripting-friendly.  I
don't think anyone has thought of an answer that better balances the
two.

Have fun,

Avery

Re: Question regarding git fetch

[Eric Raible]

Avery Pennarun <apenwarr <at> gmail.com> writes:

> Git currently promises that your refs/remotes/* branches will
> never be updated unless you explicitly request it, even if you're
> fetching, merging, and pulling other stuff.

So your claim is that "git fetch central" is somehow more
explicit than "git fetch central master"?

I understand that "git fetch central" will use
remote.central.fetch (which _is_ explicit), but
the command itself is certainly _less_ explicit about
specifying the remote branch.

Either way, AFAICT it seems purely historical that
"git fetch central master" doesn't update remotes/central.

- Eric

Re: Question regarding git fetch

[Avery Pennarun]

On Thu, Aug 27, 2009 at 4:21 PM, Eric Raible<raible@gmail.com> wrote:
> Avery Pennarun <apenwarr <at> gmail.com> writes:
>
>> Git currently promises that your refs/remotes/* branches will
>> never be updated unless you explicitly request it, even if you're
>> fetching, merging, and pulling other stuff.
>
> So your claim is that "git fetch central" is somehow more
> explicit than "git fetch central master"?

It is, because there's nothing else you could possibly do with it.  It
fetches too many things, so you don't know their names (other than the
ones it puts into refs/remotes/*).

So it's not exactly that it's more explicit - in fact it's rather
confusing and non-explicit that the three-parameter version does
something almost entirely different from the four-parameter one - but
the three-parameter version is at least completely unambiguous.

> Either way, AFAICT it seems purely historical that
> "git fetch central master" doesn't update remotes/central.

This seems true to me.

Have fun,

Avery

Re: Question regarding git fetch

[Björn Steinbrink]

On 2009.08.27 15:36:53 +0000, Avery Pennarun wrote:
> On Thu, Aug 27, 2009 at 3:30 PM, Tom Lambda<tom.lambda@gmail.com> wrote:
> > What was a little bit surprising to me is that running "git fetch central
> > master" does not update refs/remotes/central/master but simply updates
> > FETCH_HEAD.
> 
> I've often wanted this myself, especially when doing things like "git
> pull origin master".  However, I know the current behaviour is also
> useful sometimes, and changing it would introduce an unexpected side
> effect.  Git currently promises that your refs/remotes/* branches will
> never be updated unless you explicitly request it, even if you're
> fetching, merging, and pulling other stuff.  This means you can write
> scripts to do complicated things without triggering unexpected
> user-visible side effects.
> 
> So basically, I agree that it would often be much more user-friendly
> to do what you're asking.  But it would be less scripting-friendly.  I
> don't think anyone has thought of an answer that better balances the
> two.

It would also be pretty hard to implement that. Given the default fetch
refspec, it would "simply" be a matter of mapping the given ref to the
refspec, so e.g. "foo" becomes "refs/heads/foo:refs/remotes/origin/foo".
But even just using "git remote add -t master foo git://..." breaks
that, as the fetch refspec in the config will no longer be a glob, and
thus no such mapping is possible.

Björn

Re: Question regarding git fetch

[Avery Pennarun]

2009/8/27 Björn Steinbrink <B.Steinbrink@gmx.de>:
> It would also be pretty hard to implement that. Given the default fetch
> refspec, it would "simply" be a matter of mapping the given ref to the
> refspec, so e.g. "foo" becomes "refs/heads/foo:refs/remotes/origin/foo".
> But even just using "git remote add -t master foo git://..." breaks
> that, as the fetch refspec in the config will no longer be a glob, and
> thus no such mapping is possible.

Hmm, I don't really see why that introduces a problem.  If you use -t
to override explicitly which refs you want to save, then it's not a
problem if git doesn't save other refs, right?

I'd be more concerned about the inconsistency between

   git fetch git://whatever master
and
   git fetch origin master

There's no really good way for the first one to know it needs to
update any branches, even though 'origin' might be an alias for
git://whatever.  So users will still be confused.

Thinking of that also reminds me of another surprise.  If you do:

   git fetch git://whatever

...it seems to do nothing at all, as far as I can see.  Which makes
sense, I guess, since I wouldn't really expect it to be meaningful.
But it seems to connect up to the remote server anyway just in case.

I suppose I have no useful suggestions here, other than an
interpretation of why users find the current behaviour confusing.

Have fun,

Avery

Re: Question regarding git fetch

[Jeff King]

On Thu, Aug 27, 2009 at 05:22:55PM +0000, Avery Pennarun wrote:

> 2009/8/27 Björn Steinbrink <B.Steinbrink@gmx.de>:
> > It would also be pretty hard to implement that. Given the default fetch
> > refspec, it would "simply" be a matter of mapping the given ref to the
> > refspec, so e.g. "foo" becomes "refs/heads/foo:refs/remotes/origin/foo".
> > But even just using "git remote add -t master foo git://..." breaks
> > that, as the fetch refspec in the config will no longer be a glob, and
> > thus no such mapping is possible.
> 
> Hmm, I don't really see why that introduces a problem.  If you use -t
> to override explicitly which refs you want to save, then it's not a
> problem if git doesn't save other refs, right?

I think you can handle both cases just by matching the fetched refs to
the LHS of the refspec.

So if you fetch "refs/heads/foo", and if you have a refspec of:

  refs/heads/*:refs/remotes/origin/*

then you see that the LHS glob matches, and the RHS expands to
refs/remotes/origin/foo. And if you have a more restrictive refspec,
that would work, too:

  refs/heads/foo:refs/remotes/origin/foo

would still match, but

  refs/heads/bar:refs/remotes/origin/bar

does not match on the LHS, so you write nothing. It would even handle
multiple refspecs properly.

And this matching is not really any different than what the fetch code
does when applying the refspec to what the remote offers. So I don't
think it should be any significant new code; it's just a matter of
activating that matching and updating the local tracking refs based on
what we actually fetched, instead of what the remote advertised.

> I'd be more concerned about the inconsistency between
> 
>    git fetch git://whatever master
> and
>    git fetch origin master
> 
> There's no really good way for the first one to know it needs to
> update any branches, even though 'origin' might be an alias for
> git://whatever.  So users will still be confused.

Well, you can always reverse-lookup each remote to see if the URL
matches. Of course you would never know that "http://whatever" is an
alias for "git://whatever". Personally, I don't imagine that users
really expect git to reverse-map remotes in that way (after all, why
would they input git://whatever long-hand if they knew that it was a
remote).

> Thinking of that also reminds me of another surprise.  If you do:
> 
>    git fetch git://whatever
> 
> ...it seems to do nothing at all, as far as I can see.  Which makes
> sense, I guess, since I wouldn't really expect it to be meaningful.
> But it seems to connect up to the remote server anyway just in case.

Shouldn't it fetch HEAD from the remote and store it in FETCH_HEAD? And
then tell you that it did that?

I get:

  $ mkdir foo && cd foo && git init
  Initialized empty Git repository in /home/peff/foo/.git/

  $ git fetch ~/compile/git
  remote: Counting objects: 85011, done.
  remote: Compressing objects: 100% (23942/23942), done.
  remote: Total 85011 (delta 61389), reused 83076 (delta 59655)
  Receiving objects: 100% (85011/85011), 19.03 MiB | 11385 KiB/s, done.
  Resolving deltas: 100% (61389/61389), done.
  From /home/peff/compile/git
   * branch            HEAD       -> FETCH_HEAD

  $ git fetch git://git.kernel.org/pub/scm/git/git.git
  From git://git.kernel.org/pub/scm/git/git
   * branch            HEAD       -> FETCH_HEAD

-Peff

Re: Question regarding git fetch

[Jeff King]

On Thu, Aug 27, 2009 at 04:48:35PM -0400, Jeff King wrote:

> And this matching is not really any different than what the fetch code
> does when applying the refspec to what the remote offers. So I don't
> think it should be any significant new code; it's just a matter of
> activating that matching and updating the local tracking refs based on
> what we actually fetched, instead of what the remote advertised.

Sure enough, here is a really simple proof of concept. This is the first
time I have really looked into the fetch code, so I hope I'm not totally
breaking something else. :)

Basically it works like this. Both "git fetch remote" and "git fetch
remote refspec" work by making a "ref map" via get_ref_map. This is a
list of refs, with some refs having a "peer ref" that points to the
local counterpart. Those without a peer are just stored in FETCH_HEAD.
The function works by starting with a list of possible remote refs, and
then applying the refspecs to it.

Calling fetch without explicit refspecs means we will create the map
using the configured refspecs. Calling with commandline refspecs means
we will use those. So what this patch does is to first apply the
commandline refspecs to narrow the list, and then apply the configured
refspecs on top of that to make a new list, and then concatenate the
lists.

So you will end up with "two" refs to fetch, which just happen to match
the same source. One will go to FETCH_HEAD, and one will go to the
tracking ref. I.e.:

  $ git remote add origin ~/compile/git
  $ git fetch origin next
  From /home/peff/compile/git
   * branch            next       -> FETCH_HEAD
   * [new branch]      next       -> origin/next

And it should work the same if you supply a more interesting refspec,
too:

  $ git fetch -v origin next:foo
  From /home/peff/compile/git
   * [new branch]      next       -> foo
   = [up to date]      next       -> origin/next

or even a wildcard:

  $ git fetch -v origin refs/heads/*:refs/foo/*
  From /home/peff/compile/git
   * [new branch]      master     -> refs/foo/master
   * [new branch]      next       -> refs/foo/next
   * [new branch]      master     -> origin/master
   = [up to date]      next       -> origin/next

I haven't thought long enough to be convinced there aren't any bad side
effects, though. I guess if you had non-traditional configured refspecs,
you might be surprised to see them applied in this case. Something like:

  # we usually fetch the remote's master straight into our production
  # branch for deployment
  $ git config remote.origin.fetch refs/heads/master:refs/heads/production

  # but today let's demo it first
  $ git fetch origin master:demo

I don't know if people are actually using refspecs in that way. But we
do need to consider that this is a potentially destructive change.

Anyway, here is the patch.

---
diff --git a/builtin-fetch.c b/builtin-fetch.c
index 817dd6b..d9e44c7 100644
--- a/builtin-fetch.c
+++ b/builtin-fetch.c
@@ -119,6 +119,9 @@ static struct ref *get_ref_map(struct transport *transport,
 	const struct ref *remote_refs = transport_get_remote_refs(transport);
 
 	if (ref_count || tags == TAGS_SET) {
+		struct ref *tracking_refs = NULL;
+		struct ref **tracking_tail = &tracking_refs;
+
 		for (i = 0; i < ref_count; i++) {
 			get_fetch_map(remote_refs, &refs[i], &tail, 0);
 			if (refs[i].dst && refs[i].dst[0])
@@ -129,6 +132,12 @@ static struct ref *get_ref_map(struct transport *transport,
 			rm->merge = 1;
 		if (tags == TAGS_SET)
 			get_fetch_map(remote_refs, tag_refspec, &tail, 0);
+
+		for (i = 0; i < transport->remote->fetch_refspec_nr; i++)
+			get_fetch_map(ref_map, &transport->remote->fetch[i],
+					&tracking_tail, 0);
+		*tail = tracking_refs;
+		tail = tracking_tail;
 	} else {
 		/* Use the defaults */
 		struct remote *remote = transport->remote;

Re: Question regarding git fetch

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

>   # we usually fetch the remote's master straight into our production
>   # branch for deployment
>   $ git config remote.origin.fetch refs/heads/master:refs/heads/production
>
>   # but today let's demo it first
>   $ git fetch origin master:demo

I think this is a good example that any change results from this
discussion should apply _only_ to cases where command line refspecs lack
colon (i.e. used to mean "do not store this anywhere but in FETCH_HEAD").

Re: Question regarding git fetch

[Jeff King]

On Thu, Aug 27, 2009 at 02:44:54PM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> >   # we usually fetch the remote's master straight into our production
> >   # branch for deployment
> >   $ git config remote.origin.fetch refs/heads/master:refs/heads/production
> >
> >   # but today let's demo it first
> >   $ git fetch origin master:demo
> 
> I think this is a good example that any change results from this
> discussion should apply _only_ to cases where command line refspecs lack
> colon (i.e. used to mean "do not store this anywhere but in FETCH_HEAD").

I don't think the colon is the issue. Consider the same situation, but I
say:

  # but today let's demo it first
  $ git fetch origin master
  $ git checkout -b demo FETCH_HEAD

I'm still screwed. The issue is that you consider your configured
refspec destinations to be precious, and not merely a cache for what's
happening on the remote side.

-Peff

Re: Question regarding git fetch

[Jeff King]

On Thu, Aug 27, 2009 at 05:50:07PM -0400, Jeff King wrote:

> > I think this is a good example that any change results from this
> > discussion should apply _only_ to cases where command line refspecs lack
> > colon (i.e. used to mean "do not store this anywhere but in FETCH_HEAD").
> 
> I don't think the colon is the issue. Consider the same situation, but I
> say:
> 
>   # but today let's demo it first
>   $ git fetch origin master
>   $ git checkout -b demo FETCH_HEAD
> 
> I'm still screwed. The issue is that you consider your configured
> refspec destinations to be precious, and not merely a cache for what's
> happening on the remote side.

Which, btw, led me to consider whether there are heuristics for deciding
when a fetch refspec means one thing and not the other. I don't think
there are reliable ones (probably the default configured
refs/remotes/$remotename/* would not yield false positives, but I think
limiting to that would yield false negatives). So maybe this is
something that should be configurable, disabled by default for now, and
maybe enabled by default in the future (v1.7.0?).

-Peff

Re: Question regarding git fetch

[Avery Pennarun]

On Thu, Aug 27, 2009 at 9:50 PM, Jeff King<peff@peff.net> wrote:
> I don't think the colon is the issue. Consider the same situation, but I
> say:
>
>  # but today let's demo it first
>  $ git fetch origin master
>  $ git checkout -b demo FETCH_HEAD
>
> I'm still screwed. The issue is that you consider your configured
> refspec destinations to be precious, and not merely a cache for what's
> happening on the remote side.

Is the "precious remote ref" concept perhaps an imaginary one?

After all, if I *really* care about the prior state of the remote, I
can just make it a remote branch.  And if (as often happens) I just
want to know what's new in that ref since last time I merged, it's
simply

   git log master..origin/master

This works even if master has extra commits vs. origin/master, since
the double-dot invokes git-merge-base.

I think this might be a much more common than the case where people
actually want to see "what's changed since last time I checked what's
changed."  At least, the latter question has never been very
interesting to me, or if it is, it's easy for me to tell by eye.

Avery

Re: Question regarding git fetch

[Jeff King]

On Thu, Aug 27, 2009 at 10:12:50PM +0000, Avery Pennarun wrote:

> > I'm still screwed. The issue is that you consider your configured
> > refspec destinations to be precious, and not merely a cache for what's
> > happening on the remote side.
> 
> Is the "precious remote ref" concept perhaps an imaginary one?

Maybe. I certainly don't use it. But I am trying to consider corner
cases where somebody who _isn't_ me is going to get screwed by a
change we make.

> After all, if I *really* care about the prior state of the remote, I
> can just make it a remote branch.  And if (as often happens) I just

Do you mean "local branch" here?

> want to know what's new in that ref since last time I merged, it's
> simply
> 
>    git log master..origin/master
> 
> This works even if master has extra commits vs. origin/master, since
> the double-dot invokes git-merge-base.

Well, ".." doesn't use git-merge-base. But yes, I actually do this,
except I do:

  gitk master...origin/master

-Peff

Re: Question regarding git fetch

[Avery Pennarun]

On Thu, Aug 27, 2009 at 10:16 PM, Jeff King<peff@peff.net> wrote:
> On Thu, Aug 27, 2009 at 10:12:50PM +0000, Avery Pennarun wrote:
>> After all, if I *really* care about the prior state of the remote, I
>> can just make it a remote branch.  And if (as often happens) I just
>
> Do you mean "local branch" here?

Yes.

>> want to know what's new in that ref since last time I merged, it's
>> simply
>>
>>    git log master..origin/master
>>
>> This works even if master has extra commits vs. origin/master, since
>> the double-dot invokes git-merge-base.
>
> Well, ".." doesn't use git-merge-base. But yes, I actually do this,
> except I do:
>
>  gitk master...origin/master

Sorry, not my day for accuracy, it seems.  I use both, and I should
have remembered that the triple-dot is the one that worries about the
merge-base.

Have fun,

Avery

Re: Question regarding git fetch

[Junio C Hamano]

Avery Pennarun <apenwarr@gmail.com> writes:

> On Thu, Aug 27, 2009 at 3:30 PM, Tom Lambda<tom.lambda@gmail.com> wrote:
>> What was a little bit surprising to me is that running "git fetch central
>> master" does not update refs/remotes/central/master but simply updates
>> FETCH_HEAD.
>
> I've often wanted this myself, especially when doing things like "git
> pull origin master".  However, I know the current behaviour is also
> useful sometimes, and changing it would introduce an unexpected side
> effect.  Git currently promises that your refs/remotes/* branches will
> never be updated unless you explicitly request it, even if you're
> fetching, merging, and pulling other stuff.  This means you can write
> scripts to do complicated things without triggering unexpected
> user-visible side effects.

I think it is reasonable, in 1.7.0, to change "git fetch" with command
line refspacs that lack colon (i.e. saying "fetch this ref" without saying
"and store it here") so that it updates remote tracking refs if and only
if an appropriate remote.$remote.fetch is configured to do so.  E.g. when
I fetch from Eric for git-svn updates with

	$ git pull git-svn master

because I do have

	[remote "git-svn"]
                url = git://yhbt.net/git-svn
                fetch = +refs/heads/*:refs/remotes/git-svn/*

defined, it is Ok to update refs/remotes/git-svn/master (but not others).

On the other hand, if my refspecs for "git svn" _were_ like this:

	[remote "git-svn"]
		url = git://yhbt.net/git-svn
                fetch = +refs/heads/master:refs/remotes/git-svn/master

then I would _not_ want this:

	$ git fetch git-svn dev

to create a new tracking branch refs/remotes/git-svn/dev.

It used to be that the only way to check the progress of other people
were to do this:

	$ git fetch git-svn master
	$ git log git-svn/master..FETCH_HEAD

But these days, even if we changed the "git fetch" semantics, we can still
rely on reflogs to do the equivalent with:

	$ git fetch git-svn
	$ git log git-svn/master@{1}..git-svn/master

In other words, I think the "feature" that an explicit "fetch but do not
store this time" request to prevent "git fetch" from updating the tracking
branches outlived its usefulness.

Re: Question regarding git fetch

[Jeff King]

On Thu, Aug 27, 2009 at 02:20:46PM -0700, Junio C Hamano wrote:

> I think it is reasonable, in 1.7.0, to change "git fetch" with command
> line refspacs that lack colon (i.e. saying "fetch this ref" without saying
> "and store it here") so that it updates remote tracking refs if and only
> if an appropriate remote.$remote.fetch is configured to do so.  E.g. when
> I fetch from Eric for git-svn updates with
> 
> 	$ git pull git-svn master
> 
> because I do have
> 
> 	[remote "git-svn"]
>                 url = git://yhbt.net/git-svn
>                 fetch = +refs/heads/*:refs/remotes/git-svn/*

Does the colon really matter? That is, should

  $ git fetch git-svn master

store a tracking ref, but

  $ git fetch git-svn master:foo

not? In both cases, git has learned during the course of an operation
what the new value of the tracking ref could be, so why not store it?

The obstacle I see is that somebody may be using configured refspecs as
something other than a tracking ref. That is, they care about _when_
those refs on the RHS of the refspec are updated, and are not treating
them just as a cache of what the remote side has.

> On the other hand, if my refspecs for "git svn" _were_ like this:
> 
> 	[remote "git-svn"]
> 		url = git://yhbt.net/git-svn
>                 fetch = +refs/heads/master:refs/remotes/git-svn/master
> 
> then I would _not_ want this:
> 
> 	$ git fetch git-svn dev
> 
> to create a new tracking branch refs/remotes/git-svn/dev.

Of course not. It's not in your configured refspec, so you have
requested no such tracking ref. I think we would have to limit ourselves
to how the tracking refs are defined in the refspecs. Otherwise how
would we know to put the tracking ref in refs/remotes/git-svn and not
some other place?

So I think it is a simple matter of applying the configured refspecs
with information we happen to have gotten during the course of a
somewhat-related operation.

-Peff

Re: Question regarding git fetch

[Tom Lambda]

Thank you all for your answers and the interesting following discussion.

>From a user perspective (I do not know how git works internally), the
approach proposed by Juno looks very intuitive to me. Now I understand there
are other cases and users to take into account, which make the
implementation complex. I hope this will be part of git 1.7.0.

--Tom

-- 
View this message in context: http://n2.nabble.com/Question-regarding-git-fetch-tp3527289p3534609.html
Sent from the git mailing list archive at Nabble.com.