How can git pull be up-to-date and git push fail?

How can git pull be up-to-date and git push fail?

[Bill Lear]

I'm trying to explain this behavior to a co-worker, and how to solve
it:

% git pull
Already up-to-date.

% git push
error: remote 'refs/remotes/origin/flexify' is not a strict subset of
localref 'refs/remotes/origin/flexify'. maybe you are not up-to-date
and need to pull first?

I believe he is using git 1.5.0 and that the above was tried from the
master branch.

So, I'm confused: I would have thought the way you solve the problem
in step 2 is by executing step 1.  Would he need to do this:

% git checkout flexify
% git pull flexify:flexify
% git checkout master
% git push


Bill

Re: How can git pull be up-to-date and git push fail?

[Jeff King]

On Thu, Apr 05, 2007 at 07:15:54AM -0500, Bill Lear wrote:

> I'm trying to explain this behavior to a co-worker, and how to solve
> it:
> 
> % git pull
> Already up-to-date.
> 
> % git push
> error: remote 'refs/remotes/origin/flexify' is not a strict subset of
> localref 'refs/remotes/origin/flexify'. maybe you are not up-to-date
> and need to pull first?

That ref name is a bit suspect...why are you pushing your your remote
tracking branches? If you are simply pushing to a repository that is a
pure mirror, maybe your 'push' line in the config file should have a '+'
at the front?

At any rate, the reason why the pull doesn't fix it is that it is not
pulling onto flexify, but onto master, which it looks like you guessed:

> So, I'm confused: I would have thought the way you solve the problem
> in step 2 is by executing step 1.  Would he need to do this:
> 
> % git checkout flexify
> % git pull flexify:flexify
> % git checkout master
> % git push

It would need to be:
  git pull $remote flexify:flexify

I don't think that works, though, because it's _not_ the flexify branch
that's the problem. It's the refs/remotes/origin/flexify, which
generally shouldn't be getting pushed at all. Can you show us the
.git/config?

-Peff

Re: How can git pull be up-to-date and git push fail?

[Bill Lear]

On Thursday, April 5, 2007 at 09:49:54 (-0400) Jeff King writes:
>On Thu, Apr 05, 2007 at 07:15:54AM -0500, Bill Lear wrote:
>
>> I'm trying to explain this behavior to a co-worker, and how to solve
>> it:
>> 
>> % git pull
>> Already up-to-date.
>> 
>> % git push
>> error: remote 'refs/remotes/origin/flexify' is not a strict subset of
>> localref 'refs/remotes/origin/flexify'. maybe you are not up-to-date
>> and need to pull first?
>
>That ref name is a bit suspect...why are you pushing your your remote
>tracking branches? If you are simply pushing to a repository that is a
>pure mirror, maybe your 'push' line in the config file should have a '+'
>at the front?
>
>At any rate, the reason why the pull doesn't fix it is that it is not
>pulling onto flexify, but onto master, which it looks like you guessed:
>
>> So, I'm confused: I would have thought the way you solve the problem
>> in step 2 is by executing step 1.  Would he need to do this:
>> 
>> % git checkout flexify
>> % git pull flexify:flexify
>> % git checkout master
>> % git push
>
>It would need to be:
>  git pull $remote flexify:flexify
>
>I don't think that works, though, because it's _not_ the flexify branch
>that's the problem. It's the refs/remotes/origin/flexify, which
>generally shouldn't be getting pushed at all. Can you show us the
>.git/config?

Here is the config for the remote repository:

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
[remote "origin"]
        url = git://source/fusion
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master

Here is the local:

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = ssh://poire/home/jml/repos/new/fusion
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master

Hmm.... I wonder if his remote repo is non-bare... I'll try to find
out.


Bill

Re: How can git pull be up-to-date and git push fail?

[Bill Lear]

On Thursday, April 5, 2007 at 09:07:11 (-0500) Bill Lear writes:
>...
>Hmm.... I wonder if his remote repo is non-bare... I'll try to find
>out.

Yes, it is.  Here is the reply I got from my friend:

>Bill Lear writes:
> > So, the remote repo to which you pushed originally is non-bare?  That
> > is, it has working files checked out?
>
>Correct.
>
> > If so, that is a problem, but
> > perhaps not the cause of your problems.  Never push to a non-bare
> > repo: you need to go that repo and do a pull instead.
>
>I've been able to work around such problems.  I know well about the
>double-commit reversal whammy that can happen (since it happened to me).
>If I push to a non-bare repo that's otherwise got no file modifications
>in it, I can do a "git status" to see what files it thinks are out of
>date, which tells me (by the file list) that the push succeeded.  Then
>I just do a "git checkout -f" to sync up.  Up til now, has worked like
>a charm.  Alternately, I switch to some bogus temporary branch in the
>non-bare repo I'm pushing to & switch back after.


Bill

Re: How can git pull be up-to-date and git push fail?

[Jeff King]

On Thu, Apr 05, 2007 at 09:07:11AM -0500, Bill Lear wrote:

> Here is the local:
> 
> [core]
>         repositoryformatversion = 0
>         filemode = true
>         bare = false
>         logallrefupdates = true
> [remote "origin"]
>         url = ssh://poire/home/jml/repos/new/fusion
>         fetch = +refs/heads/*:refs/remotes/origin/*
> [branch "master"]
>         remote = origin
>         merge = refs/heads/master

I don't see anything there that should cause branches under refs/remotes
to be pushed. Was he using 'git-push --all' by any chance?

-Peff

Re: How can git pull be up-to-date and git push fail?

[Bill Lear]

On Thursday, April 5, 2007 at 16:46:04 (-0400) Jeff King writes:
>On Thu, Apr 05, 2007 at 09:07:11AM -0500, Bill Lear wrote:
>
>> Here is the local:
>> 
>> [core]
>>         repositoryformatversion = 0
>>         filemode = true
>>         bare = false
>>         logallrefupdates = true
>> [remote "origin"]
>>         url = ssh://poire/home/jml/repos/new/fusion
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>> [branch "master"]
>>         remote = origin
>>         merge = refs/heads/master
>
>I don't see anything there that should cause branches under refs/remotes
>to be pushed. Was he using 'git-push --all' by any chance?

No, just plain ol' git push.  One possibility is that in mediating
between him and the list, there has been a loss of information.
Perhaps he omitted details in his account of how things progressed ---
who knows, perhaps he was on the wrong branch (though I doubt this, he
only works on the master branch, and the other branch in question was
someone else's entirely).

Regardless, does my assumption --- a 'git pull' should rectify
the 'you are not up to date' problem --- hold in general?

Thanks for helping.


Bill

Re: How can git pull be up-to-date and git push fail?

[Jeff King]

On Thu, Apr 05, 2007 at 03:55:18PM -0500, Bill Lear wrote:

> No, just plain ol' git push.  One possibility is that in mediating
> between him and the list, there has been a loss of information.
> Perhaps he omitted details in his account of how things progressed ---
> who knows, perhaps he was on the wrong branch (though I doubt this, he
> only works on the master branch, and the other branch in question was
> someone else's entirely).

No, I think you gave the correct information. See the other part of the
thread between me and Junio. What's happening is that 'git-push' tries
to push every ref that both you and the remote have. Because he's
pushing into a non-bare repository, both sides have
refs/remotes/origin/flexify. But they're not necessarily at the same
point, since they're both fetched from whatever your origin is, thus the
problem. In this case, his flexify tracking branch for origin is
_behind_ the remote's, so the push doesn's work.

> Regardless, does my assumption --- a 'git pull' should rectify
> the 'you are not up to date' problem --- hold in general?

Generally, yes. But the remotes/ hierarchy is not something you pull
into. So in this case, it won't help. The real solution is that git-push
should not be pushing remotes/ at all: it should be purely a local
thing (the patch I posted elsewhere in the thread fixes this, but will
not be the final behavior -- however, it's enough to fix this problem).

As a temporary solution, fetching from origin should put him at the same
place (or ahead of) the remote, which should allow the push to go
through.

-Peff

Re: How can git pull be up-to-date and git push fail?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Thu, Apr 05, 2007 at 09:07:11AM -0500, Bill Lear wrote:
>
>> Here is the local:
>> 
>> [core]
>>         repositoryformatversion = 0
>>         filemode = true
>>         bare = false
>>         logallrefupdates = true
>> [remote "origin"]
>>         url = ssh://poire/home/jml/repos/new/fusion
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>> [branch "master"]
>>         remote = origin
>>         merge = refs/heads/master
>
> I don't see anything there that should cause branches under refs/remotes
> to be pushed. Was he using 'git-push --all' by any chance?

IIRC "git push" without explicit refspecs push the matching
refs, but I am a bit under the weather and feverish, so don't
take my word literally but look at git-push manual page please.

Re: How can git pull be up-to-date and git push fail?

[Jeff King]

On Thu, Apr 05, 2007 at 02:18:58PM -0700, Junio C Hamano wrote:

> IIRC "git push" without explicit refspecs push the matching
> refs, but I am a bit under the weather and feverish, so don't
> take my word literally but look at git-push manual page please.

Ah, yes you're right. It really doesn't make sense to push
refs/remotes/* in most cases, since they're just tracking branches, and
if the destination _does_ have them, then it is unlikely to be in sync
with you (leading to Bill's problem).  OTOH, you might want to be able
to push them explicitly if you are doing a strict mirror of a
repository.

The patch below turns off refs/remotes/ sending for "git-push" and
"git-push --all", but still allows "git-push origin
remotes/origin/master". I'm not sure about the semantics; maybe --all
should imply even remotes?

It also only impacts send-pack; I suspect pushing over dumb transports
now has different behavior, but I haven't looked. My testing was light,
so I may have totally broken something else, too. Input from more
clueful people would be helpful.

Does this seem like a sane direction to take? It just seems silly to be
pushing refs/remotes, which 99% of the time should be a purely local
thing.

-Peff

---
diff --git a/send-pack.c b/send-pack.c
index d5b5162..39829e3 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -131,6 +131,8 @@ static int one_local_ref(const char *refname, const unsigned char *sha1, int fla
 {
 	struct ref *ref;
 	int len = strlen(refname) + 1;
+	if (!prefixcmp(refname, "refs/remotes/"))
+		return 0;
 	ref = xcalloc(1, sizeof(*ref) + len);
 	hashcpy(ref->new_sha1, sha1);
 	memcpy(ref->name, refname, len);

Re: How can git pull be up-to-date and git push fail?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Thu, Apr 05, 2007 at 02:18:58PM -0700, Junio C Hamano wrote:
>
>> IIRC "git push" without explicit refspecs push the matching
>> refs, but I am a bit under the weather and feverish, so don't
>> take my word literally but look at git-push manual page please.
>
> Ah, yes you're right. It really doesn't make sense to push
> refs/remotes/* in most cases, since they're just tracking branches, and
> if the destination _does_ have them, then it is unlikely to be in sync
> with you (leading to Bill's problem).  OTOH, you might want to be able
> to push them explicitly if you are doing a strict mirror of a
> repository.
>
> The patch below turns off refs/remotes/ sending for "git-push" and
> "git-push --all", but still allows "git-push origin
> remotes/origin/master". I'm not sure about the semantics; maybe --all
> should imply even remotes?

The behaviour of git-send-pack that pushes all the matching refs
made sense when there was no separate-remotes layout (and there
was no "git push" wrapper).  Back then, the expected use was
really:

 * For integrators who own public repositories, send-pack to
   their public repositories without refspecs updated what they
   already decided to publish in refs/heads and refs/tags.

   Updating refs/tags hierarchy means moving tags which should
   have done with caution (since it changes the general public's
   view of what the version v2.6.12 really is), but no one can
   update tag without first forcing with "git tag -f" in one's
   own repository to begin with, it was not a problem in
   practice.

 * For people who used pull/push with a central repository as
   the CVS-style synchronization point, the central repository,
   being the mother of all participant repositories, was
   supposed to lack the 'origin' branch, and participant
   repositories had 'origin' that tracked 'master' of the
   central repository.  So matching push never needed to worry
   about 'origin', and 'master' of participant was pushed to
   'master' of the central repository, undergoing the usual
   fast-forward checks and all.

   Even here, we had a few trouble reports from people who
   primed their central repository by cloning from somewhere
   (and cloning without --bare) and left the 'origin' branch
   around.  We might have added instruction to our documentation
   that tells people to remove 'origin' branch from their
   central repositories to avoid confusion (but I am too lazy to
   look).

 * For anybody who has more than one private repositories
   (e.g. mothership and notebook) and uses push between them
   when switching repositories to work in, new development might
   have been on a new branch, in which case it needed an
   explicit refspec to send the new branch or tag.  But
   "matching refs" rule was useful for heads most of the time,
   as long as the two repositories were reasonably in sync
   (e.g. after supper you sit on your couch working on notebook,
   and you finish the day by pushing matching refs to your
   mothership to prepare for tomorrow, which you would start on
   your mothership machine).

In all these cases, what we meant by "matching refs" rule is to
update matching branch heads.  Even pushing matching tags is
debatable, as consciously replacing a tag that was already
pushed out with a new one is already a special case, and I would
suspect that replacing a public tag should be done via an
explicit "git send-pack $url tag v2.6.12", even though the
"matching refs" rule would silently update it.

People who built git-send-pack did not use separate remotes
layout (which has not been even invented yet), nor StGIT, so
refs/remotes, refs/bases nor refs/patches were not even on the
radar.  Speaking of StGIT, I do not think it makes sense to sync
only .git/refs/{bases,patches} without sending .git/patches.

> Does this seem like a sane direction to take? It just seems silly to be
> pushing refs/remotes, which 99% of the time should be a purely local
> thing.

So I'd suspect "git push" (not limited to git-send-pack) without
explicit refspecs should only do "matching branch heads" these
days, to keep the original spirit but yet to adjust to today's
reality.  In other words, instead of treating remotes/ specially
(in a negative sense), like your patch does, I think we should
do only "matching refs in heads/" hierarchy.

Re: How can git pull be up-to-date and git push fail?

[Jeff King]

On Fri, Apr 06, 2007 at 02:44:02PM -0700, Junio C Hamano wrote:

> was no "git push" wrapper).  Back then, the expected use was
> really:

Thanks for this for the historical information.

> So I'd suspect "git push" (not limited to git-send-pack) without
> explicit refspecs should only do "matching branch heads" these
> days, to keep the original spirit but yet to adjust to today's
> reality.  In other words, instead of treating remotes/ specially
> (in a negative sense), like your patch does, I think we should
> do only "matching refs in heads/" hierarchy.

Agreed. That was actually my initial feeling, but it meant changing the
semantics of all of the other parts of the refs/* hierarchy with respect
to publishing, which I was a bit nervous about.

A patch is attached below for comment; it's the same as my previous
patch, except:
  - matching only refs/heads/
  - rename local_ref to local_head for consistency (one of the functions
    was already get_local_heads, but the other's didn't match)
  - avoid calling strlen if we're just going to return immediately
  - add a test to simulate Bill's problem
  - update documentation for git-push

But it looks like I have broken pushing of existing tags using --tags,
so I think I might have to put the ref-culling in a different spot. Let
me look in that.

BTW, my gut feeling is that send-pack is plumbing, and git-push is
porcelain, and therefore the decisions about what to push should be made
at the push layer, and send-pack should only push what it is explicitly
told to push. But changing the defaults of send-pack at this point is
probably a bad idea.

---
 Documentation/git-push.txt      |    4 ++--
 Documentation/git-send-pack.txt |    4 ++--
 send-pack.c                     |   15 +++++++++------
 t/t5400-send-pack.sh            |   18 ++++++++++++++++++
 4 files changed, 31 insertions(+), 10 deletions(-)

diff --git a/Documentation/git-push.txt b/Documentation/git-push.txt
index f8cc2b5..0b21306 100644
--- a/Documentation/git-push.txt
+++ b/Documentation/git-push.txt
@@ -46,7 +46,7 @@ even if it does not result in a fast forward update.
 Note: If no explicit refspec is found, (that is neither
 on the command line nor in any Push line of the
 corresponding remotes file---see below), then all the
-refs that exist both on the local side and on the remote
+heads that exist both on the local side and on the remote
 side are updated.
 +
 `tag <tag>` means the same as `refs/tags/<tag>:refs/tags/<tag>`.
@@ -60,7 +60,7 @@ the remote repository.
 
 \--all::
 	Instead of naming each ref to push, specifies that all
-	refs be pushed.
+	refs under `$GIT_DIR/refs/heads/` be pushed.
 
 \--tags::
 	All refs under `$GIT_DIR/refs/tags` are pushed, in
diff --git a/Documentation/git-send-pack.txt b/Documentation/git-send-pack.txt
index 205bfd2..3271e88 100644
--- a/Documentation/git-send-pack.txt
+++ b/Documentation/git-send-pack.txt
@@ -32,7 +32,7 @@ OPTIONS
 
 \--all::
 	Instead of explicitly specifying which refs to update,
-	update all refs that locally exist.
+	update all heads that locally exist.
 
 \--force::
 	Usually, the command refuses to update a remote ref that
@@ -70,7 +70,7 @@ With '--all' flag, all refs that exist locally are transferred to
 the remote side.  You cannot specify any '<ref>' if you use
 this flag.
 
-Without '--all' and without any '<ref>', the refs that exist
+Without '--all' and without any '<ref>', the heads that exist
 both on the local side and on the remote side are updated.
 
 When one or more '<ref>' are specified explicitly, it can be either a
diff --git a/send-pack.c b/send-pack.c
index d5b5162..b2b6886 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -124,13 +124,16 @@ static int ref_newer(const unsigned char *new_sha1,
 	return found;
 }
 
-static struct ref *local_refs, **local_tail;
+static struct ref *local_heads, **local_tail;
 static struct ref *remote_refs, **remote_tail;
 
-static int one_local_ref(const char *refname, const unsigned char *sha1, int flag, void *cb_data)
+static int one_local_head(const char *refname, const unsigned char *sha1, int flag, void *cb_data)
 {
 	struct ref *ref;
-	int len = strlen(refname) + 1;
+	int len;
+	if (prefixcmp(refname, "refs/heads/"))
+		return 0;
+	len = strlen(refname) + 1;
 	ref = xcalloc(1, sizeof(*ref) + len);
 	hashcpy(ref->new_sha1, sha1);
 	memcpy(ref->name, refname, len);
@@ -141,8 +144,8 @@ static int one_local_ref(const char *refname, const unsigned char *sha1, int fla
 
 static void get_local_heads(void)
 {
-	local_tail = &local_refs;
-	for_each_ref(one_local_ref, NULL);
+	local_tail = &local_heads;
+	for_each_ref(one_local_head, NULL);
 }
 
 static int receive_status(int in)
@@ -198,7 +201,7 @@ static int send_pack(int in, int out, int nr_refspec, char **refspec)
 	/* match them up */
 	if (!remote_tail)
 		remote_tail = &remote_refs;
-	if (match_refs(local_refs, remote_refs, &remote_tail,
+	if (match_refs(local_heads, remote_refs, &remote_tail,
 		       nr_refspec, refspec, send_all))
 		return -1;
 
diff --git a/t/t5400-send-pack.sh b/t/t5400-send-pack.sh
index 477b267..ca4485a 100755
--- a/t/t5400-send-pack.sh
+++ b/t/t5400-send-pack.sh
@@ -113,4 +113,22 @@ test_expect_success \
 	! git diff .git/refs/heads/master victim/.git/refs/heads/master
 '
 
+test_expect_success \
+	'pushing does not include non-head refs' '
+	mkdir parent && cd parent &&
+	  git-init && touch file && git-add file && git-commit -m add &&
+	cd .. &&
+	git-clone parent child1 &&
+	git-clone parent child2 &&
+	cd parent &&
+	  echo parent >file && git-commit -a -m change &&
+	cd .. &&
+	cd child1 && git-pull && cd .. &&
+	cd child2 &&
+	  touch >file2 git-add file2 && git-add file2 && git-commit -m add2 &&
+	  git-pull ../child1 &&
+	  git-push ../child1 2>err &&
+	  ! grep ^error err
+'
+
 test_done

Re: How can git pull be up-to-date and git push fail?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Fri, Apr 06, 2007 at 02:44:02PM -0700, Junio C Hamano wrote:
>
>> was no "git push" wrapper).  Back then, the expected use was
>> really:
>
> Thanks for this for the historical information.
>
>> So I'd suspect "git push" (not limited to git-send-pack) without
>> explicit refspecs should only do "matching branch heads" these
>> days, to keep the original spirit but yet to adjust to today's
>> reality....
>
> Agreed. That was actually my initial feeling, but it meant changing the
> semantics of all of the other parts of the refs/* hierarchy with respect
> to publishing, which I was a bit nervous about.
> ...
> BTW, my gut feeling is that send-pack is plumbing, and git-push is
> porcelain, and therefore the decisions about what to push should be made
> at the push layer, and send-pack should only push what it is explicitly
> told to push. But changing the defaults of send-pack at this point is
> probably a bad idea.

Yes, that is exactly why I suggested "git push" without explicit
refspecs to drive the plumbings to push only matching branches.
So if you are worried about breaking backward compatibility for
real men (iow people who use git-send-pack directly), it would
be a less impact change to (1) add --matching-heads option to
send-pack and perhaps http-push, and change their behaviour only
in that case, and (2) pass that option when git-push did not get
explicit refspecs to send from the user, either from the command
line nor from the config.

Re: How can git pull be up-to-date and git push fail?

[Jeff King]

On Sat, Apr 07, 2007 at 02:26:44PM -0700, Junio C Hamano wrote:

> Yes, that is exactly why I suggested "git push" without explicit
> refspecs to drive the plumbings to push only matching branches.
> So if you are worried about breaking backward compatibility for
> real men (iow people who use git-send-pack directly), it would
> be a less impact change to (1) add --matching-heads option to
> send-pack and perhaps http-push, and change their behaviour only
> in that case, and (2) pass that option when git-push did not get
> explicit refspecs to send from the user, either from the command
> line nor from the config.

Ah, I (obviously) didn't pick that up from your original message. Let me
rework my patch and see if I can get something more reasonable.

-Peff