Git fetch/pull stopped working

Git fetch/pull stopped working

[Egor Ryabkov]

Hi guys!

git fetch and pull stopped working a few days ago on my server with
the following message:

$ git fetch
ERROR: Repository not found.
fatal: The remote end hung up unexpectedly


fetch/push/pull still work just fine with the same (GitHub) repository
from my local PC, and they also work fine with another GH repo from
another server.

I tried comparing "git remote -v" output for local PC and server,
they're the same.
"$ GIT_TRACE=1 git fetch" gives somewhat different output on local PC
and server:
http://pastecode.com/bd3fc1a79f8e9d1eaf30911d9895938051c472f4
but the output is identical for the two servers I have (both are AWS
instances, if that matters).

There are two clones of the repository on the server in question, both
giving the same error. ssh git@github.com works as it should, so SSH
keys are alright.


One quirk I noticed is that "git pull origin st+[Tab]" used to expand
to a branch name; now it expands to a name of the file in current
directory (and the filename is not the same as branch name).

I guess it's some problem with git install on the server - any ideas
how I could try and diagnose this, besides GIT_TRACE?


A discussion on Stackoverflow:
http://stackoverflow.com/questions/10438082/git-fetch-pull-have-stopped-working

Thanks in advance!

Egor.

Re: Git fetch/pull stopped working

[Junio C Hamano]

Egor Ryabkov <egor.ryabkov@gmail.com> writes:

> git fetch and pull stopped working a few days ago on my server with
> the following message:
>
> $ git fetch
> ERROR: Repository not found.
> fatal: The remote end hung up unexpectedly

Are you sure it is an error on the remote side (presumably on GitHub)?

More interestingly, neither "Repository not found" nor "ERROR:" hits
anything relevant to "git fetch" codepath.  The closest I can find is in
http-backend.c that says "Repository not exported", which is on the server
side, but you were talking about ssh connection, so it probably is not
what you are seeing.

Oh, wait.

> "$ GIT_TRACE=1 git fetch" gives somewhat different output on local PC
> and server:
> http://pastecode.com/bd3fc1a79f8e9d1eaf30911d9895938051c472f4

So you are going over http:// after all.  But it does not look like github
to me, so you may be comparing apples and oranges, which would not help in
diagnosing the issue very much.

Puzzled...

Re: Git fetch/pull stopped working

[Carlos Martín Nieto]

On Mon, May 07, 2012 at 04:44:23PM -0700, Junio C Hamano wrote:
> Egor Ryabkov <egor.ryabkov@gmail.com> writes:
> 
> > git fetch and pull stopped working a few days ago on my server with
> > the following message:
> >
> > $ git fetch
> > ERROR: Repository not found.
> > fatal: The remote end hung up unexpectedly
> 
> Are you sure it is an error on the remote side (presumably on GitHub)?
> 
> More interestingly, neither "Repository not found" nor "ERROR:" hits
> anything relevant to "git fetch" codepath.  The closest I can find is in
> http-backend.c that says "Repository not exported", which is on the server
> side, but you were talking about ssh connection, so it probably is not
> what you are seeing.
> 
> Oh, wait.
> 
> > "$ GIT_TRACE=1 git fetch" gives somewhat different output on local PC
> > and server:
> > http://pastecode.com/bd3fc1a79f8e9d1eaf30911d9895938051c472f4
> 
> So you are going over http:// after all.  But it does not look like github
> to me, so you may be comparing apples and oranges, which would not help in
> diagnosing the issue very much.

I can reproduce this issue. It happens somewhere between my phone jack
and rackspace.

I can perform 'git fetch' on other hosts and ssh -T git@github.com
works fine. However, trying to fetch from github gives me an error.

Not only this, but trying to browse to github.com or rubygems.org,
both of which seem to be hosted on the same datacentre gives me
timeouts or takes really long to answer. Routing traffic through my
server in the next country over makes it work, as does fetching from
that host.

The only difference I've been able to find is that the traffic looks
like it goes through different hosts when it goes through the UK on
its way to the US.

> 
> Puzzled...

Yeah... It looks like the Internet is being silly. So much for routing
around damage. What I'm not sure is how this would also affect AWS;
maybe the Irish one?

   cmn

Re: Git fetch/pull stopped working

[Jeff King]

On Tue, May 08, 2012 at 02:10:18AM +0200, Carlos Martín Nieto wrote:

> I can reproduce this issue. It happens somewhere between my phone jack
> and rackspace.
> 
> I can perform 'git fetch' on other hosts and ssh -T git@github.com
> works fine. However, trying to fetch from github gives me an error.
> 
> Not only this, but trying to browse to github.com or rubygems.org,
> both of which seem to be hosted on the same datacentre gives me
> timeouts or takes really long to answer. Routing traffic through my
> server in the next country over makes it work, as does fetching from
> that host.

I can't reproduce this behavior at all. Neither from rackspace's DFW
data center, nor from the IAD one (the latter is where github.com is
located).

> Yeah... It looks like the Internet is being silly. So much for routing
> around damage. What I'm not sure is how this would also affect AWS;
> maybe the Irish one?

If it was a routing issue, I wonder if it has gone away. Are you still
seeing problems?

If there's further discussion, we should probably take it off the git
list. Routing issues are definitely not a git-core problem. :)

-Peff

Re: Git fetch/pull stopped working

[Egor Ryabkov]

> Are you sure it is an error on the remote side (presumably on GitHub)?

No, I'm pretty sure it's something on my side, as clones on other PCs
still work.


>> "$ GIT_TRACE=1 git fetch" gives somewhat different output on local PC
>> and server:
>> http://pastecode.com/bd3fc1a79f8e9d1eaf30911d9895938051c472f4
>
> So you are going over http:// after all.  But it does not look like github
> to me

No, I'm using SSH connection.
Did you imply that it's better to paste logs directly in the email?
Here they go:

############# Local results #############

$ GIT_TRACE=1 git fetch
setup: git_dir: .git
setup: worktree: c:/Projects/(local_path)
setup: cwd: c:/Projects/(local_path)
setup: prefix: (null)
trace: built-in: git 'fetch'
trace: run_command: 'ssh' 'git@github.com' 'git-upload-pack
'\''(username)/(reponame).git'\'''
Enter passphrase for key '(keyname)':
trace: run_command: 'rev-list' '--quiet' '--objects' '--stdin' '--not' '--all'


############# AWS results #############

$ GIT_TRACE=1 git fetch
trace: built-in: git 'fetch'
trace: run_command: 'ssh' 'git@github.com' 'git-upload-pack
'\''(username)/(reponame).git'\'''
ERROR: Repository not found.
fatal: The remote end hung up unexpectedly



I tried re-adding remote: SSH version fails with the same error, but
HTTP version works. I'd rather have SSH working, though.
And ideas about what could've caused it, or how to diagnose it further?

Re: Git fetch/pull stopped working

[Junio C Hamano]

Egor Ryabkov <egor.ryabkov@gmail.com> writes:

>>> "$ GIT_TRACE=1 git fetch" gives somewhat different output on local PC
>>> and server:
>>> http://pastecode.com/bd3fc1a79f8e9d1eaf30911d9895938051c472f4
>> ...
> Did you imply that it's better to paste logs directly in the email?

No, I did not *imply* it; it just didn't "click" that the above was a
URL to a pastebin.

> ############# AWS results #############
>
> $ GIT_TRACE=1 git fetch
> trace: built-in: git 'fetch'
> trace: run_command: 'ssh' 'git@github.com' 'git-upload-pack
> '\''(username)/(reponame).git'\'''
> ERROR: Repository not found.
> fatal: The remote end hung up unexpectedly

Assuming that these "(REDACTED)" are not hiding some crucial difference
between the two destinations, it seems to be the git client installed on
the latter one seems to be the one that is at fault.

But as I said, "ERROR: Repository not found." does not seem to match any
message of the released version of Git, so I cannot even tell where it is
failing.

It could be that the message is coming from a customized version of ssh
login handler github uses to shard and map the user accounts to
directories (it would have said "fatal: '/var/tmp/xx' does not appear to
be a git repository" if they were running the vanilla Git), and if that is
the case, perhaps the argument thrown at git-upload-pack from your local
box and the problematic box somehow look different to Github, but that is
in the "(REDACTED)" part, so I cannot tell from what was pasted in the
pastebin.

Incidentally, oogling [github "ERROR: Repository not found."] seems to
give quite a many hits.

Re: Git fetch/pull stopped working

[Jeff King]

On Tue, May 08, 2012 at 12:37:18PM -0700, Junio C Hamano wrote:

> But as I said, "ERROR: Repository not found." does not seem to match any
> message of the released version of Git, so I cannot even tell where it is
> failing.
> 
> It could be that the message is coming from a customized version of ssh
> login handler github uses to shard and map the user accounts to
> directories[...]

That is exactly where it comes from.

> if they were running the vanilla Git), and if that is the case,
> perhaps the argument thrown at git-upload-pack from your local box and
> the problematic box somehow look different to Github, but that is in
> the "(REDACTED)" part, so I cannot tell from what was pasted in the
> pastebin.

Agreed. It's hard to tell if there's a real problem or not. If the
redacted bits are the same, then it is probably a github issue. In that
case, Egor, you should email support@github.com about it and mention the
name of the problematic repo.

I'll look into the weird routing thing that Carlos mentioned, too.

-Peff

Re: Git fetch/pull stopped working

[Carlos Martín Nieto]

On Tue, May 08, 2012 at 04:08:42PM -0400, Jeff King wrote:
> 
> I'll look into the weird routing thing that Carlos mentioned, too.

FWIW, it looks like it got solved. My ISP (or their ISP) seem pretty
inept at keeping traffic flowing, so I wouldn't be surprised if it
showed up again later. I sent an e-mail to support@ with some traces,
if you're interested.

   cmn

Re: Git fetch/pull stopped working

[Egor Ryabkov]

> Agreed. It's hard to tell if there's a real problem or not. If the
> redacted bits are the same,

Yes, I checked this a few times and tried re-creating remote with the
address copied from GH site.
HTTP worked, SSH didn't.


> then it is probably a github issue. In that
> case, Egor, you should email support@github.com about it and mention the
> name of the problematic repo.

Yes, I actually contacted them last Friday, working with Simon Rozet now.
I updated him on what we've found and sent him the link to this thread.


Thanks for your help guys :)

Re: Git fetch/pull stopped working

[Egor Ryabkov]

Problem solved.

Turns out that was a miscommunication on out side: the guy who setup
the keys on that box left, and his access to GH repo has been revoked.

And GitHub returns "repo not found" rather than "you have no access
rights for this repository".


On Wed, May 9, 2012 at 1:28 AM, Egor Ryabkov <egor.ryabkov@gmail.com> wrote:
>
> > Agreed. It's hard to tell if there's a real problem or not. If the
> > redacted bits are the same,
>
> Yes, I checked this a few times and tried re-creating remote with the
> address copied from GH site.
> HTTP worked, SSH didn't.
>
>
> > then it is probably a github issue. In that
> > case, Egor, you should email support@github.com about it and mention the
> > name of the problematic repo.
>
> Yes, I actually contacted them last Friday, working with Simon Rozet now.
> I updated him on what we've found and sent him the link to this thread.
>
>
> Thanks for your help guys :)

Re: Git fetch/pull stopped working

[jaseem abid]

On Sat, May 12, 2012 at 12:17 AM, Egor Ryabkov <egor.ryabkov@gmail.com> wrote:
> Problem solved.
>
> Turns out that was a miscommunication on out side: the guy who setup
> the keys on that box left, and his access to GH repo has been revoked.
>
> And GitHub returns "repo not found" rather than "you have no access
> rights for this repository".

I have seen a lot of github guys in this list. I hope they will fix
this soon. I had this issue a long time back and now I'm understanding
what exactly happened then :)


-- 
Jaseem Abid
http://jaseemabid.github.com

Re: Git fetch/pull stopped working

[Jeff King]

On Sat, May 12, 2012 at 01:25:42AM +0530, jaseem abid wrote:

> On Sat, May 12, 2012 at 12:17 AM, Egor Ryabkov <egor.ryabkov@gmail.com> wrote:
> > Problem solved.
> >
> > Turns out that was a miscommunication on out side: the guy who setup
> > the keys on that box left, and his access to GH repo has been revoked.
> >
> > And GitHub returns "repo not found" rather than "you have no access
> > rights for this repository".
> 
> I have seen a lot of github guys in this list. I hope they will fix
> this soon. I had this issue a long time back and now I'm understanding
> what exactly happened then :)

This is not a bug, but rather the intentional behavior. It is there to
prevent an information leak about the names of private repositories that
the user has. Git-daemon has the same behavior by default (though it has
a "friendly" mode if you are willing to leak that information for your
site).

-Peff