How do I investigate apparently random git clone reports of "error: File ... has bad hash"?

How do I investigate apparently random git clone reports of "error: File ... has bad hash"?

[Thorkil Naur]

Hello,

On a Mac OS X 10.5.8 with

> $ uname -a
> Darwin thorkil-naurs-intel-mac-mini.local 9.8.0 Darwin Kernel Version 9.8.0:
Wed Jul 15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386 i386
> $ git --version
> git version 1.7.4.1
> $ 

that I use as a so-called buildbot slave (http://trac.buildbot.net/) for GHC
(http://www.haskell.org/ghc/), git clone is used repeatedly to fetch the main
repository and several other (library) repositories on which the main repository
depends. In recent months, I have experienced apparently random failures in
which git clone reports, for example:

> $ git clone http://darcs.haskell.org/ghc.git/ build8
> Cloning into build8...
> error: File 42988feeeb76f5cb92b541e9dac277e073bcb3ef has bad hash
> error: Unable to find 42988feeeb76f5cb92b541e9dac277e073bcb3ef under
http://darcs.haskell.org/ghc.git
> Cannot obtain needed blob 42988feeeb76f5cb92b541e9dac277e073bcb3ef
> while processing commit ffb2e81c03a01e74825b3a0223e214df59241fab.
> error: Fetch failed.

There are variations of this, such as

> $ git clone http://darcs.haskell.org/ghc.git/ build6
> Cloning into build6...
> error: inflate: data stream error (incorrect header check)
> error: inflate: data stream error (incorrect header check)
> error: inflate: data stream error (incorrect header check)
> error: inflate: data stream error (incorrect header check)
> error: inflate: data stream error (incorrect header check)
> error: File 280db842d43e7e68c09bf01695434fc0f8fff796
(http://darcs.haskell.org/ghc.git/objects/c8/ba6e77a6202e7df588c3b1fa3ada5142482036)
corrupt
> error: Unable to find c8ba6e77a6202e7df588c3b1fa3ada5142482036 under
http://darcs.haskell.org/ghc.git
> Cannot obtain needed blob c8ba6e77a6202e7df588c3b1fa3ada5142482036
> while processing commit ffb2e81c03a01e74825b3a0223e214df59241fab.
> error: Fetch failed.

In most cases, the git clone succeeds, but failure nevertheless happens
sufficiently often to break most builds. See, for example
http://darcs.haskell.org/ghcBuilder/builders/tn23/437/3.html.

I tried the clone of the main repository on my Ubuntu Linux machine that says

> $ uname -a
> Linux tn24 2.6.32-33-generic #72-Ubuntu SMP Fri Jul 29 21:08:37 UTC 2011 i686
GNU/Linux
> $ git --version
> git version 1.7.0.4
> $ 

(which is connected to the same network as the Mac) and was not able to
reproduce the error.

I have looked for git options (debug, verbosity) for increasing the amount of
information reported, but have not managed to find anything that seemed useful
in this situation. The apparent randomness of the failures points to some kind
of network problem, but I have not noticed other indications that my network
connection is unstable.

I am not a git expert at all, I am not using git for anything but GHC building.
I would very much appreciate some indication of how I might investigate this
problem.

Best regards
Thorkil

Re: How do I investigate apparently random git clone reports of "error: File ... has bad hash"?

[Andreas Schwab]

Thorkil Naur <naur@post11.tele.dk> writes:

>> $ git clone http://darcs.haskell.org/ghc.git/ build8
>> Cloning into build8...
>> error: File 42988feeeb76f5cb92b541e9dac277e073bcb3ef has bad hash
>> error: Unable to find 42988feeeb76f5cb92b541e9dac277e073bcb3ef under
> http://darcs.haskell.org/ghc.git
>> Cannot obtain needed blob 42988feeeb76f5cb92b541e9dac277e073bcb3ef
>> while processing commit ffb2e81c03a01e74825b3a0223e214df59241fab.
>> error: Fetch failed.

I just tried to clone it and got this error:

$ git clone http://darcs.haskell.org/ghc.git
Cloning into ghc...
error: Recv failure: Connection reset by peer (curl_result = 56, http_code = 0, sha1 = be6810bb027643bf0697b3d237426110f064aba1)
error: Unable to find be6810bb027643bf0697b3d237426110f064aba1 under http://darcs.haskell.org/ghc.git
Cannot obtain needed commit be6810bb027643bf0697b3d237426110f064aba1
while processing commit 6942b112082fbcdff5c66f06f56fdd336861da47.
error: Fetch failed.

It looks like this is just a network problem.

Btw, the repo is rather strange.  It's not a bare repo, but does not
contain a .git directory.  Instead the files that are normally under
.git are placed directly in the working tree.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: How do I investigate apparently random git clone reports of "error: File ... has bad hash"?

[Thorkil Naur]

Hello Andreas,

Thank you very much for your response. I have separately received the
advice of upgrading my curl installation:

> Date: Fri, 30 Sep 2011 09:08:56 +0200
> From: Karel Gardas <karel.gardas@centrum.cz>
> To: cvs-ghc <cvs-ghc@haskell.org>
> Subject: Re: tn23 (x86 OSX HEAD), build 442, Failure
> ...
> Hello,
>
> my opensolaris builder machine also suffered from the same issue like
> tn23 and sometimes even mbolingbroke and others. Symptoms are you are
> not able to grab the ghc code or subrepos code. The solution is
> simple: (1) either remove curl from your path or (2) update curl to
> the latest version (7.21.7 works for me) and make sure it is really
> using its latest libcurl. Once I did (2) here I've never seen the
> issue again.
> ...

Additional details:

> http://www.haskell.org/pipermail/cvs-ghc/2011-October/066434.html

So it appears that upgrading curl has removed the problem.

Best regards
Thorkil

On Sun, Sep 11, 2011 at 09:59:15PM +0200, Andreas Schwab wrote:
> Thorkil Naur <naur@post11.tele.dk> writes:
> 
> >> $ git clone http://darcs.haskell.org/ghc.git/ build8
> >> Cloning into build8...
> >> error: File 42988feeeb76f5cb92b541e9dac277e073bcb3ef has bad hash
> >> error: Unable to find 42988feeeb76f5cb92b541e9dac277e073bcb3ef under
> > http://darcs.haskell.org/ghc.git
> >> Cannot obtain needed blob 42988feeeb76f5cb92b541e9dac277e073bcb3ef
> >> while processing commit ffb2e81c03a01e74825b3a0223e214df59241fab.
> >> error: Fetch failed.
> 
> I just tried to clone it and got this error:
> 
> $ git clone http://darcs.haskell.org/ghc.git
> Cloning into ghc...
> error: Recv failure: Connection reset by peer (curl_result = 56, http_code = 0, sha1 = be6810bb027643bf0697b3d237426110f064aba1)
> error: Unable to find be6810bb027643bf0697b3d237426110f064aba1 under http://darcs.haskell.org/ghc.git
> Cannot obtain needed commit be6810bb027643bf0697b3d237426110f064aba1
> while processing commit 6942b112082fbcdff5c66f06f56fdd336861da47.
> error: Fetch failed.
> 
> It looks like this is just a network problem.
> 
> Btw, the repo is rather strange.  It's not a bare repo, but does not
> contain a .git directory.  Instead the files that are normally under
> .git are placed directly in the working tree.
> 
> Andreas.
> 
> -- 
> Andreas Schwab, schwab@linux-m68k.org
> GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
> "And now for something completely different."

Re: How do I investigate apparently random git clone reports of "error: File ... has bad hash"?

[Andreas Schwab]

Thorkil Naur <naur@post11.tele.dk> writes:

> I have looked for git options (debug, verbosity) for increasing the amount of
> information reported, but have not managed to find anything that seemed useful
> in this situation.

Try adding -v -v when cloning.  That also shows you that the server does
not support smart-http, which is suboptimal.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."