git pull failure, truncated object

git pull failure, truncated object

[Bill Lear]

Our QA dude had a disk fill up a while back.  Forgot about it.  Went
to pull during a nightly build.  Got this:

% git pull
remote: Generating pack...
remote: Done counting 104 objects.
remote: Result has 72 objects.
remote: Deltifying 72 objects.
remote:  100% (72/72) done
Unpacking 72 objects
remote: Total 72 (delta 46), reused 52 (delta 26)
error: failed to read delta-pack base object
4b93eb81265ea4f2b436618a4b1c3bea2bedf06d
fatal: unpack-objects died with error code 1

He did a git-gc, twice, and retried.  Still failed.

So, he called me in and we tried to see if the server was acting up
--- perhaps an NFS problem, as we've had those before, but got very
different error messages.  Watched the log file from git-daemon, and
saw nothing.  Finally we took a look at the local repos
.git/objects/4b, and 4b93eb81265ea4f2b436618a4b1c3bea2bedf06d was of
length 0.

So, I looked in the man page of git-gc and thought to try --prune,
as this was not an active repository.  This worked, and then
the pull did as well.

I'm wondering why git-gc did not at least warn us of this problem when
we tried it.  It appeared to us that git-gc gave our repo a clean bill
of health, and so we turned our attention to the remote and
investigated there, instead of continuing in the local repo.

Should we have tried git fsck instead in this case?

It might be nice to have a better error message in this case, one
that locates the problem in the local repo.

If the failure is a short/bad (?) read of a local object, then it
might be nice to say something like:

error: this repo's local object 4b93eb81265ea4f2b436618a4b1c3bea2bedf06d
       in .git/objects/4b is corrupt.  You will need to run
       XXX to diagnose and YYY to repair this.

Or some such.


Bill

Re: git pull failure, truncated object

[Shawn O. Pearce]

Bill Lear <rael@zopyra.com> wrote:
...
> error: failed to read delta-pack base object
> 4b93eb81265ea4f2b436618a4b1c3bea2bedf06d
> fatal: unpack-objects died with error code 1

Danger Will Robinson!  Danger!  Your repository is not acting
normally...

> So, I looked in the man page of git-gc and thought to try --prune,
> as this was not an active repository.  This worked, and then
> the pull did as well.

Don't run git-gc on a repository that is acting strangely, unless
you have concluded that the correct course of action is to just
repack the repository.  (It rarely is, btw.)  You could make things
worse if a packfile contains a corrupt object and you have the
same valid loose object; a gc would delete the valid object and
keep the corrupt one.

> I'm wondering why git-gc did not at least warn us of this problem when
> we tried it.  It appeared to us that git-gc gave our repo a clean bill
> of health, and so we turned our attention to the remote and
> investigated there, instead of continuing in the local repo.
> 
> Should we have tried git fsck instead in this case?

Yes, git-fsck is meant for checking the "health" of a repository.
In this case it should have caught the 0 length object and told
you that loose object was corrupt.  A sure sign that you have a
problem.  If you have a loose object problem, `git prune` might
fix it, as it did here, but only if that object isn't actually
needed.  A `git fsck` immediately after would tell you if the
prune fixed the issue, or not.

-- 
Shawn.

Re: git pull failure, truncated object

[Shawn O. Pearce]

"Shawn O. Pearce" <spearce@spearce.org> wrote:
> Don't run git-gc on a repository that is acting strangely, unless
> you have concluded that the correct course of action is to just
> repack the repository.  (It rarely is, btw.)  You could make things
> worse if a packfile contains a corrupt object and you have the
> same valid loose object; a gc would delete the valid object and
> keep the corrupt one.

OK, my statement is a little blown-out-of-proportion.  Its pretty
hard these days to corrupt an object within a packfile such that
we'll be able to reuse it during the repack that goes on in git-gc,
but still actually have it be corrupt enough that the object is
useless.  The recent index version 2 work from Nico makes it even
harder, as the index adds an additional checksum over the entire
object header and body.

But still, even though the risk is pretty small, I think that
running a destructive operation like git-gc in a repository that is
not acting normally is a bad idea.  You should try to diagnose and
correct the issue before making further changes (or reorganizations)
to that repository's contents.

-- 
Shawn.

Re: git pull failure, truncated object

[Nicolas Pitre]

On Tue, 8 May 2007, Bill Lear wrote:

> He did a git-gc, twice, and retried.  Still failed.
> 
> So, he called me in and we tried to see if the server was acting up
> --- perhaps an NFS problem, as we've had those before, but got very
> different error messages.  Watched the log file from git-daemon, and
> saw nothing.  Finally we took a look at the local repos
> .git/objects/4b, and 4b93eb81265ea4f2b436618a4b1c3bea2bedf06d was of
> length 0.
> 
> So, I looked in the man page of git-gc and thought to try --prune,
> as this was not an active repository.  This worked, and then
> the pull did as well.
> 
> I'm wondering why git-gc did not at least warn us of this problem when
> we tried it.  It appeared to us that git-gc gave our repo a clean bill
> of health, and so we turned our attention to the remote and
> investigated there, instead of continuing in the local repo.

git-gc != git-fsck.


Nicolas

Re: git pull failure, truncated object

[Bill Lear]

On Tuesday, May 8, 2007 at 11:41:14 (-0400) Nicolas Pitre writes:
>On Tue, 8 May 2007, Bill Lear wrote:
>
>> He did a git-gc, twice, and retried.  Still failed.
>> 
>> So, he called me in and we tried to see if the server was acting up
>> --- perhaps an NFS problem, as we've had those before, but got very
>> different error messages.  Watched the log file from git-daemon, and
>> saw nothing.  Finally we took a look at the local repos
>> .git/objects/4b, and 4b93eb81265ea4f2b436618a4b1c3bea2bedf06d was of
>> length 0.
>> 
>> So, I looked in the man page of git-gc and thought to try --prune,
>> as this was not an active repository.  This worked, and then
>> the pull did as well.
>> 
>> I'm wondering why git-gc did not at least warn us of this problem when
>> we tried it.  It appeared to us that git-gc gave our repo a clean bill
>> of health, and so we turned our attention to the remote and
>> investigated there, instead of continuing in the local repo.
>
>git-gc != git-fsck.

Indeed, as is now clear to me.  Would it be prudent to have git-gc
run a quick git-fsck internally and warn if things are not in a kosher
state?


Bill

Re: git pull failure, truncated object

[Nicolas Pitre]

On Tue, 8 May 2007, Bill Lear wrote:

> On Tuesday, May 8, 2007 at 11:41:14 (-0400) Nicolas Pitre writes:
> >On Tue, 8 May 2007, Bill Lear wrote:
> >
> >> He did a git-gc, twice, and retried.  Still failed.
> >> 
> >> So, he called me in and we tried to see if the server was acting up
> >> --- perhaps an NFS problem, as we've had those before, but got very
> >> different error messages.  Watched the log file from git-daemon, and
> >> saw nothing.  Finally we took a look at the local repos
> >> .git/objects/4b, and 4b93eb81265ea4f2b436618a4b1c3bea2bedf06d was of
> >> length 0.
> >> 
> >> So, I looked in the man page of git-gc and thought to try --prune,
> >> as this was not an active repository.  This worked, and then
> >> the pull did as well.
> >> 
> >> I'm wondering why git-gc did not at least warn us of this problem when
> >> we tried it.  It appeared to us that git-gc gave our repo a clean bill
> >> of health, and so we turned our attention to the remote and
> >> investigated there, instead of continuing in the local repo.
> >
> >git-gc != git-fsck.
> 
> Indeed, as is now clear to me.  Would it be prudent to have git-gc
> run a quick git-fsck internally and warn if things are not in a kosher
> state?

No.  git-fsck is a potentially expensive operation and it is up to you 
to remember that git-gc isn't about repository sanity. git-gc only 
repacks things and if it encounters an object that is corrupt it will 
abort and leave your object store as is.  In your case the corrupted 
object wasn't one that needed to be repacked which explains why git-gc 
succeeded.


Nicolas