Recover broken git repository?

Recover broken git repository?

[Florian Breitwieser]

Hi, 
I have problems with my git repository, attached below are the steps I tried
to resolve it. But now I am stuck. Is there any good way to recover?

Thank you
Florian

-- 
Florian Breitwieser
PhD Student

Research Center of Molecular Medicine
of the Austrian Academy of Science

-------------------

$ git commit -m "Some message"
error: invalid object 1086b1c606a04bcb78b92d1d411a299d20d18034
fatal: Error building trees

$ git-fsck --full
dangling tree c2549a3cdd83098a523cb707f217f4656cde7eb5

$ git prune
$ git commit -m "Some message"
Created commit e32d5dd: Some message
 2 files changed, 167 insertions(+), 10 deletions(-)

$ git push
Counting objects: 15, done.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (8/8), 2.44 KiB, done.
Total 8 (delta 4), reused 0 (delta 0)
Unpacking objects: 100% (8/8), done.
fatal: unresolved deltas left after unpacking
error: unpack failed: unpacker exited with error code
To /mnt/extstore2/bioinformatic/git/ppi.git
 ! [remote rejected] master -> master (n/a (unpacker error))
error: failed to push some refs to
'/mnt/extstore2/bioinformatic/git/ppi.git'

-- 
View this message in context: http://n2.nabble.com/Recover-broken-git-repository--tp3255106p3255106.html
Sent from the git mailing list archive at Nabble.com.

Re: Recover broken git repository?

[Alex Riesen]

On Tue, Jul 14, 2009 at 15:20, Florian Breitwieser<florian.bw@gmail.com> wrote:
> I have problems with my git repository, attached below are the steps I tried
> to resolve it. But now I am stuck. Is there any good way to recover?

There are recovery instructions all over the web, and on this mailing list.
Try googling for "git repository corruption recovery".
There is even one in Git's repository:

  Documentation/howto/recover-corrupted-blob-object.txt

> $ git commit -m "Some message"
> error: invalid object 1086b1c606a04bcb78b92d1d411a299d20d18034

Looks like you have hit a genuine repository corruption.

You can try to look for the object in all copies of the project (or just in any
projects you possibly have laying around) and copy it into you repo.
To test if the object is present in a repo:

  git cat-file -t 1086b1c606a04bcb78b92d1d411a299d20d18034

You can just copy (or reference in .git/objects/info/alternates) the whole
repo containing the object in the broken repo. You can cleanup unused
objects afterwards.

> fatal: Error building trees

You can try "git log --pretty=raw --raw --no-abbrev HEAD | less" and
look for the first commit which introduced the invalid object. The history
up to this commit must be intact, BTW. Try to find out what object it was.
If it was a file, than maybe you have the original source somewhere,
or the latest version is present and good enough. Then you can checkout
the good commit, introduce the file from the source, commit, and cherry-pick
all the newer commits on top of it.
If it was a tree, you can try to guess what files did it contain in the
commit which references it and try to rebuild it by using git add and
git write-tree (the latter prints the SHA1 of the generated tree on
stdout, so you can retry indefinitely).
If it happens to be a commit (you'll find the SHA1 in parents of
the good commits, but it does not look like it is the case), you can
try git fsck --lost-found to find all the commits which are not
referenced, try connecting them (by using grafts, for example.
Or checkout and cherry-pick described above).

> $ git-fsck --full
> dangling tree c2549a3cdd83098a523cb707f217f4656cde7eb5
>
> $ git prune

Now that was dangerous. When something was unreferenced
(but useful) now it is lost.

Re: Recover broken git repository?

[Nicolas Sebrecht]

The 14/07/09, Florian Breitwieser wrote:

> I have problems with my git repository, attached below are the steps I tried
> to resolve it. But now I am stuck. Is there any good way to recover?

You could git-format-patch your topics, clone a new repo and git-am the
patches.

-- 
Nicolas Sebrecht

Re: Recover broken git repository?

[Jakub Narebski]

Nicolas Sebrecht <nicolas.s.dev@gmx.fr> writes:
> The 14/07/09, Florian Breitwieser wrote:
> 
> > I have problems with my git repository, attached below are the steps I tried
> > to resolve it. But now I am stuck. Is there any good way to recover?
> 
> You could git-format-patch your topics, clone a new repo and git-am the
> patches.

Better solution, if possible, would be to use git-fast-export /
git-fast-import pair.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Recover broken git repository?

[Linus Torvalds]

On Tue, 14 Jul 2009, Florian Breitwieser wrote:
>
> $ git commit -m "Some message"
> error: invalid object 1086b1c606a04bcb78b92d1d411a299d20d18034
> fatal: Error building trees

Hmm. That "invalid object" error comes from 'update_one()' when it cannot 
find the object it is looking for. That, in turn, tends to be an issue of 
the index containing a SHA1 that does not exist in the database.

Do you trust your filesystem? The symptoms really sound like you did a 
"git add .." earlier, and populated the index, but now the object that got 
populated is no longer found.

What OS, what filesystem?  Oh, and what version of git?

> $ git-fsck --full
> dangling tree c2549a3cdd83098a523cb707f217f4656cde7eb5

The fsck seems to imply that things are ok. The fsck _should_ have checked 
the index too.

The 'git commit' obviously disagrees.

Two things to check:

 - do you use grafts to hide old history, rather than connect additional 
   history?

   That can be really dangerous, and will make fsck (and other tools) not 
   look at the hidden state. 

 - do you have alternates (is multiple _different_ repositories tat point 
   to each other)? That can cause problems if you then do things like git 
   prune in them. It wouldn't explain this particular case, but it's 
   something I look out for when I hear about corruption.

> $ git prune

Not a good idea. When you suspect corruption, the _last_ thing you want to 
do is prune things. Who knows what happened?

> $ git commit -m "Some message"
> Created commit e32d5dd: Some message
>  2 files changed, 167 insertions(+), 10 deletions(-)

So now the same commit worked? It _really_ sounds like your filesystem has 
a hard time finding the files it just created. Some local caching issue.

Is it a network filesystem? If so, what's the filesystem server and 
version, if you can find it out?

> $ git push
> Counting objects: 15, done.
> Compressing objects: 100% (6/6), done.
> Writing objects: 100% (8/8), 2.44 KiB, done.
> Total 8 (delta 4), reused 0 (delta 0)
> Unpacking objects: 100% (8/8), done.
> fatal: unresolved deltas left after unpacking
> error: unpack failed: unpacker exited with error code
> To /mnt/extstore2/bioinformatic/git/ppi.git
>  ! [remote rejected] master -> master (n/a (unpacker error))
> error: failed to push some refs to
> '/mnt/extstore2/bioinformatic/git/ppi.git'

Here the other end doesn't have some object that we expect it to have, and 
that we sent a delta to it against. Sounds like potentially the exact same 
problem (it created some new file, but then couldn't see it). Is the thign 
you are pushing to a similar machine with similar filesystems?

We've had issues with both CIFS and NFS. Using

	[core]
		fsyncobjectfiles = true

might work around some issues where the filesystem does crazy things. But 
I'd really like to hear what OS and filesystem versions you're running.

			Linus

Re: Recover broken git repository?

[Stefan-W. Hahn]

Also sprach Linus Torvalds am Tue, 14 Jul 2009 at 10:33:58 -0700:
> 
> 
> On Tue, 14 Jul 2009, Florian Breitwieser wrote:
> >
> > $ git commit -m "Some message"
> > error: invalid object 1086b1c606a04bcb78b92d1d411a299d20d18034
> > fatal: Error building trees
> 
> Hmm. That "invalid object" error comes from 'update_one()' when it cannot 
> find the object it is looking for. That, in turn, tends to be an issue of 
> the index containing a SHA1 that does not exist in the database.
> 
> Do you trust your filesystem? The symptoms really sound like you did a 
> "git add .." earlier, and populated the index, but now the object that got 
> populated is no longer found.
> 
> What OS, what filesystem?  Oh, and what version of git?
> 
> > $ git-fsck --full
> > dangling tree c2549a3cdd83098a523cb707f217f4656cde7eb5
> 
> The fsck seems to imply that things are ok. The fsck _should_ have checked 
> the index too.
> 
> The 'git commit' obviously disagrees.
> 
> Two things to check:
> 
>  - do you use grafts to hide old history, rather than connect additional 
>    history?
> 
>    That can be really dangerous, and will make fsck (and other tools) not 
>    look at the hidden state. 
> 
>  - do you have alternates (is multiple _different_ repositories tat point 
>    to each other)? That can cause problems if you then do things like git 
>    prune in them. It wouldn't explain this particular case, but it's 
>    something I look out for when I hear about corruption.
> 
> > $ git prune
> 
> Not a good idea. When you suspect corruption, the _last_ thing you want to 
> do is prune things. Who knows what happened?
> 
> > $ git commit -m "Some message"
> > Created commit e32d5dd: Some message
> >  2 files changed, 167 insertions(+), 10 deletions(-)
> 
> So now the same commit worked? It _really_ sounds like your filesystem has 
> a hard time finding the files it just created. Some local caching issue.
> 
> Is it a network filesystem? If so, what's the filesystem server and 
> version, if you can find it out?
> 
> > $ git push
> > Counting objects: 15, done.
> > Compressing objects: 100% (6/6), done.
> > Writing objects: 100% (8/8), 2.44 KiB, done.
> > Total 8 (delta 4), reused 0 (delta 0)
> > Unpacking objects: 100% (8/8), done.
> > fatal: unresolved deltas left after unpacking
> > error: unpack failed: unpacker exited with error code
> > To /mnt/extstore2/bioinformatic/git/ppi.git
> >  ! [remote rejected] master -> master (n/a (unpacker error))
> > error: failed to push some refs to
> > '/mnt/extstore2/bioinformatic/git/ppi.git'
> 
> Here the other end doesn't have some object that we expect it to have, and 
> that we sent a delta to it against. Sounds like potentially the exact same 
> problem (it created some new file, but then couldn't see it). Is the thign 
> you are pushing to a similar machine with similar filesystems?
> 
> We've had issues with both CIFS and NFS. Using
> 
> 	[core]
> 		fsyncobjectfiles = true
> 
> might work around some issues where the filesystem does crazy things. But 
> I'd really like to hear what OS and filesystem versions you're running.
> 
> 			Linus
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Stefan-W. Hahn                          It is easy to make things.
/ mailto:stefan.hahn@s-hahn.de /        It is hard to make things simple.			

Please note that according to the German law on data retention,
information on every electronic information exchange with me is
retained for a period of six months.
Bundesgesetzblatt:
http://www.bgblportal.de/BGBL/bgbl1f/bgbl107s3198.pdf

Re: Recover broken git repository?

[Stefan-W. Hahn]

Also sprach Linus Torvalds am Tue, 14 Jul 2009 at 10:33:58 -0700:

Hi,

> > $ git commit -m "Some message"
> > error: invalid object 1086b1c606a04bcb78b92d1d411a299d20d18034
> > fatal: Error building trees

I've trapped in a similar problem few days ago on windows using
cygwin.

A simple "git status" told me an invalid object. Just doing
git status corrupted the index file and only this. The reason was a
wrong configured (unseen reconfigured from another script) mount via
cygwin using "textmode" not "binary mode".

After remounting with "binary mode" a simple "git read-tree HEAD"
repaired it. Just the index file had been corrupted.


Stefan

-- 
Stefan-W. Hahn                          It is easy to make things.
/ mailto:stefan.hahn@s-hahn.de /        It is hard to make things simple.			

Please note that according to the German law on data retention,
information on every electronic information exchange with me is
retained for a period of six months.
Bundesgesetzblatt:
http://www.bgblportal.de/BGBL/bgbl1f/bgbl107s3198.pdf

[PATCH] Improve on the 'invalid object' error message at commit time

[Linus Torvalds]

Not that anybody should ever get it, but somebody did (probably because
of a flaky filesystem, but whatever).  And each time I see an error
message that I haven't seen before, I decide that next time it will look
better.

So this makes us write more relevant information about exactly which
file ended up having issues with a missing object.  Which will tell
whether it was a tree object, for example, or just a regular file in the
index (and which one).

Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---

Not a big deal. But since somebody actually saw this message, let's just 
make it more informative. 

A lot of the "these can't happen unless you're seriously screwed" messages 
aren't very good, because there is little upside when doing development. 
But let's try to improve on them in case they happen in the future.

Not that this would make debugging much easier, but one thing that I 
started wondering about was whether the problem Florian saw was about one 
of the files he had done "git add" on, or whether it was a tree entry that 
was the result of "find_subtree()".

 cache-tree.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/cache-tree.c b/cache-tree.c
index 16a65df..d917437 100644
--- a/cache-tree.c
+++ b/cache-tree.c
@@ -329,7 +329,8 @@ static int update_one(struct cache_tree *it,
 			entlen = pathlen - baselen;
 		}
 		if (mode != S_IFGITLINK && !missing_ok && !has_sha1_file(sha1))
-			return error("invalid object %s", sha1_to_hex(sha1));
+			return error("invalid object %06o %s for '%.*s'",
+				mode, sha1_to_hex(sha1), entlen+baselen, path);
 
 		if (ce->ce_flags & CE_REMOVE)
 			continue; /* entry being removed */

Re: [PATCH] Improve on the 'invalid object' error message at commit time

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> ... one thing that I 
> started wondering about was whether the problem Florian saw was about one 
> of the files he had done "git add" on, or whether it was a tree entry that 
> was the result of "find_subtree()".

Same here.

Historically, cache-tree had issues that same corner case codepaths forgot
invalidating subtrees when they should that could have led to kinds of
bugs that are hard to diagnose.  I think the API got harder to make that
kind of mistakes while we squashed those bugs, though.