Recovering from missing objects?

Recovering from missing objects?

[Geert Uytterhoeven]

	Hi all,

I start to see suspicious messages about missing objects in one of my working
repositories:

| vixen$ git gc
| error: Could not read c406ab0be69c912ea59233595a071478103cdad8
| fatal: bad tree object c406ab0be69c912ea59233595a071478103cdad8
| error: failed to run repack
| vixen$ 

My setup:
  - I have one reference repository (cloned from Linus' linux-2.6.git)
  - I have several working repositories, cloned using --reference to my
    reference repository. A working repository has several remotes (cloned from
    other Linux kernel repositories).

I always do a `git pull' in the reference repository, before doing a `git
remote update' in a working repository. When I do `git gc' in a working
repository, it cleans up all objects that are not in the reference repository.
Hence I only need to care about backup of the .git directories in the working
repositories (the stuff I'm working on), and not about the reference
repository (its objects are publicly available and replicated all over the
world).

I identified the missing object listed above to be part of a remote repository.
Doing a `git remote update' doesn't fetch it again, as git is too smart and
thinks I already have everything.

If I clone the remote repository, I have the object in the new clone.
However, how do I get the missing object back into the .git directory of my
working repository? Is there an easy way to do that, or should I just recreate
my working repository from scratch, and reimport anything I can recover?

As for the cause of the problem: most probably I once added the remote
repository to my reference repository, and removed it later, causing all
objects to be removed during a subsequent gc.

Thanks for your answers!

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/

A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010

Re: Recovering from missing objects?

[Junio C Hamano]

Geert Uytterhoeven <Geert.Uytterhoeven@sonycom.com> writes:

> 	Hi all,
>
> I start to see suspicious messages about missing objects in one of my working
> repositories:
>
> | vixen$ git gc
> | error: Could not read c406ab0be69c912ea59233595a071478103cdad8
> | fatal: bad tree object c406ab0be69c912ea59233595a071478103cdad8
> | error: failed to run repack
> | vixen$ 
>
> My setup:
>   - I have one reference repository (cloned from Linus' linux-2.6.git)
>   - I have several working repositories, cloned using --reference to my
>     reference repository. A working repository has several remotes (cloned from
>     other Linux kernel repositories).
>
> I always do a `git pull' in the reference repository, before doing a `git
> remote update' in a working repository. When I do `git gc' in a working
> repository, it cleans up all objects that are not in the reference repository.
> Hence I only need to care about backup of the .git directories in the working
> repositories (the stuff I'm working on), and not about the reference
> repository (its objects are publicly available and replicated all over the
> world).
>
> I identified the missing object listed above to be part of a remote repository.
> Doing a `git remote update' doesn't fetch it again, as git is too smart and
> thinks I already have everything.
>
> If I clone the remote repository, I have the object in the new clone.
> However, how do I get the missing object back into the .git directory of my
> working repository?

In the new clone:

	$ IT=c406ab0be69c912ea59233595a071478103cdad8
	$ TYPE=$(git cat-file -t $IT)
        $ git cat-file $TYPE $IT >/var/tmp/$IT.raw

Go to the repository that lacks the object and then

	$ git hash-object -t $TYPE -w --stdin </var/tmp/$IT.raw

After that you may find objects that $IT needs to reference.  You can
obviously repeat the above procedure until you have nothing missing.


I also suspect you could do this instead; I haven't thought things through
and that is why I say "suspect" but this is safe (i.e. not destructive)
and may worth a try.

In the new clone:

	$ IT=c406ab0be69c912ea59233595a071478103cdad8
	$ H=$(git rev-list --objects $IT | git pack-objects mine)
        $ mv mine-$H.pack /var/tmp

Go to the repository that lacks the object and then

	$ git unpack-objects </var/tmp/mine-$H.pack

Re: Recovering from missing objects?

[Jeff King]

On Thu, Feb 19, 2009 at 04:29:12PM -0800, Junio C Hamano wrote:

> > If I clone the remote repository, I have the object in the new clone.
> > However, how do I get the missing object back into the .git directory of my
> > working repository?
> [...]
> In the new clone:
> 
> 	$ IT=c406ab0be69c912ea59233595a071478103cdad8
> 	$ H=$(git rev-list --objects $IT | git pack-objects mine)
>         $ mv mine-$H.pack /var/tmp
> 
> Go to the repository that lacks the object and then
> 
> 	$ git unpack-objects </var/tmp/mine-$H.pack

Might it not be simpler to just copy or hardlink the pack from the new
clone into the old directory's .git/objects/pack? That will get more
than you need, but things should start working, at which point a "git
repack -a -d" will make it small again.

Or am I misunderstanding something?

-Peff

Re: Recovering from missing objects?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Might it not be simpler to just copy or hardlink the pack from the new
> clone into the old directory's .git/objects/pack? That will get more
> than you need, but things should start working, at which point a "git
> repack -a -d" will make it small again.
>
> Or am I misunderstanding something?

If you already have a good clone and a half-broken one locally, of course,
what you said is the easiest.

I just assumed that Geert did not really want to copy the whole pack.
Maybe he doesn't mind in this particular case, but the next breakage may
involve gigapacks he'd rather not re-clone.

I also assumed that anybody who is reading the message can easily guess
that the copy I was demonstrating in the description could be done across
machines, instead of via local /var/tmp/, and "In the new clone" steps
could even be done in the original one.  Presenting the knowledge that
way, the solution hopefully would be adjustable for more people who are
reading this thread.

Re: Recovering from missing objects?

[Jeff King]

On Thu, Feb 19, 2009 at 10:09:20PM -0800, Junio C Hamano wrote:

> > Might it not be simpler to just copy or hardlink the pack from the new
> > clone into the old directory's .git/objects/pack? That will get more
> > than you need, but things should start working, at which point a "git
> > repack -a -d" will make it small again.
> >
> > Or am I misunderstanding something?
> 
> If you already have a good clone and a half-broken one locally, of course,
> what you said is the easiest.
> 
> I just assumed that Geert did not really want to copy the whole pack.
> Maybe he doesn't mind in this particular case, but the next breakage may
> involve gigapacks he'd rather not re-clone.

Ah, OK. I had the impression that he had already made a new valid clone
on the local box.

> I also assumed that anybody who is reading the message can easily guess
> that the copy I was demonstrating in the description could be done across
> machines, instead of via local /var/tmp/, and "In the new clone" steps
> could even be done in the original one.  Presenting the knowledge that
> way, the solution hopefully would be adjustable for more people who are
> reading this thread.

Makes sense. Thanks.

-Peff

Re: Recovering from missing objects?

[Geert Uytterhoeven]

	Hi Junio,

On Thu, 19 Feb 2009, Junio C Hamano wrote:
> Geert Uytterhoeven <Geert.Uytterhoeven@sonycom.com> writes:
> > I start to see suspicious messages about missing objects in one of my working
> > repositories:
> >
> > | vixen$ git gc
> > | error: Could not read c406ab0be69c912ea59233595a071478103cdad8
> > | fatal: bad tree object c406ab0be69c912ea59233595a071478103cdad8
> > | error: failed to run repack
> > | vixen$ 
> >
> > I identified the missing object listed above to be part of a remote repository.
> > Doing a `git remote update' doesn't fetch it again, as git is too smart and
> > thinks I already have everything.
> >
> > If I clone the remote repository, I have the object in the new clone.
> > However, how do I get the missing object back into the .git directory of my
> > working repository?
> 
> In the new clone:
> 
> 	$ IT=c406ab0be69c912ea59233595a071478103cdad8
> 	$ TYPE=$(git cat-file -t $IT)
>         $ git cat-file $TYPE $IT >/var/tmp/$IT.raw
> 
> Go to the repository that lacks the object and then
> 
> 	$ git hash-object -t $TYPE -w --stdin </var/tmp/$IT.raw
> 
> After that you may find objects that $IT needs to reference.  You can
> obviously repeat the above procedure until you have nothing missing.

Thanks! It worked fine for most of the missing objects.

For some of them (type commit), I still get:

| vixen$ git show 32582324956483840d1ae90726bbe879cc48f63d
| fatal: unable to read destination tree (32582324956483840d1ae90726bbe879cc48f63d
| vixen$ 

after importing it from the other repository.

> I also suspect you could do this instead; I haven't thought things through
> and that is why I say "suspect" but this is safe (i.e. not destructive)
> and may worth a try.
> 
> In the new clone:
> 
> 	$ IT=c406ab0be69c912ea59233595a071478103cdad8
> 	$ H=$(git rev-list --objects $IT | git pack-objects mine)
>         $ mv mine-$H.pack /var/tmp
> 
> Go to the repository that lacks the object and then
> 
> 	$ git unpack-objects </var/tmp/mine-$H.pack

That one did the trick for the "stubborn" commits I couldn't get imported the
other way.

BTW, I kept a copy of the repository with the missing commit
32582324956483840d1ae90726bbe879cc48f63d, just in case you have another idea to
revive that single one without creating a (big) pack ;-)

Anyway, I think it wouldn't hurt to have an option for "git remote update" to
retrieve all lost commits from the remote...

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/

A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010