Debugging corrupt object generation

Debugging corrupt object generation

[Julian Phillips]

I have been trying to import a repository using fast-import, and as part 
of this I have extended fast-import to support copying a path from any 
arbitrary commit.  This works fine on my small test repository.

However, when I run against my (vastly more complicated) real work 
repository things don't go so well.  I get a few thousand commits in, and 
then it breaks.  It appears that I have somehow managed to create a 
corrupt object or something.

After a certain commit, and attempt to use that commit generates a "fatal:
unable to apply delta".  This appears to be coming from unpack_delta_entry
in sha1_file.c.

Can anyone give me any hints as to how I find out what is causing the
problem?  I'm not even sure what it is that isn't working ... and all
attempts to replicate the problem with my test repository have failed.

-- 
Julian

  ---
...the prevailing Catholic odor - incense, wax, centuries of mild bleating
from the lips of the flock.
-- Thomas Pynchon, _Gravity's Rainbow_

Re: Debugging corrupt object generation

[Nicolas Pitre]

On Thu, 1 Nov 2007, Julian Phillips wrote:

> I have been trying to import a repository using fast-import, and as part of
> this I have extended fast-import to support copying a path from any arbitrary
> commit.  This works fine on my small test repository.
> 
> However, when I run against my (vastly more complicated) real work repository
> things don't go so well.  I get a few thousand commits in, and then it breaks.
> It appears that I have somehow managed to create a corrupt object or
> something.
> 
> After a certain commit, and attempt to use that commit generates a "fatal:
> unable to apply delta".  This appears to be coming from unpack_delta_entry
> in sha1_file.c.

I suppose you mean "fatal: failed to apply delta", because "unable to 
apply delta" doesn't appear anywhere in the current source tree.

> Can anyone give me any hints as to how I find out what is causing the
> problem?  I'm not even sure what it is that isn't working ... and all
> attempts to replicate the problem with my test repository have failed.

Well, something is screwed for sure.  Some object you're requesting is 
made of a delta, and that delta is bad, therefore patch_delta() returns 
NULL (you could instrument it to determine exactly why).

Maybe fast-import hasn't flushed the needed data to the pack yet?


Nicolas

Re: Debugging corrupt object generation

[Julian Phillips]

On Thu, 1 Nov 2007, Nicolas Pitre wrote:

> On Thu, 1 Nov 2007, Julian Phillips wrote:
>
>> After a certain commit, and attempt to use that commit generates a "fatal:
>> unable to apply delta".  This appears to be coming from unpack_delta_entry
>> in sha1_file.c.
>
> I suppose you mean "fatal: failed to apply delta", because "unable to
> apply delta" doesn't appear anywhere in the current source tree.

I did indeed.  Read 'n' type never seems to be as reliable as copy 'n' 
paste ...

>> Can anyone give me any hints as to how I find out what is causing the
>> problem?  I'm not even sure what it is that isn't working ... and all
>> attempts to replicate the problem with my test repository have failed.
>
> Well, something is screwed for sure.  Some object you're requesting is
> made of a delta, and that delta is bad, therefore patch_delta() returns
> NULL (you could instrument it to determine exactly why).

Ah, ok - that sounds like it might be useful.  I guess that looking at the 
opposite side where fast-import is creating the data might also prove 
fruitful.

> Maybe fast-import hasn't flushed the needed data to the pack yet?

Well, fast-import completes quite happily and outputs the normal summary 
status.  I can look at logs and trees etc provided that I don't try and 
look at one particular part of the tree on one particular commit.  I 
think the problem is that I've managed to do something inside fast-import 
that corrupts one particular tree object (though I've no idea what that 
might be).

I shall have to start instrumenting and try again.

Thanks.

-- 
Julian

  ---
Many changes of mind and mood; do not hesitate too long.

Re: Debugging corrupt object generation

[Shawn O. Pearce]

Julian Phillips <julian@quantumfyre.co.uk> wrote:
> On Thu, 1 Nov 2007, Nicolas Pitre wrote:
> >
> >Maybe fast-import hasn't flushed the needed data to the pack yet?
> 
> Well, fast-import completes quite happily and outputs the normal summary 
> status.  I can look at logs and trees etc provided that I don't try and 
> look at one particular part of the tree on one particular commit.  I 
> think the problem is that I've managed to do something inside fast-import 
> that corrupts one particular tree object (though I've no idea what that 
> might be).

Ahhh.  I'm betting you messed up the version 0 and version 1 arrays
inside of the struct tree_entry.  This could cause the delta
generator to look at the wrong base information when it creates
the delta, thus causing the delta to be created for a different
base than what the object is actually using in the packfile.

The version[0] is meant to hold the mode and SHA-1 of the tree_entry
in the base object.  The version[1] is meant to hold the current
mode and SHA-1 of the tree_entry in the new object.  A mode of 0
means the entry doesn't exist in that particular tree; so an add
is shown as "version[0].mode = 0; version[1].mode = 0100644".

Look at the store_tree function, this is where we regenerate the
canonical representation of both the version[0] and the version[1]
trees so the store_object function can generate a delta.  Note that
we assume the base object name is root->versions[0].sha1.  Maybe the
version[0] array doesn't actually match the tree named by that sha1?

-- 
Shawn.