Tips for debugging fast-import front-ends

Tips for debugging fast-import front-ends

[Julian Phillips]

I was wondering if anyone had some handy tips for debugging fast-import 
based importers?

I'm trying to import an SVN repository that was initially created by 
cvs2svn and, since we started using branches quite heavily _before_ 
switching to SVN, there is some quite messy history.

At some point in the "messy" phase I get an error from fast-import saying:

fatal: path foo not in branch

This seems to happen when I attempt to copy a file that is also changed in 
the same commit.  What I would like to do is find out what things look 
like just before that point - the problem is that fast-import seems to not 
create anything on failure.  So, apart restarting the import and stopping 
just before the broken commit, is there any way to find out what 
fast-import thinks the branch looks like?

Also, does anyone have any handy hints 'n' tips for working with 
fast-import?

-- 
Julian

  ---
Air pollution is really making us pay through the nose.

Re: Tips for debugging fast-import front-ends

[Johannes Sixt]

Julian Phillips wrote:
> At some point in the "messy" phase I get an error from fast-import saying:
> 
> fatal: path foo not in branch
> 
> This seems to happen when I attempt to copy a file that is also changed in
> the same commit.  What I would like to do is find out what things look
> like just before that point - the problem is that fast-import seems to not
> create anything on failure.  So, apart restarting the import and stopping
> just before the broken commit, is there any way to find out what
> fast-import thinks the branch looks like?

Redirect the input stream to a file instead of to fast-import, then feed
fast-import from this file. This way you can reproduce the failure more
quickly. Also, you can run fast-import from a debugger.

-- Hannes

Re: Tips for debugging fast-import front-ends

[Shawn O. Pearce]

Johannes Sixt <J.Sixt@eudaptics.com> wrote:
> Julian Phillips wrote:
> > At some point in the "messy" phase I get an error from fast-import saying:
> > 
> > fatal: path foo not in branch
> > 
> > This seems to happen when I attempt to copy a file that is also changed in
> > the same commit.  What I would like to do is find out what things look
> > like just before that point - the problem is that fast-import seems to not
> > create anything on failure.  So, apart restarting the import and stopping
> > just before the broken commit, is there any way to find out what
> > fast-import thinks the branch looks like?
> 
> Redirect the input stream to a file instead of to fast-import, then feed
> fast-import from this file. This way you can reproduce the failure more
> quickly. Also, you can run fast-import from a debugger.

I tend to dump my test stream to a file, then process that through
fast-import.  Running fast-import in the debugger is uh, ugly, as
I make heavy use of FLEX_ARRAY whenever there is interesting data.
So simple printing of any useful data (like the name of a tree
entry) is not so simple without doing pointer addition and type
casting in gdb.  Sometimes I just break down and add a few targeted
printf()'s to the code.  That always tells me what's up.

One easy thing that you can do is add a 'checkpoint' command
right before the bad commit (assuming you know roughly where it is
breaking).  This will cause fast-import to flush all of its buffers
back to disk, and update the Git repository, before reading the
next command.  The result is that when you crash you have everything
that fast-import processed right up to that last checkpoint.

Clearly doing too many checkpoints is going to remove the fast part
of fast-import, but at least you have a -import.  ;-)


After reading this thread I have decided to add some frontend
debugging support into fast-import, like dumping out the most
recently received commands and the current branch and tree state
before it die()'s with a protocol error or bad path error, and maybe
also try to do a partial checkpoint (close out the packfile, don't
update refs or marks, but instead write them to the state dump file).
This way the frontend developer can inspect the overall state and
better understand what might have gone wrong.

Unfortunately I won't get it done in the next few days, but maybe
before the weekend.

-- 
Shawn.