Again with git-svn: File was not found in commit

Again with git-svn: File was not found in commit

[David E. Wheeler]

Hello Gits,

I'm looking to migrate [Bricolage][] from Subversion to Git. As such,  
I'm running `git svn` to create a git repository for the Bricolage  
repository. I moved the files around in Subversion quite a while ago  
(the trunk, branches, and tags directories used to be at the root, but  
now they're under a "bricolage" subdirectory), so there's a fair  
amount of duplication, but I don't mind that, because I can prune  
those dupes.

[Bricolage]: http://bricolage.cc/

So I ran `git svn` like so:

     md brictmp
     cd brictmp
     git svn init http://svn.bricolage.cc/bricolage --no-metadata -s
     git config svn.authorsfile ~/Desktop/bricolage_committers.txt
     git svn fetch

After around 30 hours of the fan running full tilt on my MacBook Pro,  
creating 343 branches (yeah, we have a lot of tags), `git svn` exited  
with this error:

     bricolage/branches/rev_1_8/lib/Bric/App/ApacheConfig.pm was not  
found in commit
     e5145931069a511e98a087d4cb1a8bb75f43f899 (r5256)

This seemed strange to me, so I had a look at SVN:

     svn list -r5256 http://svn.bricolage.cc/bricolage/branches/rev_1_8/lib/Bric/App/ApacheConfig.pm
     ApacheConfig.pm

So the file *is* there in that revision. I had been running 1.6.1.2,  
so I upgraded to 1.6.2.2 and ran `git svn fetch` again to see if it  
would pick up where it left off, but it returned the same error  
(please don't tell me I have to start over!).

I see that this issue has come up [before][], earlier this year; has  
there been any progress on it? Is there perhaps something I can do to  
get this working?

[before]: http://lists-archives.org/git/677591-git-svn-file-was-not-found-in-commit.html

Many thanks,

David

Re: Again with git-svn: File was not found in commit

[Eric Wong]

"David E. Wheeler" <david@kineticode.com> wrote:
> After around 30 hours of the fan running full tilt on my MacBook Pro,  
> creating 343 branches (yeah, we have a lot of tags), `git svn` exited  
> with this error:
>
>     bricolage/branches/rev_1_8/lib/Bric/App/ApacheConfig.pm was not  
> found in commit
>     e5145931069a511e98a087d4cb1a8bb75f43f899 (r5256)
>
> This seemed strange to me, so I had a look at SVN:
>
>     svn list -r5256 http://svn.bricolage.cc/bricolage/branches/rev_1_8/lib/Bric/App/ApacheConfig.pm
>     ApacheConfig.pm
>
> So the file *is* there in that revision. I had been running 1.6.1.2, so I 
> upgraded to 1.6.2.2 and ran `git svn fetch` again to see if it would pick 
> up where it left off, but it returned the same error (please don't tell 
> me I have to start over!).

Interesting.  I finally managed to reproduce it over the weekend.  I'll
try to look into it later tonight.

-- 
Eric Wong

Re: Again with git-svn: File was not found in commit

[David E. Wheeler]

On Apr 21, 2009, at 11:02 AM, Eric Wong wrote:

>>    bricolage/branches/rev_1_8/lib/Bric/App/ApacheConfig.pm was not
>> found in commit
>>    e5145931069a511e98a087d4cb1a8bb75f43f899 (r5256)
>>
>> This seemed strange to me, so I had a look at SVN:
>>
>>    svn list -r5256 http://svn.bricolage.cc/bricolage/branches/rev_1_8/lib/Bric/App/ApacheConfig.pm
>>    ApacheConfig.pm
>>
>> So the file *is* there in that revision. I had been running  
>> 1.6.1.2, so I
>> upgraded to 1.6.2.2 and ran `git svn fetch` again to see if it  
>> would pick
>> up where it left off, but it returned the same error (please don't  
>> tell
>> me I have to start over!).
>
> Interesting.  I finally managed to reproduce it over the weekend.   
> I'll
> try to look into it later tonight.

Thanks, appreciate it. I've gotten around it by going back to the old  
CVS repository on SourceForge and migrating that, and then migrating  
the SVN repo only from after we moved to SVN. Then I'll just have to  
stitch the CVS and SVN migrations together and I'll be done.

Best,

David

Re: Again with git-svn: File was not found in commit

[Eric Wong]

"David E. Wheeler" <david@kineticode.com> wrote:
> On Apr 21, 2009, at 11:02 AM, Eric Wong wrote:
>
>>>    bricolage/branches/rev_1_8/lib/Bric/App/ApacheConfig.pm was not
>>> found in commit
>>>    e5145931069a511e98a087d4cb1a8bb75f43f899 (r5256)
>>>
>>> This seemed strange to me, so I had a look at SVN:
>>>
>>>    svn list -r5256 http://svn.bricolage.cc/bricolage/branches/rev_1_8/lib/Bric/App/ApacheConfig.pm
>>>    ApacheConfig.pm
>>>
>>> So the file *is* there in that revision. I had been running 1.6.1.2, 
>>> so I
>>> upgraded to 1.6.2.2 and ran `git svn fetch` again to see if it would 
>>> pick
>>> up where it left off, but it returned the same error (please don't  
>>> tell
>>> me I have to start over!).
>>
>> Interesting.  I finally managed to reproduce it over the weekend.   
>> I'll
>> try to look into it later tonight.
>
> Thanks, appreciate it. I've gotten around it by going back to the old  
> CVS repository on SourceForge and migrating that, and then migrating the 
> SVN repo only from after we moved to SVN. Then I'll just have to stitch 
> the CVS and SVN migrations together and I'll be done.

Hopefully the workaround is working out for you.

The problem with r5256 is quite nasty and I can't think of a good
automated solution right now involving --stdlayout/branches/tags

------------------------------------------------------------------------
r5256 | theory | 2004-05-09 19:17:49 -0700 (Sun, 09 May 2004) | 2 lines
Changed paths:
   A /bricolage/branches/rev_1_8 (from /bricolage/branches/rev_1_8_temp/bricolage:5255)
   D /bricolage/branches/rev_1_8_temp/bricolage

Permanent home.

------------------------------------------------------------------------

The problem is that the "bricolage" subdirectory became the top-level
tree and git-svn doesn't have a way of detecting that.  This is
occasionally a problem with other repos, too

Overall, this repo is very nasty given the amount of times follow-parent
needs to be used (refs with a "@<revno>" suffix).  The --stdlayout
switch is really for the common repositories with a consistent depth for
branches/tags.  Unfortunately, SVN is way too freeform in this regard
and mixed-depth branches/tags hierarchies have always been a problem :<

It might be a bit easier to write (or script) the "fetch" lines
(like the one --stdlayout generates for trunk) for each tag/branch
and maybe use --no-follow-parent.

-- 
Eric Wong

Re: Again with git-svn: File was not found in commit

[David E. Wheeler]

On Apr 25, 2009, at 6:53 PM, Eric Wong wrote:

> Hopefully the workaround is working out for you.

[Finally], yes.

[Finally]: http://justatheory.com/computers/vcs/git/bricolage-to-git.html

> The problem with r5256 is quite nasty and I can't think of a good
> automated solution right now involving --stdlayout/branches/tags
>
> ------------------------------------------------------------------------
> r5256 | theory | 2004-05-09 19:17:49 -0700 (Sun, 09 May 2004) | 2  
> lines
> Changed paths:
>   A /bricolage/branches/rev_1_8 (from /bricolage/branches/ 
> rev_1_8_temp/bricolage:5255)
>   D /bricolage/branches/rev_1_8_temp/bricolage
>
> Permanent home.
>
> ------------------------------------------------------------------------
>
> The problem is that the "bricolage" subdirectory became the top-level
> tree and git-svn doesn't have a way of detecting that.  This is
> occasionally a problem with other repos, too

We moved shit around a *lot*. There were two times we moved things  
around: First, when Bricolage was first imported from CVS, it assumed  
that the root directory was the project directory, even though we had  
several projects. So we had

/trunk/
/branches/
/tags/

We spent several hundred commits moving things around so that we would  
have the more standard:

/bricolage/trunk/
/bricolage/branches/
/bricolage/tags/

Fortunately, we did all of this moving before we committed anything  
useful to Subversion. So for this project, I first converted the old  
SourceForge CVS repository to Git, then converted the Subversion  
repository starting from after all this moving around (r5517, well  
after the problematic commit), and then I joined them together, as  
detailed in my blog post.

> Overall, this repo is very nasty given the amount of times follow- 
> parent
> needs to be used (refs with a "@<revno>" suffix).  The --stdlayout
> switch is really for the common repositories with a consistent depth  
> for
> branches/tags.  Unfortunately, SVN is way too freeform in this regard
> and mixed-depth branches/tags hierarchies have always been a  
> problem :<

Tell me about it. This is why it took 30 hours before `git-svn` failed.

The other time we moved stuff around was to reorganize tags into  
subdirectories. This was because we had so many tags that they were  
unwieldy as a directory listing in SVN. This turned out to be a PITA  
when migrating, but I was able to [work around it] using GitX and some  
pretty hacky code to generate grafts. It'd be nice if there was a way  
to tell git-svn how deep a hierarchy to look for for tags (and perhaps  
branches), although it wouldn't have helped us, because we *changed*  
that hierarchy.

[work around it]: http://justatheory.com/computers/vcs/git/bricolage-svn-to-git.html

I'm so glad to have it all in Git now where things are far more  
rationally organized.

> It might be a bit easier to write (or script) the "fetch" lines
> (like the one --stdlayout generates for trunk) for each tag/branch
> and maybe use --no-follow-parent.

--no-follow-parent was indeed key for me being able to get this done.  
Thanks for writing such a capable client that, although it couldn't  
reconstruct what we were thinking when we moved things around, it let  
me tell it where to stop and what stuff to ignore (ultimately, I had  
it ignore all tag directories) so that I could get the job done.

The new Git repository is now on [GitHub], if you're curious.

[GitHub]: http://github.com/bricoleurs/bricolage/

Thanks,

David

Re: Again with git-svn: File was not found in commit

[Jakub Narebski]

"David E. Wheeler" <david@kineticode.com> writes:

> On Apr 25, 2009, at 6:53 PM, Eric Wong wrote:
> 
> > Hopefully the workaround is working out for you.
> 
> [Finally], yes.
> 
> [Finally]: http://justatheory.com/computers/vcs/git/bricolage-to-git.html

Thank you for posting this detailed analysis.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Again with git-svn: File was not found in commit

[David E. Wheeler]

On Apr 26, 2009, at 1:17 AM, Jakub Narebski wrote:

>> [Finally]: http://justatheory.com/computers/vcs/git/bricolage-to-git.html
>
> Thank you for posting this detailed analysis.

My pleasure; hopefully someone will get something out of it to help  
with *their* migration. I'll likely publish some final notes and the  
scripts I wrote sometime this week.

Best,

David

Re: Again with git-svn: File was not found in commit

[Jakub Narebski]

"David E. Wheeler" <david@kineticode.com> writes:

> On Apr 26, 2009, at 1:17 AM, Jakub Narebski wrote:
> 
>>> [Finally]: http://justatheory.com/computers/vcs/git/bricolage-to-git.html
>>
>> Thank you for posting this detailed analysis.
> 
> My pleasure; hopefully someone will get something out of it to help
> with *their* migration. I'll likely publish some final notes and the
> scripts I wrote sometime this week.

BTW the above story was mentioned in "Perlbuzz news roundup 2009-04-27"
http://perlbuzz.com/2009/04/perlbuzz-news-roundup-2009-04-27.html

P.S. Any relation to David A. Wheeler (http://www.dwheeler.com/blog)?
-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Again with git-svn: File was not found in commit

[David E. Wheeler]

On Apr 28, 2009, at 2:40 AM, Jakub Narebski wrote:

> BTW the above story was mentioned in "Perlbuzz news roundup  
> 2009-04-27"
> http://perlbuzz.com/2009/04/perlbuzz-news-roundup-2009-04-27.html
>
> P.S. Any relation to David A. Wheeler (http://www.dwheeler.com/blog)?

No, just my doppelgänger. :-)

Best,

David