[GSoC update] git-remote-svn: The final one

[GSoC update] git-remote-svn: The final one

[Ramkumar Ramachandra]

Hi,

The GSoC is over. Not that it changes anything; I'm still
here. Anyway, here's a report; this isn't a movie, so don't skip the
credits at the end of the email!

Admittedly, I couldn't deliver everything I'd promised in the
beginning, mainly because I misjudged the size/ difficulty of the
project. While I thought I'd spend most of my time in Git-land
converting and mapping histories, I ended up spending a huge amount of
time with Subversion writing svnrdump. The upside is that we now have
an excellent dumpfile importer/ exporter merged into the Subversion
trunk, while the downside is that we don't have a fully functional
remote helper for Subversion yet. In all, git-remote-svn involved
(involves) the following things:

1. Getting the revision history out of Subversion in a sane format
   (dumpfile v3). This turned out to be MUCH harder than expected;
   fortunately, svnrdump does this better than anything else.
2. Converting the revision history from dumpfile v3 format into a git
   fast-import stream. svn-dump-fast-export does this better than
   anything else. In fact, we've even come up with a proof that
   nothing can even theoretically do better. Unfortunately, it only
   does dumpfile v2 so far, and I still haven't managed to extend it
   to do dumpfile v3.
3. Getting revision history into Subversion. The load functionality in
   svnrdump that I wrote after the mid-term does a good job of this. I
   don't know of any tool that even does something similar.
4. Converting the revision history from a fast-export stream to
   dumpfile v3. I've just started thinking about this, and haven't
   started work on it- it's quite a non-trivial challenge.
5. Stitching everything together with a remote helper
   application. This is much easier than I thought- I wrote a remote
   helper in one afternoon [1], but didn't get it merged because it's
   missing the supporting infrastructure.
6. A branch/tag mapper. I dropped this due to lack of time. It's last
   priority even now, and I'll work on it only when the rest of the
   infrastructure is in place.

I had certain objectives that I'd personally set out to meet in the
beginning of the GSoC. Here they are in brief, and my personal
evaluation of each one. Do feel free to criticize the approach I've
followed and the trade-offs that I've chosen to make.

1. Integration. I was determined to get *everything* merged. I think
   I've been largely successful on this front.
2. Code quality. In terms of bug-fixing and polish, I'm happy with the
   current state of the code, thanks to the sheer number of code
   reviews.
3. Community interaction. I was determined to post atleast once a week
   to the list, keep everyone in the loop, and remain active all the
   time. I think I've done justice to this.
4. Off-topic work. This may sound weird; In addition to my GSoC work, I
   was determined to do some other off-topic work as well to get the
   hang of contributing and participating in wider
   discussions. Although I couldn't pay full attention to this due to
   my heavy workload, I did manage to get some minor OT patches in.
5. Completion. I'm lagging heavily on this front.

On a final note, thanks to everyone for helping! Off the top of my
head, the following people have been especially awesome (in no
particular order). I'm sorry if I've forgotten to include someone (!)

1. Jonathan Nieder - for his painfully detailed reviews, constant
   cleanups and re-rolls.
2. David Barr - for authoring svn-dump-fast-export, without which a
   lot of what we've achieved wouldn't have been possible.
3. Daniel Shahaf - for his endless support while I was writing
   svnrdump. I couldn't have finished it without him.
4. Stefan Sperling - for his detailed review of especially of the
   first iteration of svnrdump, which got things moving in the first
   place.
5. Bert Huijben - for his innumerable contributions and killer
   bugfixes!
6. Sverre Rabbelier - for being an awesome mentor, and being
   supportive throughout.
7. Will Palmer - for writing command-line parsing UI and a test script
   for svnrdump.
8. Sam Vilain - for his writeup on the challenges of mapping in
   Subversion. I realized the challenges involved, and decided not to
   pursue this in the SoC term.
9. Philip Martin - for authoring svnmucc, which served as the
   inspiration for the load functionality in svnrdump.
10. Jonas Gehrig - for authoring rsvndump, which served as the
    inspiration for the dump functionality in svnrdump.
11. Ævar Arnfjörð Bjarmason - for getting me access to a powerful
    server to run lots of validation tests; I ended up fixing many
    bugs that I wouldn't have otherwise been able to do.

Thanks for reading.

[1] http://thread.gmane.org/gmane.comp.version-control.git/147715

-- Ram

Re: [GSoC update] git-remote-svn: The final one

[Stephen Bash]

Ram-

First, congratulations on making it through GSoC.  I've been avidly following your progress since our off-list discussion at the beginning of the summer, and I'm excited to watch the work continue.

> 1. Getting the revision history out of Subversion in a sane format
> (dumpfile v3).
>      ...
> 2. Converting the revision history from dumpfile v3 format into a git
> fast-import stream.
>      ...
> 5. Stitching everything together with a remote helper
> application.
>      ...
> 6. A branch/tag mapper.

So being selfish, what I care about is an improved method for converting SVN repos to git repos once and never going back.  It sounds to me like the major hurdles remaining there are getting svn-fe to handle dumpfile v3 and handling branches/tags.  Is that correct?

Thanks, and keep up the good work!

Stephen

Re: [GSoC update] git-remote-svn: The final one

[Ramkumar Ramachandra]

Hi Stephen,

Stephen Bash writes:
> Ram-
> 
> First, congratulations on making it through GSoC.  I've been avidly following your progress since our off-list discussion at the beginning of the summer, and I'm excited to watch the work continue.

Thank you :)

> > 1. Getting the revision history out of Subversion in a sane format
> > (dumpfile v3).
> >      ...
> > 2. Converting the revision history from dumpfile v3 format into a git
> > fast-import stream.
> >      ...
> > 5. Stitching everything together with a remote helper
> > application.
> >      ...
> > 6. A branch/tag mapper.
> 
> So being selfish, what I care about is an improved method for converting SVN repos to git repos once and never going back.  It sounds to me like the major hurdles remaining there are getting svn-fe to handle dumpfile v3 and handling branches/tags.  Is that correct?

We're doing better than that- when the whole thing is complete, we
should have perfectly seamless two-way compatibility. But yes, you're
right- there are several non-trivial tasks to handle, the top three of
which are:
1. Making svn-fe git-aware (and hence handle dumpfile v3)
2. Writing a fast-export to dumpfilev3 converter
3. Implementing the branch/ tag mapper in an extensible way

Don't worry about it though- we're all working hard on it :)

-- Ram

Re: [GSoC update] git-remote-svn: The final one

[Erik Faye-Lund]

On Tue, Aug 17, 2010 at 5:59 PM, Ramkumar Ramachandra
<artagnon@gmail.com> wrote:
> Stephen Bash writes:
>> First, congratulations on making it through GSoC.  I've been avidly following your progress since our off-list discussion at the beginning of the summer, and I'm excited to watch the work continue.

+1 on the congratulation! I've also been following the progress, but
from a distance. As I use git-svn to work with SVN at $dayjob, I can't
wait to get something running.

> We're doing better than that- when the whole thing is complete, we
> should have perfectly seamless two-way compatibility.

Does this include handling SVN repos where the layout have changed
during it's lifetime? If so, how will this look on the git side?

Re: [GSoC update] git-remote-svn: The final one

[Sverre Rabbelier]

Heya,

On Tue, Aug 17, 2010 at 11:10, Erik Faye-Lund <kusmabite@gmail.com> wrote:
> On Tue, Aug 17, 2010 at 5:59 PM, Ramkumar Ramachandra
> <artagnon@gmail.com> wrote:
>> We're doing better than that- when the whole thing is complete, we
>> should have perfectly seamless two-way compatibility.
>
> Does this include handling SVN repos where the layout have changed
> during it's lifetime? If so, how will this look on the git side?

Perhaps it makes sense to start a new thread to hash that out for when
the time comes to implement that?

-- 
Cheers,

Sverre Rabbelier

Re: [GSoC update] git-remote-svn: The final one

[Ramkumar Ramachandra]

Hi Sverre,

Sverre Rabbelier writes:
> Heya,
> 
> On Tue, Aug 17, 2010 at 11:10, Erik Faye-Lund <kusmabite@gmail.com> wrote:
> > On Tue, Aug 17, 2010 at 5:59 PM, Ramkumar Ramachandra
> > <artagnon@gmail.com> wrote:
> >> We're doing better than that- when the whole thing is complete, we
> >> should have perfectly seamless two-way compatibility.
> >
> > Does this include handling SVN repos where the layout have changed
> > during it's lifetime? If so, how will this look on the git side?
> 
> Perhaps it makes sense to start a new thread to hash that out for when
> the time comes to implement that?

Good idea. I've posted a new timeline. I'll put up some branch/ tag
mapper notes next week.

-- Ram

Re: [GSoC update] git-remote-svn: The final one

[Ramkumar Ramachandra]

Hi Erik,

Erik Faye-Lund writes:
> On Tue, Aug 17, 2010 at 5:59 PM, Ramkumar Ramachandra
> <artagnon@gmail.com> wrote:
> > Stephen Bash writes:
> >> First, congratulations on making it through GSoC.  I've been avidly following your progress since our off-list discussion at the beginning of the summer, and I'm excited to watch the work continue.
> 
> +1 on the congratulation! I've also been following the progress, but
> from a distance. As I use git-svn to work with SVN at $dayjob, I can't
> wait to get something running.

Thanks :)

> > We're doing better than that- when the whole thing is complete, we
> > should have perfectly seamless two-way compatibility.
> 
> Does this include handling SVN repos where the layout have changed
> during it's lifetime? If so, how will this look on the git side?

Yes, but we don't know yet. There's been some speculation about
re-implementing the branch/ tag mapper in git-svn on top of
git-remote-svn once it's complete. Anyway, Sam's illustration has been
most insightful to me on this issue [1]. Will has also started
thinking about this and has some preliminary notes outlining the
challenges. When the notes are more mature, I'll post them to the list
so we can get a wider audience to discuss this at length :)

[1] http://article.gmane.org/gmane.comp.version-control.git/150007

-- Ram

Re: [GSoC update] git-remote-svn: The final one

[Stephen Bash]

> > So being selfish, what I care about is an improved method for
> > converting SVN repos to git repos once and never going back. It
> > sounds to me like the major hurdles remaining there are getting
> > svn-fe to handle dumpfile v3 and handling branches/tags. Is that
> > correct?
> 
> We're doing better than that- when the whole thing is complete, we
> should have perfectly seamless two-way compatibility.

That makes complete sense.  It just happens I'm interested in a subset of the problem (the order has come down from on high that we will be using git for future development), so I'm curious if/how a one-time conversion simplifies things.

> But yes, you're
> right- there are several non-trivial tasks to handle, the top three of
> which are:
> 1. Making svn-fe git-aware (and hence handle dumpfile v3)
> 2. Writing a fast-export to dumpfilev3 converter
> 3. Implementing the branch/ tag mapper in an extensible way

Thanks for clarifying.

> Don't worry about it though- we're all working hard on it :)

I believe that :)  And you're not holding me up; I've spent my spare time this summer coming up with some Perl scripts to work with svn-fe for the one-time conversion, so I'll have my point case solved in the near future.  But I'm sure the problem will come up again, and I'll be happy to use the general tool then.

Thanks,
Stephen

Re: [GSoC update] git-remote-svn: The final one

[Ramkumar Ramachandra]

Hi Stephen,

Stephen Bash writes:
> > > So being selfish, what I care about is an improved method for
> > > converting SVN repos to git repos once and never going back. It
> > > sounds to me like the major hurdles remaining there are getting
> > > svn-fe to handle dumpfile v3 and handling branches/tags. Is that
> > > correct?
> > 
> > We're doing better than that- when the whole thing is complete, we
> > should have perfectly seamless two-way compatibility.
> 
> That makes complete sense.  It just happens I'm interested in a subset of the problem (the order has come down from on high that we will be using git for future development), so I'm curious if/how a one-time conversion simplifies things.

Thanks for bringing this up- it's quite interesting :)

A one-time conversion thing would have changed our priorities. Mainly,
I wouldn't have been able to justify the time I invested in building
svnrdump. I'd have probably settled for complete mirror + dump using a
combination of existing tools. The second difference:
svn-dump-fast-export is quite complex; although it was probably an
unconcious decision on David's part, it makes a really good generic
parser, and I suspect that it'll help us generate the dumpfile we need
for loading revision history data back into Subversion. Third, it
simplifies the branch/ tag mapper greatly- we'd only need to achieve a
perfect one-way mapping. A symmetric mapping is harder for several
reasons- consider preserving props for instance.

-- Ram