Fixing branches from a (really messed up) subversion repository

Fixing branches from a (really messed up) subversion repository

[Joshua Shrader]

I've finally convinced my project to switch to Git...  However, when
my project first began, the people responsible for branching/tagging
were fairly unfamiliar with standard procedures.  As such, the
beginnings of our repository look a little something like this:

.
`----project
     |
     `branch1
     |          |
     |          `----project
     |                        `src
     `branch2
     |          |
     |          `----project
     |                        `src
     `src

That is, the branches were essentially inside the trunk.  In order to
check out the "mainline" of development, you also got all of the
branches.  After a few branches, they realized that this was a
problem, and someone found out about the trunk/branches/tags paradigm.
 Our branches were svn mv'd to separate directories under "branches",
and all was well.  Now everything looked like a sane version control
repository.

Now, we want to import this into Git.  I tried git svn import
--stdlayout..., but this just stalled.  I'm assuming because the first
version didn't have a trunk/branches/tags structure.  So, I had to
import it leaving off the --stdlayout option.  Is there any way to
manually fix this in Git?  After the import is complete, my master
branch will have a trunk, branches, and tags directory in it.  Can I
create real branches out of the directories under the branches
directory and then remove them from master?

Any help is greatly appreciated.  I've been hoping for a while that
we'd switch to Git, and I don't want this to hang us up.

Thanks,

Josh

Re: Fixing branches from a (really messed up) subversion repository

[Ævar Arnfjörð Bjarmason]

On Tue, Jun 15, 2010 at 16:26, Joshua Shrader <jshrader83@gmail.com> wrote:
> I've finally convinced my project to switch to Git...  However, when
> my project first began, the people responsible for branching/tagging
> were fairly unfamiliar with standard procedures.  As such, the
> beginnings of our repository look a little something like this:
>
> .
> `----project
>     |
>     `branch1
>     |          |
>     |          `----project
>     |                        `src
>     `branch2
>     |          |
>     |          `----project
>     |                        `src
>     `src
>
> That is, the branches were essentially inside the trunk.  In order to
> check out the "mainline" of development, you also got all of the
> branches.  After a few branches, they realized that this was a
> problem, and someone found out about the trunk/branches/tags paradigm.
>  Our branches were svn mv'd to separate directories under "branches",
> and all was well.  Now everything looked like a sane version control
> repository.
>
> Now, we want to import this into Git.  I tried git svn import
> --stdlayout..., but this just stalled.  I'm assuming because the first
> version didn't have a trunk/branches/tags structure.  So, I had to
> import it leaving off the --stdlayout option.  Is there any way to
> manually fix this in Git?  After the import is complete, my master
> branch will have a trunk, branches, and tags directory in it.  Can I
> create real branches out of the directories under the branches
> directory and then remove them from master?
>
> Any help is greatly appreciated.  I've been hoping for a while that
> we'd switch to Git, and I don't want this to hang us up.

I have the same problem on another project, and after looking at the
components involved it seems that the easiest way is to filter the SVN
dump, then import it into Git.

It's quite easy to write a filter with SVN::Dump that does this,
i.e. just do a few string replacements on the paths before a given
revision, and drop the revision where you moved things.

A lot easier than convincing git-svn to do what you want, anyway.

Re: Fixing branches from a (really messed up) subversion repository

[Joshua Shrader]

Not quite sure if I understand correctly.  Can Git import directly
from an SVN dump, or are you suggesting to essentially re-build the
svn repository correctly, and then import?

On Tue, Jun 15, 2010 at 12:29 PM, Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> On Tue, Jun 15, 2010 at 16:26, Joshua Shrader <jshrader83@gmail.com> wrote:
>> I've finally convinced my project to switch to Git...  However, when
>> my project first began, the people responsible for branching/tagging
>> were fairly unfamiliar with standard procedures.  As such, the
>> beginnings of our repository look a little something like this:
>>
>> .
>> `----project
>>     |
>>     `branch1
>>     |          |
>>     |          `----project
>>     |                        `src
>>     `branch2
>>     |          |
>>     |          `----project
>>     |                        `src
>>     `src
>>
>> That is, the branches were essentially inside the trunk.  In order to
>> check out the "mainline" of development, you also got all of the
>> branches.  After a few branches, they realized that this was a
>> problem, and someone found out about the trunk/branches/tags paradigm.
>>  Our branches were svn mv'd to separate directories under "branches",
>> and all was well.  Now everything looked like a sane version control
>> repository.
>>
>> Now, we want to import this into Git.  I tried git svn import
>> --stdlayout..., but this just stalled.  I'm assuming because the first
>> version didn't have a trunk/branches/tags structure.  So, I had to
>> import it leaving off the --stdlayout option.  Is there any way to
>> manually fix this in Git?  After the import is complete, my master
>> branch will have a trunk, branches, and tags directory in it.  Can I
>> create real branches out of the directories under the branches
>> directory and then remove them from master?
>>
>> Any help is greatly appreciated.  I've been hoping for a while that
>> we'd switch to Git, and I don't want this to hang us up.
>
> I have the same problem on another project, and after looking at the
> components involved it seems that the easiest way is to filter the SVN
> dump, then import it into Git.
>
> It's quite easy to write a filter with SVN::Dump that does this,
> i.e. just do a few string replacements on the paths before a given
> revision, and drop the revision where you moved things.
>
> A lot easier than convincing git-svn to do what you want, anyway.
>

Re: Fixing branches from a (really messed up) subversion repository

[Ævar Arnfjörð Bjarmason]

On Tue, Jun 15, 2010 at 16:40, Joshua Shrader <jshrader83@gmail.com> wrote:
> Not quite sure if I understand correctly.  Can Git import directly
> from an SVN dump, or are you suggesting to essentially re-build the
> svn repository correctly, and then import?

I mean:

  1. svnadmin dump /path/to/repo > svn.dump
  2. *filter svn.dump*
  3. svnadmin load ...
  4. git svn /path/to/loaded/dump

Re: Fixing branches from a (really messed up) subversion repository

[Ævar Arnfjörð Bjarmason]

On Fri, Jun 18, 2010 at 20:59, Joshua Shrader <jshrader83@gmail.com> wrote:
> Thanks.  It took a couple days to get everything straightened out, but
> this seems to have worked very well.

Mind sharing the details or code for what you did?

I've only proved to myself that this approach will work, I haven't
actually done it.

Re: Fixing branches from a (really messed up) subversion repository

[Joshua Shrader]

I started with svnadmin dump /path/to/repo > svn.dump.  Then I had to
manually edit svn.dump (I'm sure someone smarter then me could roll
this into the Perl script I used in the next step) to create a "trunk"
and "branches" directory in the first commit.  I found the commits
that added "trunk" and "branches" when we re-organized our repo to
follow this convention, and deleted them.  Then I had to locate where
all of the branches were created - looked like

Node-path: branchname
Node-kind: dir
Node-action: add
Node-copyfrom-rev: 23
Node-copyfrom-path:

and change that blank "Node-copyfrom-path: " to "Node-copyfrom-path:
trunk".  The blank path is what really caused issues because everytime
a branch was created, it would contain copies of branches which
contained copies of branches, etc...  So branch2 had a subfolder
branch1, and branch3 contained branch2 and branch 1, where branch 2
contained branch 1...

Then, I removed all of the contents of the commits (not the commits
themselves) where we re-organized the repo.  i.e.

Node-path: trunk/src
Node-action: add
Node-copyfrom-ref: 23
Node-copyfrom-path: src

and the ones where we deleted nested branches and moved the individual
branches into a "branches" directory.

I made these edits and saved them.  Then I ran that file through a
perl filter, tried to svnadmin load newrepo < filteredDumpFile, and if
I got errors, I iterated - changed the pre-filtered file, blow away
the svn repo that gave me an error during load, re-create it, and
re-load.  I iterated this over several days, as I discovered more and
more inconsistencies with my edited dump file.  I think I have all of
the steps above, but I may be missing one or two, and if I am, I
apologize.  I'd imagine everyone's situation would be a bit different,
so adjust accordingly.

Now, I'm quite the Perl hack so I doubt you want to look at the
script.  It essentially did the following:

Read each line of the dump file.
If the line doesn't start with "Node-path:" or "Node-copyfrom-path:",
then just print it un-altered.
If it does,
  does what follows the ':' start with "trunk", "branches", or "tags"?
 If so, then just print un-altered.
Otherwise,
  for each $branchname(@branchnames) if what follows the ':' begins
with $branchname, replace it with "branches/$branchname", and print.
If none of the branch names match,
  prepend "trunk" to the beginning of the path, and print.

This essentially re-writes all of the paths so that everything not in
a branch, or already under "branches", "tags", or "trunk", is put
under trunk.  It also moves all of the /branchenames to
/branches/branchnames.

After this, we did
svnadmin hotcopy repo repo-backup
rm -rf repo
svnadmin create repo
svnadmin load repo <filteredDumpFile

Changed the ownership to match the ownership of the original
repository, copied over the contents of the repo-backup/hooks
directory into the new repo/hooks, and we were all set.

git svn clone --stdlayout... then worked fine.  Now I just need to
work on a workflow for my co-workers how want to also use git (i.e.
create a bare git repo, and keep that in sync with the git-svn repo).
Ideally, all of us git users can clone the bare repo and push to it,
then every once in a while, I can pull from the bare into the git-svn
repo, and do a dcommit, but I've ran into a couple yet-unresolved
problems.  If you have an references or ideas, I'd appreciate them,
but I suppose that's another thread.



On Fri, Jun 18, 2010 at 5:01 PM, Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> On Fri, Jun 18, 2010 at 20:59, Joshua Shrader <jshrader83@gmail.com> wrote:
>> Thanks.  It took a couple days to get everything straightened out, but
>> this seems to have worked very well.
>
> Mind sharing the details or code for what you did?
>
> I've only proved to myself that this approach will work, I haven't
> actually done it.
>