Re: Trying to sync two svn repositories with git-svn (repost)

[Josef Wolf <jw@raven.inka.de>]

Thanks for your answer, Avery!

On Tue, Apr 28, 2009 at 11:19:51PM -0400, Avery Pennarun wrote:
> On Tue, Apr 28, 2009 at 6:37 PM, Josef Wolf <jw@raven.inka.de> wrote:
> > Currently, there exist multiple independent repositories (for security
> > reasons).  In the past, the repositories were "synchronized" manually.
> > So technically, the repositories have no common history (at least not
> > in svn's metadata).  But the contents are actually rather "similar",
> > since they were synchronized multiple times in the past.
> >
> > In the long term, I'd like to move everything completely to git.  That
> > would make it much easier to move changes from one repos to the other
> > while keeping the (intended) differences in the policy.
> >
> > So my first goal is to bring the contents into sync.  The next step would
> > be to create a "reference" (the official) git repository, which can be
> > cloned by the administrations to create their localized repositories.
> >
> > In the meantime, I need a way to synchronize the contents from time to
> > time.  I guess it will take some time to create the official repos and
> > get used to the work flow.
> 
> Okay, I think I'm following you.  And I think the difficulty of your
> solution will depend on how important it is to cherry-pick each
> individual commit from each repo vs. just merging everything as a
> batch.

I've already done the cherry-picking.  Basically, I've done this:

  # first, move patches from second-svn to first-svn
  git checkout first-svn
  git svn rebase
  git cherry-pick sha1 # repeat as needed
  git merge -s ours second-svn

  # Now, the other way around
  git checkout second-svn
  git svn rebase
  git cherry-pick sha1 # repeat as needed
  git merge -s ours first-svn

The first git-svn-rebase after the merge causes all the (already picked)
commits from the other branch to be pulled into the current branch.
Adding the --no-ff option does not help.  Omitting the cherry-picking
does not help, either.

To be honest, I do not understand this behavior at all.  I thought
"-s ours" should mark the other branch as "already merged".  IMHO, this
should prevent future merges from pulling those commits again.  But
this seems not to be true: git-svn-rebase tries to apply them _again_,
causing almost everything to conflict.

> At Versabanq, we're using git for a bunch of stuff including our
> autobuilder (http://github.com/apenwarr/gitbuilder) and my own

Interesting project.  One question: the README mentions that the
gitbuilder as capable to update itself.  But I can not actually see
this functionality in the scripts.  Is that just a typo or am I missing
something?

> Yes, it is possible to rebase everything from git onto an svn branch,
> and then git svn dcommit it.

AFAICS, this is the preferred work flow of git-svn.

> However, in my experience, this is
> fairly nasty (and it also tries to linearize non-linear history, which
> is just messy).  What we've been doing lately is just merging all
> changes from git into the svn branch as a single commit:
> 
>    git checkout git-svn
>    git merge --no-ff mybranch   # --no-ff prevents git-svn from
> crazily linearizing things
>    git svn dcommit
> 
>    git checkout mybranch
>    git merge git-svn

How would this be changed if the commits are coming from the other svn
repo instead of directly from git?

So should I replace all my "git cherry-pick sha1" by corresponding
"git merge -s ??? --no-ff sha1" commands?

> As long as you "git config merge.summary true" (to make the merge
> commit list all the commits it's merging)

How does this option influence the merge operation?  Or is this meant
to provide additional information to the person who does the next merge?

> and you merge frequently
> enough, this is reasonably painless.  You end up with a lot of merge
> commits, but the git history is recording everything fully, so if you
> want to throw away svn someday, you can just go ahead.

Sounds good, but I still don't get it :-)  Can you provide a more
verbose example of the workflow?

> Now, your problem is a little more complex, because it sounds like
> people are checking in two types of things on both sides: private
> things and public things.  So if you want *only* the private things,

I want both.  The difference is that I (usually) want to pull the public
things unmodified, while I want to generalize/localize the private things.
So when merging the private part, I would not want to pick the specific
entries.  But I still want to pick the _structure_ (possibly removing or
modifying the localized entries).

> you're going to have to cherry-pick, and cherry-picking is going to
> confuse your merging.
> 
> If you could convince the people using svn to use two branches: one
> for private stuff and one for public stuff, then your life would be
> easier.  You could just merge the public stuff in git, and ignore the
> private stuff.
> 
> If that's not an option, you *can* combine cherry-pick with -s ours as
> you suggest, though it's kind of nasty.  The trick is to merge -s ours
> in *both* directions at the right time, so you can avoid conflicts.
> 
>     git checkout git-svn
>     git merge mybranch
> 
>     git checkout mybranch
>     [git cherry-pick or git merge *everything* you're missing from git-svn...]
>     git merge -s ours --no-ff git-svn
>       # future merges from git-svn will ignore everything in mybranch up to now
> 
>     git checkout git-svn
>       # we know git-svn is already up to date, because of the first merge above
>     git merge -s ours --no-ff mybranch
>       # future merges from mybranch will ignore everything in git-svn up to now
>     git svn dcommit

Does that mean I should do a normal merge _before_ I go cherry-pick?

> After these steps (WARNING: I didn't actually run them, so I might
> have made a mistake), you should have both branches in sync, and you
> *should* be able to merge in both directions whenever you want (make
> sure you use --no-ff), until the next time someone commits something
> private and screws you over again.
> 
> If you have more than one svn server, the above method should be
> extensible; just use another svn branch in place of 'mybranch' or keep
> cross-merging across all the branches.

Maybe I got confused about what I should do on which branch.  Currently,
I have five branches:

  first-svn/trunk    # svn-remote branch of my repos
  second-svn/trunk   # svn-remote branch of their repos
  first-svn          # created by "git checkout -b first-svn/trunk"
  second-svn         # created by "git checkout -b second-svn/trunk"
  master             # I don't use this one yet

For this, I have the following configuration:

  [svn-remote "first-svn"]
          url = file:///var/tmp/builds/git-sync/svn/first-svn
          fetch = trunk:refs/remotes/first-svn/trunk
          branches = branches/*:refs/remotes/first-svn/*
          tags = tags/*:refs/remotes/first-svn/tags/*
  [svn-remote "second-svn"]
          url = file:///var/tmp/builds/git-sync/svn/second-svn
          fetch = trunk:refs/remotes/second-svn/trunk
          branches = branches/*:refs/remotes/second-svn/*
          tags = tags/*:refs/remotes/second-svn/tags/*