Stitching together private svn repo and public git repo

Stitching together private svn repo and public git repo

[Gregory Jefferis]

Short Version: 

Repo B is a completely linear repo that has been tracking public repo A for
some time.  A number of manual merges were done to bring new A releases
into B but there are no connections between the two repos.  (How) can I
stitch B to A to reflect this relationship?  I want to retain B's history
and leave myself with an arrangement in which I can pull from A into B
periodically?  Many thanks for your suggestions, Greg.

--
Long Version:

I have been tracking my modifications of a public project released as tar
balls with my own local svn repository.  My svn repo was completely linear
and I always merged in changes manually each time a new tar ball was
released.  This started to get painful.

Now I want to do better.  I have set up a public git repository ("A") to
track the tar ball releases from the public project.  I have converted my
svn repo with git-svn to a git repo ("B").

My goal is to end up with a (new?) git repo tracking the public repo (A) so
that I can pull in any changes and merge easily. Fine, but I also want to
keep the history that I have imported from my svn repository (B).  So my
question is, can anyone suggest some pointers for how to stitch together A
and B?

Right now I have been trying to pull B into A to splice:

A $ git checkout v1.91
B $ git checkout v1.91-manualmerge
B $ git pull --no-commit -s ours ../A

This looks right when I run gitk, but if I repeat this for a second merge
point then the previous merge seems to disappear from history when I bring
up gitk.

I haven't found any docs/wiki info that has enlightened me yet.  Any
suggestions, wisdom, links etc very much appreciated - I do want to try to
get this right now, so I don't have to rejig everything in the future.  Best
wishes and many thanks,

Greg.

-- 
Gregory Jefferis, PhD                               and:
Research Fellow
Department of Zoology                               St John's College
Downing Street                                      Cambridge
Cambridge, CB2 3EJ                                  CB2 1TP

Re: Stitching together private svn repo and public git repo

[Charles Bailey]

On Wed, Jan 02, 2008 at 07:25:41PM +0000, Gregory Jefferis wrote:
> 
> Right now I have been trying to pull B into A to splice:
> 
> A $ git checkout v1.91
> B $ git checkout v1.91-manualmerge
> B $ git pull --no-commit -s ours ../A

So you have something like this in B:

Av1* -- Av1_b -- Av1_c -- Av1_d -- Av2_d* -- Av2_e -- Av3_e*

Where the * are manual merges of the offical Avn releases...

And you want this:

Av1   --       --       -- Av2   --      -- Av3
  \                          \                \
  Av1_b -- Av1_c -- Av1_d -- Av2_d -- Av2_e -- Av3_e  ???

This is a "rewrite history" job as parent lists are part of each
commit.  (See Linus' big bold-face warning about history re-writes.)

First of all I would add a remote for the 'A' repository so that those
commits are available in the 'B' repository.

Something like:
git remote add repoA /path/to/A
git fetch repoA

You could then do this with a 'git filter-branch --parent-filter' to
rewrite the parents of the merge commits.  As far as I can see, you
would need to call filter-branch once per merge to rewrite everything
from the merge commit forwards.  At this point all later commits would
have different ids, so attempting to rewrite subsequent parent-ids in
the same filter-branch invocation is probably futile.

It might be possible to use an intelligent script in a single
--commit-filter invocation of filter-branch, but the script of the
actual filter would have to be a bit (a lot!) more sophisticated,
remember the ids of the new commits as it created them with 'git
commit-tree' and merging in the repoA parents at the right points.

Leaving that aside and concentrating on the multiple filter-branch
invocation option... for example, the first parent-filter script could
be:
sed -e 's/^$/-p <commit id of Av1>/'

( This is if you want your exisiting tree to branch from Av1 in the
original repo.  If you wanted to replace your tree root with a root at
Av1, because your current root is just a copy of Av1 then you want:
sed -e 's/<commit id of repoB's Av1 copy>/<commit id of Av1 in A>/'
)

Then, for whatever the Av2_d commit's new id is, you could do a
parent-filter of:
sed -e 's/\(<new commit id of Av1_d>\)/\1 -p <commit id of Av2>/'

This causes the Av2_d commit to be rewritten, maintaining its original
parent and adding another parent which comes from repoA.  It then
rewrites all the descendent commits of Av2_d to take account of this
new commit id and all the subsequent new commit ids.

Now you can do similar for the next merge commit:
sed -e 's/\(<new commit id of Av2_e>\)/\1 -p <commit id of Av3>/'

As noted, you end up with completely new commit objects with a
completely new history, so you will screw everyone who is
pulling/cloning from you.

Charles.

Re: Stitching together private svn repo and public git repo

[Dmitry Potapov]

On Jan 3, 2008 12:40 AM, Charles Bailey <charles@hashpling.org> wrote:
> You could then do this with a 'git filter-branch --parent-filter' to
> rewrite the parents of the merge commits.  As far as I can see, you
> would need to call filter-branch once per merge to rewrite everything
> from the merge commit forwards.

I believe there is a much easier way to do that using .git/info/grafts
The first step is to create .git/info/grafts, which specifies commit-id
and its parents for each commit that you want to change. Then you
can check the result using gitk, and if you are satisfied with what
you see then you run git filter-branch on it to convert 'fake' parents
into real ones.

Dmitry

Re: Stitching together private svn repo and public git repo

[Charles Bailey]

On Thu, Jan 03, 2008 at 01:13:54AM +0300, Dmitry Potapov wrote:
> I believe there is a much easier way to do that using .git/info/grafts
> The first step is to create .git/info/grafts, which specifies commit-id
> and its parents for each commit that you want to change. Then you
> can check the result using gitk, and if you are satisfied with what
> you see then you run git filter-branch on it to convert 'fake' parents
> into real ones.
> 
> Dmitry
> 

Oh yes, this is much easier.  Unless I'm missing something, the
documentation on grafts is fairly sparse, though.  They are mentioned
(almost in passing) in git help filter-branch but the file format is
only documented in repository-layout.txt which seems more developer
than user oriented.

Charles.

Re: Stitching together private svn repo and public git repo

[Gregory Jefferis]

Many thanks to both for your replies.  Starting from your comments and
consulting:

http://www.kernel.org/pub/software/scm/git/docs/repository-layout.html

for (fairly scant) details of .git/info/grafts and
 
http://www.kernel.org/pub/software/scm/git/docs/git-filter-branch.html

I think I have now been able to do the necessary.  Since I found this a
trivial operation (in the true mathematical sense of quasi-impossible until
someone helps you to figure it out and simple once you have) and I tripped
over a couple of times, I thought I would share my final shell script below.

One point that I missed initially.  The graft file format is unforgiving.
You are allowed comments but each regular line is:

Commit-SHA-1 Parent1-SHA-1 Parent2-SHA-1\n
      
If you put e.g. more than a single space between your SHA-1s you will get
"bad graft data:" - I missed one in the fake merge line and ran round in
circles for a bit thinking it was something to do with the state of my
trees.

Anyway, boy did it feel great when I was able to git-pull the latest public
release into my private repository:

    34 files changed, 11563 insertions, 8399 deletions

and only 2 simple conflicts to fix with git-mergetool.  That was not a merge
I wanted to do by hand!

Best wishes and thanks again,

Greg.
-- 

#!/usr/bin/env bash
# makemygitbygraft.sh
# merge my private repo (converted from svn) with a public git repo
# retains history of both and will allow 3 way merges etc in future

# work on a copy of my svn imported repo just in case
cd ~/dev/Physiology
rm -rf ~/dev/Physiology/nm-graft
cp -R nm-git-svn2 nm-graft
cd nm-graft

# Now get to work, bring in public repo as a remote
git remote add nm-ucl ../nm-ucl
git fetch nm-ucl

# set up the grafts
cat > .git/info/grafts <<EOF
# connect r2 of git-svn to v1.86 of nm-ucl
# r1 (which was an identical initial import) will now be detached (and
forgotten)
52b1933421ef524811407fa4c240da58ceec5749
eca3db14fcf25744fdf585456f03599a9db2af96
#
# Make fake merge with Neuromatic v1.91b
# svn-manualmerge1 svn-manualmerge1~1 public-v1.91b
120282ee5275027312dde386c8995218f361cf35
0962cb27f7ad92f44def04630e8e6a22b86e0699
6735057f53dd57248c7ec23f6ae9f22085d98fba
EOF

# running git-status seems to stop git thinking that the working dir is
dirty
git-status
# run git-filter-branch to instantiate the new parents
# just use cat so that parent names are untouched by the process
# but will be fed by .git/grafts/info as appropriate
git-filter-branch --parent-filter cat HEAD

Nicely formatted snippet also at
http://pastie.textmate.org/private/pv1n1nbmcmtxnxbq4zd7w

On 2/1/08 22:46, "Charles Bailey" <charles@hashpling.org> wrote:

> On Thu, Jan 03, 2008 at 01:13:54AM +0300, Dmitry Potapov wrote:
>> I believe there is a much easier way to do that using .git/info/grafts
>> The first step is to create .git/info/grafts, which specifies commit-id
>> and its parents for each commit that you want to change. Then you
>> can check the result using gitk, and if you are satisfied with what
>> you see then you run git filter-branch on it to convert 'fake' parents
>> into real ones.
>> 
>> Dmitry
>> 
> 
> Oh yes, this is much easier.  Unless I'm missing something, the
> documentation on grafts is fairly sparse, though.  They are mentioned
> (almost in passing) in git help filter-branch but the file format is
> only documented in repository-layout.txt which seems more developer
> than user oriented.
> 
> Charles.

-- 
Gregory Jefferis, PhD
Division of Neurobiology
MRC Laboratory of Molecular Biology,
Hills Road,
Cambridge, CB2 0QH, UK.

http://www2.mrc-lmb.cam.ac.uk/NB/jefferis_g
http://www.neuroscience.cam.ac.uk/directory/profile.php?gsxej2
http://flybrain.stanford.edu

Re: Stitching together private svn repo and public git repo

[Junio C Hamano]

I suspect your message was linewrapped by your MUA and ...

Gregory Jefferis <jefferis@gmail.com> writes:

> ...
> # set up the grafts
> cat > .git/info/grafts <<EOF
> # connect r2 of git-svn to v1.86 of nm-ucl
> # r1 (which was an identical initial import) will now be detached (and
> forgotten)
> 52b1933421ef524811407fa4c240da58ceec5749
> eca3db14fcf25744fdf585456f03599a9db2af96
> #
> # Make fake merge with Neuromatic v1.91b
> # svn-manualmerge1 svn-manualmerge1~1 public-v1.91b
> 120282ee5275027312dde386c8995218f361cf35
> 0962cb27f7ad92f44def04630e8e6a22b86e0699
> 6735057f53dd57248c7ec23f6ae9f22085d98fba
> EOF
>
> # running git-status seems to stop git thinking that the working dir is
> dirty
> git-status
> ...

... your grafts file look as if you are marking five commits as
root commits, which is not what you wanted ;-).

The reason I mention this is because I think with a bit more
polishing and some background description your message is a good
material to add to Documentation/howto/.