How to combine git repos with similar code and keep all branches and tags?

How to combine git repos with similar code and keep all branches and tags?

[Jose de Leon]

Hi All,


I've got an interesting problem and the possible solutions I've found from searching google don't seem to work for us.  In a nutshell, I need to combine multiple git repositories into single repository and preserve all history, branches and tags from each repository.    

Does anybody have a solution for this, how do this kind of thing, is it even possible?

For some unknown reason to me, our developers started a git project, called Ver1, this was the first version.  Then sometime later, they created a new git repository called Ver2, the initial commit for Ver2 was essentially a copy of the code in Ver1 from the master.  They didn't clone it, they just copied the code at the latest point.  This wasn't done by forking.  Then they repeated this for Ver3 and Ver4, etc.  On top of all that, they have been adding new code to Ver1, Ver2, etc. that has caused each to divert from the other and each have their own branch and tag sets.  They have similar directory structure and similar file names, but there are some slight differences in structure.

Well, this has become unmanageable and now we want to combine them into one single git repository.   

Logically, these are the same project but at different versions, basically, Ver1 and Ver2, etc, should be simply branches at different times if these were combined into a single repository.

Re: How to combine git repos with similar code and keep all branches and tags?

[Matthieu Moy]

Jose de Leon <jdeleon@ensim.com> writes:

> For some unknown reason to me, our developers started a git project,
> called Ver1, this was the first version. Then sometime later, they
> created a new git repository called Ver2, the initial commit for Ver2
> was essentially a copy of the code in Ver1 from the master. They
> didn't clone it, they just copied the code at the latest point.

This is why "graft points" were created, and then superseeded by "git
replace".

See http://git-scm.com/blog/2010/03/17/replace.html and
http://git-scm.com/docs/git-replace

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: How to combine git repos with similar code and keep all branches and tags?

[Max Horn]

Hi Jose,

On 14.04.2015, at 18:44, Jose de Leon <jdeleon@ensim.com> wrote:

> Hi All,
> 
> 
> I've got an interesting problem and the possible solutions I've found from searching google don't seem to work for us.  In a nutshell, I need to combine multiple git repositories into single repository and preserve all history, branches and tags from each repository.    
> 
> Does anybody have a solution for this, how do this kind of thing, is it even possible?
> 
> For some unknown reason to me, our developers started a git project, called Ver1, this was the first version.  Then sometime later, they created a new git repository called Ver2, the initial commit for Ver2 was essentially a copy of the code in Ver1 from the master.  They didn't clone it, they just copied the code at the latest point.  This wasn't done by forking.  Then they repeated this for Ver3 and Ver4, etc.  On top of all that, they have been adding new code to Ver1, Ver2, etc. that has caused each to divert from the other and each have their own branch and tag sets.  They have similar directory structure and similar file names, but there are some slight differences in structure.
> 
> Well, this has become unmanageable and now we want to combine them into one single git repository.   
> 
> Logically, these are the same project but at different versions, basically, Ver1 and Ver2, etc, should be simply branches at different times if these were combined into a single repository.

Here is one possible way to go about this using grafts (I used something
similar in the past);

1) Get all the data into a single git repository.

  Since everything is already in a git repositories, you could e.g. create a
  clone of Ver1; then add remotes for Ver2 ... VerN, and fetch them all,
  along with tags. If there are conflicts between branch or tag names, deal
  with them at this point.

2) Create graft points to tie the history together.

   Identify the commit in Ver1 at which Ver2 branched off. Then, graft that
   as parent for the initial commits of Ver2. See here for some basic
   instructions <https://git.wiki.kernel.org/index.php/GraftPoint> or feel
   free to ask for details (or Google, or... :)

   Repeat vor Ver2+Ver3, Ver3+Ver4, etc.

3) Finally, you can get rid of the graft points, and turn everything into a
"proper" history, by running "git filter-branch". Something like

  git filter-branch -- --all

ought to do it, but I might be forgetting something (I am sure somebody will
correct me soon in that case, though ;-). Best to have a look at
<http://git-scm.com/docs/git-filter-branch> for yourself, though.


This all is under the assumption that you want to stay as close to how
things really were (usually a good idea). But sometimes it may be desirable
to make further adjustments. E.g. you may wish to adjust committer names,
rearrange some stuff (though usually git is quite good at doing the right
thing automatically, etc. How to do that of course depends on what exactly
you want to do, but in many cases, filter-branch is your friend.

Hope that helps!

Max

Re: How to combine git repos with similar code and keep all branches and tags?

[Junio C Hamano]

Matthieu Moy <Matthieu.Moy@grenoble-inp.fr> writes:

> Jose de Leon <jdeleon@ensim.com> writes:
>
>> For some unknown reason to me, our developers started a git project,
>> called Ver1, this was the first version. Then sometime later, they
>> created a new git repository called Ver2, the initial commit for Ver2
>> was essentially a copy of the code in Ver1 from the master. They
>> didn't clone it, they just copied the code at the latest point.
>
> This is why "graft points" were created, and then superseeded by "git
> replace".
>
> See http://git-scm.com/blog/2010/03/17/replace.html and
> http://git-scm.com/docs/git-replace

After setting up either grafts or replaces, I'd strongly recommend
running filter-branch or bfg to rewrite the history of the combined
result, and have the developers use that rewritten history _after_
removing the grafts (or replaces).

And if you are going to go that route, then graft is sufficient and
a lot more light-weight.  The only advantage replace has over grafts
is that they can be transferred using "git push" and "git fetch"
(while grafts can be transferred with some file transfer mechanism
outside Git) but if you are rewriting the history, that advantage is
lost.

RE: How to combine git repos with similar code and keep all branches and tags?

[Jose de Leon]

Thank you!

-----Original Message-----
From: Max Horn [mailto:max@quendi.de] 
Sent: Tuesday, April 14, 2015 10:15 AM
To: Jose de Leon
Cc: git@vger.kernel.org
Subject: Re: How to combine git repos with similar code and keep all branches and tags?

Hi Jose,

On 14.04.2015, at 18:44, Jose de Leon <jdeleon@ensim.com> wrote:

> Hi All,
> 
> 
> I've got an interesting problem and the possible solutions I've found from searching google don't seem to work for us.  In a nutshell, I need to combine multiple git repositories into single repository and preserve all history, branches and tags from each repository.    
> 
> Does anybody have a solution for this, how do this kind of thing, is it even possible?
> 
> For some unknown reason to me, our developers started a git project, called Ver1, this was the first version.  Then sometime later, they created a new git repository called Ver2, the initial commit for Ver2 was essentially a copy of the code in Ver1 from the master.  They didn't clone it, they just copied the code at the latest point.  This wasn't done by forking.  Then they repeated this for Ver3 and Ver4, etc.  On top of all that, they have been adding new code to Ver1, Ver2, etc. that has caused each to divert from the other and each have their own branch and tag sets.  They have similar directory structure and similar file names, but there are some slight differences in structure.
> 
> Well, this has become unmanageable and now we want to combine them into one single git repository.   
> 
> Logically, these are the same project but at different versions, basically, Ver1 and Ver2, etc, should be simply branches at different times if these were combined into a single repository.

Here is one possible way to go about this using grafts (I used something similar in the past);

1) Get all the data into a single git repository.

  Since everything is already in a git repositories, you could e.g. create a
  clone of Ver1; then add remotes for Ver2 ... VerN, and fetch them all,
  along with tags. If there are conflicts between branch or tag names, deal
  with them at this point.

2) Create graft points to tie the history together.

   Identify the commit in Ver1 at which Ver2 branched off. Then, graft that
   as parent for the initial commits of Ver2. See here for some basic
   instructions <http://cp.mcafee.com/d/1jWVIq6hEi6jqb2pEVKOyUqenztPqqdNPaabzz1Jd6UVB54sepdFET7cEEKefCQQrzCkknxPVIhgY_WM1nMF9nBPvMF9nBPqa91x1dZ_HYCCMOeWZOWr8W_c3AnXYJteOaqJT6ul3PWApmU6CQjqpK_nd7bwUsMyqemnPtPpesRG9px6k-ciaOUHYouvM070bU8U4EWXcblB2pJngY52PvMgBO7CS1Ob1I5-Aq80LkMq89Rd42V2kfd416kPh17RGQd41ykOpEwF9mh-Nd45GEuq8dwwq82L-Iq8dffd412eMIj-xEwS21EwDF6x8SeudOB_GPPAJ5zzZ> or feel
   free to ask for details (or Google, or... :)

   Repeat vor Ver2+Ver3, Ver3+Ver4, etc.

3) Finally, you can get rid of the graft points, and turn everything into a "proper" history, by running "git filter-branch". Something like

  git filter-branch -- --all

ought to do it, but I might be forgetting something (I am sure somebody will correct me soon in that case, though ;-). Best to have a look at <http://cp.mcafee.com/d/2DRPowcCQm4PhPtB5MQsL6XCQQrzCkkn763qqdNPaa8UsOrjhKephhssvdFET7cEEL3DPoyxV_Rw2LxiiLbC_xiiLbCQki322rX_nVddxAtRXBQShR-o78LTVqWtAkRrKcYG7DR8OJMddFCQPt-Kqen1MVx4QsILCXCM0p6kOZbaYLy5rkMzaptShY_yn8hS5ypJngY52PvMgBO7CS1Ob1I5-Aq80LkMq89Rd42V2kfd416kPh17RGQd41ykOpEwF9mh-Nd45GEuq8dwwq82L-Iq8dffd412eMIj-xEwS21EwDF6x8SeudXyyH> for yourself, though.


This all is under the assumption that you want to stay as close to how things really were (usually a good idea). But sometimes it may be desirable to make further adjustments. E.g. you may wish to adjust committer names, rearrange some stuff (though usually git is quite good at doing the right thing automatically, etc. How to do that of course depends on what exactly you want to do, but in many cases, filter-branch is your friend.

Hope that helps!

Max