move files between disparate repos and maintain version history

move files between disparate repos and maintain version history

[davetron5000]

Using git-svn at work, and had to do some svn moves that I'm wondering
would be handled with git?

Our repo is setup like this:

$SVNROOT/main/core/{trunk,tags,branches}
$SVNROOT/main/app/{trunk,tags,branches}

core and app are treated as different project roots internally, and my
git-svn treats them as such.

We recently needed to move some files from app to core.  I could've
just moved them in between my git repos, but the version history would
be lost, since these are two separate repos.  I did the move via svn
mv, which works simply because our "separate projects" are really just
in the same SVN repo.

So, is there a way I can move a file between two git repositories,
maintaining the change history?  I guess it would be something like
"apply all patches that this file was involved in, but ONLY apply the
ones affecting this file, to the new repo, then delete from the old"?

Re: move files between disparate repos and maintain version history

[Jeff King]

On Mon, Mar 02, 2009 at 12:30:58PM -0800, davetron5000 wrote:

> So, is there a way I can move a file between two git repositories,
> maintaining the change history?  I guess it would be something like
> "apply all patches that this file was involved in, but ONLY apply the
> ones affecting this file, to the new repo, then delete from the old"?

Yes, that's more or less how you would do it. There are actually two
separate issues, and each has two possible solutions.

Issue 1 is moving the history to the new repo.

One solution is, as you guessed, to export as patches and import into
the new repo:

  cd /path/to/app
  git format-patch --stdout --root -- <path> >~/patches
  cd /path/to/core
  git am ~/patches

where <path> can be a file, directory, or a list of either or both.
This should work fine if the history of that path is linear, since a
list of patches is, by definition, linear. You can see the "shape" of
the history with:

  cd /path/to/app
  gitk -- <path>

The other solution is to actually rewrite a version of git history that
sees only those paths, then merge it in. That will retain the actual
shape of the history. You can do this using "git filter-branch":

  cd /path/to/app
  # we do our work on a temporary branch
  git branch just-path
  # the actual filter; note you could do more than just grep here
  # if you wanted to munge the pathnames
  git filter-branch --index-filter '
    git ls-files -s | grep path |
      GIT_INDEX_FILE="$GIT_INDEX_FILE.new" git update-index --index-info &&
    mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"
  ' --prune-empty just-path

  # you can inspect just-path at this point with "gitk just-path"
  cd /path/to/core
  # and then pull it into core's history
  git pull /path/to/app just-path

The second issue is what you want to have happen in the "app"
repository. Presumably you no longer want the moved files there. You
_could_ filter-branch to pretend as if they were always in "core" and
never in "app". But that is probably not a good idea because:

  1. You are rewriting history, which will make merging more complex for
     your users.

  2. You may be breaking historical builds which expect the files to be
     there in the past.

So I think you're probably best to just remove them with a new commit
and have the duplicated history in both.

-Peff

PS I think you might be able to replace the filter-branch invocation
with a fast-export / fast-import pipeline, but I haven't tried.

Re: move files between disparate repos and maintain version history

[David Copeland]

The first option worked, insomuch the history of diffs is preserved,
but the dates are all today.  The second option was a little over my
head; is the idea there that you are setting up a branch that has ONLY
the files I care about (with all their history), and then I pull from
the other repo as if they are related?  That seems like it might
preserve the dates...

Dave

On Mon, Mar 2, 2009 at 11:13 PM, Jeff King <peff@peff.net> wrote:
> On Mon, Mar 02, 2009 at 12:30:58PM -0800, davetron5000 wrote:
>
>> So, is there a way I can move a file between two git repositories,
>> maintaining the change history?  I guess it would be something like
>> "apply all patches that this file was involved in, but ONLY apply the
>> ones affecting this file, to the new repo, then delete from the old"?
>
> Yes, that's more or less how you would do it. There are actually two
> separate issues, and each has two possible solutions.
>
> Issue 1 is moving the history to the new repo.
>
> One solution is, as you guessed, to export as patches and import into
> the new repo:
>
>  cd /path/to/app
>  git format-patch --stdout --root -- <path> >~/patches
>  cd /path/to/core
>  git am ~/patches
>
> where <path> can be a file, directory, or a list of either or both.
> This should work fine if the history of that path is linear, since a
> list of patches is, by definition, linear. You can see the "shape" of
> the history with:
>
>  cd /path/to/app
>  gitk -- <path>
>
> The other solution is to actually rewrite a version of git history that
> sees only those paths, then merge it in. That will retain the actual
> shape of the history. You can do this using "git filter-branch":
>
>  cd /path/to/app
>  # we do our work on a temporary branch
>  git branch just-path
>  # the actual filter; note you could do more than just grep here
>  # if you wanted to munge the pathnames
>  git filter-branch --index-filter '
>    git ls-files -s | grep path |
>      GIT_INDEX_FILE="$GIT_INDEX_FILE.new" git update-index --index-info &&
>    mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"
>  ' --prune-empty just-path
>
>  # you can inspect just-path at this point with "gitk just-path"
>  cd /path/to/core
>  # and then pull it into core's history
>  git pull /path/to/app just-path
>
> The second issue is what you want to have happen in the "app"
> repository. Presumably you no longer want the moved files there. You
> _could_ filter-branch to pretend as if they were always in "core" and
> never in "app". But that is probably not a good idea because:
>
>  1. You are rewriting history, which will make merging more complex for
>     your users.
>
>  2. You may be breaking historical builds which expect the files to be
>     there in the past.
>
> So I think you're probably best to just remove them with a new commit
> and have the duplicated history in both.
>
> -Peff
>
> PS I think you might be able to replace the filter-branch invocation
> with a fast-export / fast-import pipeline, but I haven't tried.
>

Re: move files between disparate repos and maintain version history

[Jeff King]

On Tue, Mar 03, 2009 at 11:58:42AM -0500, David Copeland wrote:

> The first option worked, insomuch the history of diffs is preserved,
> but the dates are all today.

That's odd. It works fine here. Can you confirm that the correct dates
in the "patches" file (i.e., the output of format-patch)? What are you
using to look at the patches? Note that gitk will show you both the
"committer" and the "author" fields. The "author" field should have the
original author and time of the patch, but the "committer" will be you,
today.

> The second option was a little over my head; is the idea there that
> you are setting up a branch that has ONLY the files I care about (with
> all their history), and then I pull from the other repo as if they are
> related?  That seems like it might preserve the dates...

Yes, that is exactly what is happening in the second example.

-Peff

Re: move files between disparate repos and maintain version history

[David Copeland]

The patch file looks correct.  I'm wondering if this is a result of
both repos being connected to svn?

my process was:

- format patch
- go to other repo
- git svn rebase
- apply patch
- git svn dcommit

Could dcommit change the dates since, to svn, they are appear as
commits right now?

Dave

On Tue, Mar 3, 2009 at 12:18 PM, Jeff King <peff@peff.net> wrote:
> On Tue, Mar 03, 2009 at 11:58:42AM -0500, David Copeland wrote:
>
>> The first option worked, insomuch the history of diffs is preserved,
>> but the dates are all today.
>
> That's odd. It works fine here. Can you confirm that the correct dates
> in the "patches" file (i.e., the output of format-patch)? What are you
> using to look at the patches? Note that gitk will show you both the
> "committer" and the "author" fields. The "author" field should have the
> original author and time of the patch, but the "committer" will be you,
> today.
>
>> The second option was a little over my head; is the idea there that
>> you are setting up a branch that has ONLY the files I care about (with
>> all their history), and then I pull from the other repo as if they are
>> related?  That seems like it might preserve the dates...
>
> Yes, that is exactly what is happening in the second example.
>
> -Peff
>

Re: move files between disparate repos and maintain version history

[Peter Baumann]

On Tue, Mar 03, 2009 at 01:08:17PM -0500, David Copeland wrote:
> The patch file looks correct.  I'm wondering if this is a result of
> both repos being connected to svn?
> 
> my process was:
> 
> - format patch
> - go to other repo

> - git svn rebase

This will change the date of commits already in svn because it uses
git rebase (with all its problems, see its manpage)

> - apply patch
> - git svn dcommit
> 
> Could dcommit change the dates since, to svn, they are appear as
> commits right now?
> 
> Dave
> 
> On Tue, Mar 3, 2009 at 12:18 PM, Jeff King <peff@peff.net> wrote:
> > On Tue, Mar 03, 2009 at 11:58:42AM -0500, David Copeland wrote:
> >
> >> The first option worked, insomuch the history of diffs is preserved,
> >> but the dates are all today.
> >
> > That's odd. It works fine here. Can you confirm that the correct dates
> > in the "patches" file (i.e., the output of format-patch)? What are you
> > using to look at the patches? Note that gitk will show you both the
> > "committer" and the "author" fields. The "author" field should have the
> > original author and time of the patch, but the "committer" will be you,
> > today.
> >
> >> The second option was a little over my head; is the idea there that
> >> you are setting up a branch that has ONLY the files I care about (with
> >> all their history), and then I pull from the other repo as if they are
> >> related?  That seems like it might preserve the dates...
> >
> > Yes, that is exactly what is happening in the second example.
> >
> > -Peff

Re: move files between disparate repos and maintain version history

[Jeff King]

On Tue, Mar 03, 2009 at 01:08:17PM -0500, David Copeland wrote:

> The patch file looks correct.  I'm wondering if this is a result of
> both repos being connected to svn?
> 
> my process was:
> 
> - format patch
> - go to other repo
> - git svn rebase
> - apply patch
> - git svn dcommit
> 
> Could dcommit change the dates since, to svn, they are appear as
> commits right now?

I know very little about git-svn. But you could test your theory by
checking the dates between the penultimate and ultimate steps.

-Peff