[git-svn] [FEATURE-REQ] track merges from git

[git-svn] [FEATURE-REQ] track merges from git

[Ximin Luo]

Hi,

I'm have 2 separate svn projects from googlecode imported into a single git
repo. One is a semi-fork of the other, so I thought I'd be able to use git's
merge feature to repeatedly merge from the mother project (and possibly vice
versa too).

However, this doesn't happen. I "git pull" and this works fine, but when I "git
svn dcommit" back into svn, this rewrites my git history and it loses track of
the merge (and next time I try to pull, the same conflicts appear).

For now I just have a .git/info/grafts, but this doesn't get exported anywhere,
so if other people "git svn clone" from svn, or "git clone" from my git repo,
they don't get the merge information.

It would be nice if git-svn saved the merge info somewhere instead of getting
rid of it. #git tells me this is impossible at the moment, hence the mail.
Relevant parts of the convo are pasted below.

I understand if this is a low priority, but I don't think it would be a major
PITA to implement (some suggestions are listed in the convo log). And it'd be
useful for people converting from svn to git.

Thanks for your time.

X

P.S. please don't troll me.

(17:13:10) The topic for #git is: 1.6.4.1 | Homepage: http://git-scm.com |
Everyone asleep or clueless? Try git@vger.kernel.org | Git User's Survey 2009!
http://tinyurl.com/GitSurvey2009 | Channel log http://tinyurl.com/gitlog |
Mailing list archives: http://tinyurl.com/gitml | Gits on git:
http://tinyurl.com/gittalks | Pastebin: http://gist.github.com/ | GSoC '09:
http://socghop.appspot.com/org/home/google/gsoc2009/git
(17:13:14) infinity0: hi
(17:13:21) infinity0: i've used git-svn to import two svn repo
(17:13:23) infinity0: repos*
(17:13:28) infinity0: and used git to merge the two
(17:13:45) infinity0: the problem is, when i git-svn dcommit back to svn,
git-svn rewrites my git history
(17:13:50) infinity0: and loses the merge i just did
(17:14:01) offby1: infinity0: of course
(17:14:04) infinity0: how do i get it to retain knowledge of the merge?
(17:14:11) offby1: infinity0: you don't.  Next questions.
(17:14:16) infinity0: why not?
(17:14:29) offby1: svn is incapable of storing a merge, at least in the sense
that we git people use the term "merge"
(17:14:46) Grum: you should be able to store the result of a merge as a commit
(17:14:51) offby1: sure
(17:14:57) infinity0: sure, but why does git-svn have to rewrite my *git*
history to remove knowledge of the merge?
(17:15:02) offby1: but not as a "merge commit", whatever that might mean in svn
(17:15:35) Grum: because it has to be representative of the svn repo after you
dcommit there obviously
(17:15:42) offby1: infinity0: it's trying to mirror the svn repository in your
git repository.  I assume the original, un-rewritten commits are still in your
git repository; they're just not pointed at by any branch.  Poke around in the
reflog; I imagine you'll find 'em in there
(17:16:09) infinity0: ok, but that's not useful if they're dangling
(17:16:26) infinity0: it's trying to mirror the svn repo yes... but as you
said, svn doesn't know about merges
(17:16:26) ***offby1 idly wonders if it'd be possible for git svn to indeed
store merge commits, by applying the appropriate svn:mergeinfo properties
(17:16:40) infinity0: i read a thread where it says those are different things
(17:16:41) offby1: infinity0: I suspect you're using git svn for something for
which it wasn't designed.
(17:17:17) infinity0: would it be possible, in theory, to have git-svn store
the git merge information in eg. the same way it stores the git-svn tag in the
svn commit message
(17:17:33) Grum: then just use svn?
(17:17:37) Grum: and a postit?
(17:18:01) infinity0: i'm trying to link two separate svn repos together via git
(17:18:17) Grum: and that is just what offby1 said
(17:18:30) infinity0: "what" is
(17:18:40) Grum: I suspect you're using git svn for something for which it
wasn't designed.
(17:18:42) infinity0: as you all are saying, git merges and svn "merges" are
different things
(17:18:58) infinity0: ok, but it would be possible to make git-svn have this
functionality? or not
(17:18:59) offby1: certainly
(17:19:16) offby1: I fear not, since Eric Wong seems like a smart fella; if it
were doable, I suspect he'd have done it already.
(17:19:21) offby1: But then ... who knows, maybe he's busy.
(17:20:07) infinity0: well afaic it would just involve adding some extra
git-svn info to the svn commit messages, but meh
(17:20:10) infinity0: i'll go file a bug
(17:21:14) offby1: infinity0: if it were me, I'd shy away from cramming more
junk into the svn commit messages; that strikes me as an unreliable storage medium
(17:21:22) offby1: I'd use properties instead
(17:21:24) Grum: ok lets do this properly
(17:21:31) Grum: why do you want to 'merge' 2 svn 'repos' this way?
(17:21:34) Grum: as you are not actually merging them
(17:21:45) offby1: Grum: go, man, go!
(17:21:58) infinity0: what do you mean "not actually merging them"
(17:22:17) infinity0: they are "actually merged" in git
(17:22:24) infinity0: then i git-svn dcommit and they become unmerged again
(17:22:34) Grum: why do you want to merge them?
(17:22:42) infinity0: so i can grab changes from one into the other
(17:22:42) Grum: and what is your goal with dcommitting this?
(17:22:57) infinity0: long story short, the two projects use svn on google code
(17:23:06) infinity0: one is a semi-fork of the other
(17:23:10) offby1: aaahhh
(17:23:13) infinity0: i need to pull changes quite often
(17:23:17) offby1: and you want to keep them in sync, kinda
(17:23:19) infinity0: yeah
(17:23:28) offby1: yeesh, dunno how I'd do that
(17:24:09) infinity0: ok well i guess the only thing i can do atm is file a bug
for git-svn and tell people to manually add the .git/info/grafts
(17:24:12) infinity0: but thanks for your info
(17:27:24) infinity0: uh, where is the git bug tracker
(17:27:40) Grum: its the mailinglist =)
(17:27:41) sitaram: no bugs, so no tracker
(17:27:44) sitaram: :
(17:27:45) sitaram: :D
(17:27:51) infinity0: lol
(17:27:59) Grum: and erm you want a feature, its nt a bug you are reporting
(17:27:59) Grum: s
(17:28:00) infinity0: ok i'll post on the mailing list then.. *grumble*
(17:28:08) infinity0: ok, *feature request :p
(17:28:08) Grum: you are not using the tool for what it is meant to
(17:28:13) Grum: and you can still do waht you want in either case
(17:28:17) Grum: just not by merging
(17:28:22) infinity0: well, how then?
(17:29:48) infinity0: aww fuck 120 messages a day? oh come on... are
non-members allowed to post to it?
(17:30:00) Grum: infinity0: just 120 yeah, and yeah they are
(17:30:06) infinity0: ah ok

Re: [git-svn] [FEATURE-REQ] track merges from git

[Bryan Donlan]

On Wed, Aug 26, 2009 at 12:42 PM, Ximin Luo<xl269@cam.ac.uk> wrote:
> Hi,
>
> I'm have 2 separate svn projects from googlecode imported into a single git
> repo. One is a semi-fork of the other, so I thought I'd be able to use git's
> merge feature to repeatedly merge from the mother project (and possibly vice
> versa too).
>
> However, this doesn't happen. I "git pull" and this works fine, but when I "git
> svn dcommit" back into svn, this rewrites my git history and it loses track of
> the merge (and next time I try to pull, the same conflicts appear).
>
> For now I just have a .git/info/grafts, but this doesn't get exported anywhere,
> so if other people "git svn clone" from svn, or "git clone" from my git repo,
> they don't get the merge information.
>
> It would be nice if git-svn saved the merge info somewhere instead of getting
> rid of it. #git tells me this is impossible at the moment, hence the mail.
> Relevant parts of the convo are pasted below.
>
> I understand if this is a low priority, but I don't think it would be a major
> PITA to implement (some suggestions are listed in the convo log). And it'd be
> useful for people converting from svn to git.
>
> Thanks for your time.
>
> X
>
> P.S. please don't troll me.

For the particular use case you have, I suspect svk would be a better
tool for merging between those two projects...
git-svn is rather specifically designed to only deal with a single
upstream repository, you see, and it isn't very easy to change this to
accept multiple repos.

Re: [git-svn] [BUG] merge-tracking inconsistencies; Was: [FEATURE-REQ] track merges from git

[Ximin Luo]

Ximin Luo wrote:
> the tip of trunk is NOT a tip of test1

er, that should read "the tip of trunk is NOT a parent of the tip of test1"

X

Re: [git-svn] [BUG] merge-tracking inconsistencies; Was: [FEATURE-REQ] track merges from git

[Ximin Luo]

Ximin Luo wrote:
> The only difference between the two runs is that in the unfucked version, we
> run "git svn dcommit" after every git commit.

Hmm, now that I think about it, the "bug" would be quite hard to "fix"...

Basically, it happens if you try to dcommit a commit A which has two parents, B
and X, where X is in a different branch, and hasn't already been dcommited. It
would seem that there isn't (in general) a way to detect whether X would become
(ie. in the future) a dcommitted svn commit - and actually this might not even
be the case, if eg. someone "svn commited" before we could get that dcommit in.

However about having git-svn outputting a warning when it detects merge
commits, one of whose parents is *not* a dcommitted commit, but does belongs to
a branch that is also being tracked by git-svn?

Something like "Warning: commit aaaa has parent bbbb; however, parent bbbb has
not been dcommited to the remote svn yet. If you proceed with this dcommit, the
merge history will be lost; to preserve the history, dcommit the branch
containing bbbb instead and then continue to dcommit this branch"?

X

Re: [git-svn] [FEATURE-REQ] track merges from git

[Eric Wong]

Ximin Luo <xl269@cam.ac.uk> wrote:
> Hi,

Hi, sorry for the late reply.

> I'm have 2 separate svn projects from googlecode imported into a single git
> repo. One is a semi-fork of the other, so I thought I'd be able to use git's
> merge feature to repeatedly merge from the mother project (and possibly vice
> versa too).
> 
> However, this doesn't happen. I "git pull" and this works fine, but when I "git
> svn dcommit" back into svn, this rewrites my git history and it loses track of
> the merge (and next time I try to pull, the same conflicts appear).

You may want to try the "set-tree" function of git svn instead of
dcommit, it was originally named "commit" back in the day  set-tree does
not rewrite any history.

It fell out of favor because you could end up with a lot of non-linear
history making it difficult for sharing diffs with SVN-using cow-orkers.

It is useful if you don't want to share your individual changesets, but
push your work upstream to the SVN repos as one big ugly change.

But if you want a staircase effect in gitk, set-tree can be used to make
individual commits where every change ends up as a merge (and you'll see
two commits for every change you made)

> For now I just have a .git/info/grafts, but this doesn't get exported anywhere,
> so if other people "git svn clone" from svn, or "git clone" from my git repo,
> they don't get the merge information.
> 
> It would be nice if git-svn saved the merge info somewhere instead of getting
> rid of it. #git tells me this is impossible at the moment, hence the mail.
> Relevant parts of the convo are pasted below.

It should actually get logged to the unhandled.log file, but right
now I'm not aware of any tools that reads that file (which is designed
to be machine parseable).

> I understand if this is a low priority, but I don't think it would be a major
> PITA to implement (some suggestions are listed in the convo log). And it'd be
> useful for people converting from svn to git.

I've considered stuffing git commit information in SVN properties,
but that information is likely to not even be useful to git users
other than yourself

> Thanks for your time.
> 
> X
> 
> P.S. please don't troll me.

Don't worry, despite my domain name, it's been a long time since I've
done any real trolling :)

> (17:13:14) infinity0: hi
> (17:13:21) infinity0: i've used git-svn to import two svn repo
> (17:13:23) infinity0: repos*
> (17:13:28) infinity0: and used git to merge the two
> (17:13:45) infinity0: the problem is, when i git-svn dcommit back to svn,
> git-svn rewrites my git history
> (17:13:50) infinity0: and loses the merge i just did
> (17:14:01) offby1: infinity0: of course
> (17:14:04) infinity0: how do i get it to retain knowledge of the merge?
> (17:14:11) offby1: infinity0: you don't.  Next questions.
> (17:14:16) infinity0: why not?
> (17:14:29) offby1: svn is incapable of storing a merge, at least in the sense
> that we git people use the term "merge"

I'm not sure if that statement is still true with newer SVN versions;
I still haven't had the time to look too hard at newer SVN features.

On the other hand, SVN could be storing "merges" that git simply can't
track as merges; SVN branches and directories are interchangeable, so
SVN could be merging from places that git isn't tracking at all or
merging individual files.

> (17:14:46) Grum: you should be able to store the result of a merge as a commit
> (17:14:51) offby1: sure
> (17:14:57) infinity0: sure, but why does git-svn have to rewrite my *git*
> history to remove knowledge of the merge?
> (17:15:02) offby1: but not as a "merge commit", whatever that might mean in svn
> (17:15:35) Grum: because it has to be representative of the svn repo after you
> dcommit there obviously
> (17:15:42) offby1: infinity0: it's trying to mirror the svn repository in your
> git repository.  I assume the original, un-rewritten commits are still in your
> git repository; they're just not pointed at by any branch.  Poke around in the
> reflog; I imagine you'll find 'em in there
> (17:16:09) infinity0: ok, but that's not useful if they're dangling
> (17:16:26) infinity0: it's trying to mirror the svn repo yes... but as you
> said, svn doesn't know about merges
> (17:16:26) ***offby1 idly wonders if it'd be possible for git svn to indeed
> store merge commits, by applying the appropriate svn:mergeinfo properties

Some of should be mappable to git merges, it depends on the project and
the type of merge done.   As I said earlier, SVN could be merging
from places git svn doesn't know about and can't track...  SVN can
get extremely complicated :<

I'm not sure how widely used that feature is used in the SVN world,
either.

> (17:16:40) infinity0: i read a thread where it says those are different things
> (17:16:41) offby1: infinity0: I suspect you're using git svn for something for
> which it wasn't designed.
> (17:17:17) infinity0: would it be possible, in theory, to have git-svn store
> the git merge information in eg. the same way it stores the git-svn tag in the
> svn commit message
> (17:17:33) Grum: then just use svn?
> (17:17:37) Grum: and a postit?

I don't agree with having git-specific metadata on the SVN side itself.
Often times that git-specific metadata has SHA1s unique to the user that
committed it, so it wouldn't be useful to anyone else unless users are
merging from each others git repos (which is not an easy/natural
workflow if SVN is the mainline).  Patch exchange is more
reliable/easier...

I've also worked in places where alternative tools are frowned upon, so
sending git-specific metadata over to SVN should always be optional.

The majority of folks I've worked with on SVN-hosted projects have never
known about my git usage (that is changing as git popularity increases,
however).

> (17:18:01) infinity0: i'm trying to link two separate svn repos together via git
> (17:18:17) Grum: and that is just what offby1 said
> (17:18:30) infinity0: "what" is
> (17:18:40) Grum: I suspect you're using git svn for something for which it
> wasn't designed.
> (17:18:42) infinity0: as you all are saying, git merges and svn "merges" are
> different things
> (17:18:58) infinity0: ok, but it would be possible to make git-svn have this
> functionality? or not
> (17:18:59) offby1: certainly
> (17:19:16) offby1: I fear not, since Eric Wong seems like a smart fella; if it
> were doable, I suspect he'd have done it already.
> (17:19:21) offby1: But then ... who knows, maybe he's busy.

I'm not smart but I am busy :)

Summary of the git svn merge tracking situation:

Mapping git <-> git merges to SVN:

  * already doable for the committing user with set-tree,
    but makes history ugly for:

    a) yourself (with every commit set-tree'd individually)
    b) SVN users (single set-tree with the newest commit)
    c) all of the above (varying granularity)

  * Pushing git metadata to SVN will annoy SVN-only users

  * Putting git metadata in SVN may not be useful since SHA1s
    may be specific to the user that made that commit.

Mapping SVN <-> SVN merges to SVN via git svn:

  * most likely to be doable, they'll become plain SVN <-> SVN merges,
    see problems with getting SVN <-> SVN merges back to git, however...

Mapping SVN <-> SVN merges to git:

  * SVN can represent merges that git can't, SVN can be/is extremely
    complicated when it comes to merges.

  * I don't see many projects (I care about) use SVN merge tracking,
    which projects actually use it?  Maybe it's still too new and
    distros/users are behind the upgrade curve...

I've probably missed some, I've been dozing off while replying to
emails...

-- 
Eric Wong

Re: [git-svn] [FEATURE-REQ] track merges from git

[Ximin Luo]

Eric Wong wrote:
> You may want to try the "set-tree" function of git svn instead of
> dcommit, it was originally named "commit" back in the day  set-tree does
> not rewrite any history.
> 
> It fell out of favor because you could end up with a lot of non-linear
> history making it difficult for sharing diffs with SVN-using cow-orkers.
> 
> It is useful if you don't want to share your individual changesets, but
> push your work upstream to the SVN repos as one big ugly change.
> 
> But if you want a staircase effect in gitk, set-tree can be used to make
> individual commits where every change ends up as a merge (and you'll see
> two commits for every change you made)

My problem no longer requires using set-tree (see below), but just to let you
know that when I try to set-tree, it gives:

  $ git svn set-tree HEAD
  config --get svn-remote.svn.fetch :refs/remotes/git-svn$: command returned
error: 1

In my repo, "remotes/git-svn" doesn't exist; I have

  $ git branch -a
  * master
    test1
    remotes/test1
    remotes/trunk

but the manual doesn't tell me how to select an svn-remote that's not "git-svn"..

>> (17:16:40) infinity0: i read a thread where it says those are different things
>> (17:16:41) offby1: infinity0: I suspect you're using git svn for something for
>> which it wasn't designed.
>> (17:17:17) infinity0: would it be possible, in theory, to have git-svn store
>> the git merge information in eg. the same way it stores the git-svn tag in the
>> svn commit message
>> (17:17:33) Grum: then just use svn?
>> (17:17:37) Grum: and a postit?
> 
> I don't agree with having git-specific metadata on the SVN side itself.
> Often times that git-specific metadata has SHA1s unique to the user that
> committed it, so it wouldn't be useful to anyone else unless users are
> merging from each others git repos (which is not an easy/natural
> workflow if SVN is the mainline).  Patch exchange is more
> reliable/easier...
> 
> I've also worked in places where alternative tools are frowned upon, so
> sending git-specific metadata over to SVN should always be optional.
> 
> The majority of folks I've worked with on SVN-hosted projects have never
> known about my git usage (that is changing as git popularity increases,
> however).
> 
>> (17:18:01) infinity0: i'm trying to link two separate svn repos together via git
>> (17:18:17) Grum: and that is just what offby1 said
>> (17:18:30) infinity0: "what" is
>> (17:18:40) Grum: I suspect you're using git svn for something for which it
>> wasn't designed.
>> (17:18:42) infinity0: as you all are saying, git merges and svn "merges" are
>> different things
>> (17:18:58) infinity0: ok, but it would be possible to make git-svn have this
>> functionality? or not
>> (17:18:59) offby1: certainly
>> (17:19:16) offby1: I fear not, since Eric Wong seems like a smart fella; if it
>> were doable, I suspect he'd have done it already.
>> (17:19:21) offby1: But then ... who knows, maybe he's busy.
> 
> I'm not smart but I am busy :)
> 
> Summary of the git svn merge tracking situation:
> 
> Mapping git <-> git merges to SVN:
> 
>   * already doable for the committing user with set-tree,
>     but makes history ugly for:
> 
>     a) yourself (with every commit set-tree'd individually)
>     b) SVN users (single set-tree with the newest commit)
>     c) all of the above (varying granularity)
> 
>   * Pushing git metadata to SVN will annoy SVN-only users
> 
>   * Putting git metadata in SVN may not be useful since SHA1s
>     may be specific to the user that made that commit.
> 
> Mapping SVN <-> SVN merges to SVN via git svn:
> 
>   * most likely to be doable, they'll become plain SVN <-> SVN merges,
>     see problems with getting SVN <-> SVN merges back to git, however...
> 
> Mapping SVN <-> SVN merges to git:
> 
>   * SVN can represent merges that git can't, SVN can be/is extremely
>     complicated when it comes to merges.
> 
>   * I don't see many projects (I care about) use SVN merge tracking,
>     which projects actually use it?  Maybe it's still too new and
>     distros/users are behind the upgrade curve...
> 
> I've probably missed some, I've been dozing off while replying to
> emails...
> 

Actually, it turns out that my original problem is simpler than any of these
scenarios. What I was doing was git <-> git merges, where both of these git
branches were tracking *different* SVN branches (in my original case, from
different repos; in my simplified test case, from the same repo).

Consider this scenario:

----A0*-----A1---+
     \            \
      B0*----B1----B2

branchA: A1
branchB: B2

Where A0* and B0* have both been dcommited into their SVN branches, but A1, B1
and B2 are present in the git repo only. A0* and B0* have the "git-svn-id" tag,
and show my svn committer name; A1, B1 and B2 are still pure git commits, and
show my git commiter name.

Scenario 1:

If HEAD is at B2, and we try to "git-svn dcommit", then what happens currently
is, git-svn will dcommit B1 then B2, re-writing them in the process (adding
git-svn-id and using the svn credentials instead of git credentials); however,
it will *miss out* dcommitting A1. So we get this:

----A0*-----A1----+
     \             \
      B0*----B1*----B2*

branchA: A1
branchB: B2*

where B1* and B2* are the re-written versions of B1 and B2, with the added
git-svn-id and the svn committer name instead of the git committer name. There
isn't a problem yet; but when we switch to branchA and dcommit, we get this:

      +-----A1*
     /
----A0*-----A1----+
     \             \
      B0*----B1*----B2*

branchA: A1*
branchB: B2*

Where A1* is the re-written version of A1. But B2* still has A1 as a parent,
and now we have an extra "duplicate" commit in our git repo. What's even worse
is that A1* is **not a parent** of B2*, and so future merges on the branches
will potentially need to resolve conflicts that were resolved already when
merging (A1, B1) to B2.

Scenario 2:

If however, we dcommit A1, then B1, then B2, the commits will be re-written in
such a way that the proper merge history is preserved, including the correct
parents.

In one of the follow-up emails to my original posting, I supplied a test script
(gitsvntest.sh) which demonstrates the 2 scenarios. You can use a GUI to review
the history graphs. I can re-send it if you can't find that email.

I'm not sure how hard this is to fix; I guess it would involve making dcommit
detecting the case where a commit has more than 1 parent, and recursing down
all the parents to see if they are tracking an svn branch.

At the very least, I think this situation can be detected and the user warned.
In switching from svn to git, git-svn was very helpful to me, but this
behaviour confused me completely - sometimes things worked fine, depending on
the order in which I dcommited stuff, so it was incredibly hard to figure out,
especially since at that time I had no understanding of the concept of object
graphs and rewriting commits and that sort of thing.

X

Re: [git-svn] [FEATURE-REQ] track merges from git

[Jakub Narebski]

Ximin Luo wrote:

> For now I just have a .git/info/grafts, but this doesn't get exported anywhere,
> so if other people "git svn clone" from svn, or "git clone" from my git repo,
> they don't get the merge information.

In future version of git (I'm not sure if it is in master yet) there
would be refs/replace feature (grafts-like), which can be exported.
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git