How do I manage this setup with git-svn and/or git remotes?

How do I manage this setup with git-svn and/or git remotes?

[David Kastrup]

I have a Subversion repository which contains the following:

https://repo.here/svn/dsp/{trunk,tags,branches}
and
https://repo.here/svn/projects/great/{trunk/tags/branches}

I have those currently checked out into separate git repositories.
However,

https://repo.here/svn/projects/great/{trunk/tags/branches}

contain a subdirectory dsp that has been copied via Subversion from
the main dsp trunk.  Subsequently this copy and the original dsp have
progressed.

Now is there any chance to set up a git structure that will me allow
to let _git_ perform merges between the standalone dsp project and the
part that has started off as a copy of it in a subdirectory from
projects/great, so that I have a merge history in my git mirror?

Or is basically the only thing I can do to create selective patches
and apply them, thus not tracking the merges?

-- 
David Kastrup

Re: How do I manage this setup with git-svn and/or git remotes?

[Linus Torvalds]

On Fri, 17 Aug 2007, David Kastrup wrote:
> 
> Now is there any chance to set up a git structure that will me allow
> to let _git_ perform merges between the standalone dsp project and the
> part that has started off as a copy of it in a subdirectory from
> projects/great, so that I have a merge history in my git mirror?

Yes. That's what git "submodule" support is all about. You could create 
that "dsp" project as its own git project, and then include it within the 
bigger project as a submodule. Then, that "dsp" thing is really a totally 
independent git project in its own right, with git support for just 
"tying" it into the superproject.

A few words of caution:

 - while the low-level core submodule support has been around for a while
   now, the actual "porcelain" level helper stuff is new to 1.5.3 (which 
   is unreleased, so you'd have to use the current git "master" branch, of 
   course)

 - submodules are (very much by design) meant to be git projects in their
   own right, and kept separate. That very explicit separation means that 
   you will *see* it as a separate project, and you may decide that it's 
   not worth the extra setup/complexity if the "dsp" thing just isn't big 
   enough or the merge complexity just isn't worth the effort.

 - not very many people use it. The msysgit people seem to be using it,
   and Andy Parkins has been using it for a while (and seems happy), but I 
   think you basically end up having less documentation and somewhat less 
   support for it (and it doesn't get the same kind of testing that the 
   "basic" git behaviours get).

Another alternative is to do what git has long done with "gitk": you can 
maintain a separate project and just merge it directly into another git 
project, and it works fine that way, but it gets impossible to merge back 
and forth between the two projects (you can only merge one way: make all 
the major changes in the "dsp" project, and then you can just merge it 
into the project that uses it (but if you fix things in the bigger 
project, you can't merge the fixes back, you'll have to export the fixes 
as patches and do them in the "dsp" tree).

			Linus

Re: How do I manage this setup with git-svn and/or git remotes?

[David Kastrup]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> On Fri, 17 Aug 2007, David Kastrup wrote:
>> 
>> Now is there any chance to set up a git structure that will me
>> allow to let _git_ perform merges between the standalone dsp
>> project and the part that has started off as a copy of it in a
>> subdirectory from projects/great, so that I have a merge history in
>> my git mirror?
>
> Yes. That's what git "submodule" support is all about.  You could
> create that "dsp" project as its own git project, and then include
> it within the bigger project as a submodule. Then, that "dsp" thing
> is really a totally independent git project in its own right, with
> git support for just "tying" it into the superproject.

But it isn't an independent git project: the superproject has its
_own_ copy of dsp, with its _own_ specific commits and fixes that are
not supposed to ever end up in the dsp "mothership".  There are
sometimes cross merges, but the stuff in the "dsp" subdirectory of
"great" is maintained completely together with the branches of
"great": tags, branches and all.

But I would like to be able to merge this _subdirectory_ with branches
from the "mothership" dsp from which it was originally copied.

With Subversion, I can actually merge files in different projects of
the repository even when they are in different directory levels.  Of
course, since Subversion does not track any merge info, that is not an
accomplishment.

I don't see quite how this would work with submodules, and in
particular not when git-svn gets involved as well.

> A few words of caution:
>
>  - while the low-level core submodule support has been around for a
>  while now, the actual "porcelain" level helper stuff is new to
>  1.5.3 (which is unreleased, so you'd have to use the current git
>  "master" branch, of course)

Have to for a variety of reasons, anyway.

>  - submodules are (very much by design) meant to be git projects in
>  their own right, and kept separate. That very explicit separation
>  means that you will *see* it as a separate project, and you may
>  decide that it's not worth the extra setup/complexity if the "dsp"
>  thing just isn't big enough or the merge complexity just isn't
>  worth the effort.

And that is the problem here: in this case it does not make sense to
see it as a separate project, and in particular, it needs to be in
synch with the tags/branches of the superproject, and particularly
while I am using git-svn.

> Another alternative is to do what git has long done with "gitk": you
> can maintain a separate project and just merge it directly into
> another git project, and it works fine that way, but it gets
> impossible to merge back and forth between the two projects (you can
> only merge one way: make all the major changes in the "dsp" project,
> and then you can just merge it into the project that uses it (but if
> you fix things in the bigger project, you can't merge the fixes
> back, you'll have to export the fixes as patches and do them in the
> "dsp" tree).

Well, that would be at least quite handy for propagating upstream dsp
fixes into project/great.  How do I merge one project into a
_subdirectory_ of another one?

-- 
David Kastrup

Re: How do I manage this setup with git-svn and/or git remotes?

[Linus Torvalds]

On Fri, 17 Aug 2007, David Kastrup wrote:
> 
> But it isn't an independent git project: the superproject has its
> _own_ copy of dsp, with its _own_ specific commits and fixes that are
> not supposed to ever end up in the dsp "mothership". 

Sure. And that's different from any git "branch" exactly how?

So you'd have different branches in the superproject - the way you always 
have when you have two copies of a git project. And then you merge between 
the two at will.

> With Subversion, I can actually merge files in different projects of
> the repository even when they are in different directory levels.  Of
> course, since Subversion does not track any merge info, that is not an
> accomplishment.

Right. Git can do that too. It's called "patches".

> And that is the problem here: in this case it does not make sense to
> see it as a separate project, and in particular, it needs to be in
> synch with the tags/branches of the superproject, and particularly
> while I am using git-svn.

I do agree that the "git svn" interaction may end up making it unworkable, 
if only because git-svn simply doesn't know/understand about submodules. 
But the submodule *would* be well-specified by the tags and branches in 
the superproject, so that shouldn't be a problem in itself.

> > Another alternative is to do what git has long done with "gitk": you
> > can maintain a separate project and just merge it directly into
> > another git project, and it works fine that way, but it gets
> > impossible to merge back and forth between the two projects (you can
> > only merge one way: make all the major changes in the "dsp" project,
> > and then you can just merge it into the project that uses it (but if
> > you fix things in the bigger project, you can't merge the fixes
> > back, you'll have to export the fixes as patches and do them in the
> > "dsp" tree).
> 
> Well, that would be at least quite handy for propagating upstream dsp
> fixes into project/great.  How do I merge one project into a
> _subdirectory_ of another one?

There's a special "subtree" merge that does exactly that: it basically is 
the normal recursive merge, except it merges into a subtree. I think 
that's how Junio does the "git-gui" merges. Junio?

			Linus

Re: How do I manage this setup with git-svn and/or git remotes?

[David Kastrup]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> On Fri, 17 Aug 2007, David Kastrup wrote:
>> 
>> But it isn't an independent git project: the superproject has its
>> _own_ copy of dsp, with its _own_ specific commits and fixes that are
>> not supposed to ever end up in the dsp "mothership". 
>
> Sure. And that's different from any git "branch" exactly how?

Different directory structure.  That can't easily be rectified as far
as I can see since the stuff still needs to get synched with
Subversion.

> So you'd have different branches in the superproject - the way you
> always have when you have two copies of a git project. And then you
> merge between the two at will.

The problem is the different location: root vs subdirectory.

>> With Subversion, I can actually merge files in different projects
>> of the repository even when they are in different directory levels.
>> Of course, since Subversion does not track any merge info, that is
>> not an accomplishment.
>
> Right. Git can do that too. It's called "patches".

Yup.  The question is just whether it is possible to do better than
that.

>> Well, that would be at least quite handy for propagating upstream
>> dsp fixes into project/great.  How do I merge one project into a
>> _subdirectory_ of another one?
>
> There's a special "subtree" merge that does exactly that: it
> basically is the normal recursive merge, except it merges into a
> subtree.

Sounds like just the thing.

> I think that's how Junio does the "git-gui" merges. Junio?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: How do I manage this setup with git-svn and/or git remotes?

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> On Fri, 17 Aug 2007, David Kastrup wrote:
>> 
>> But it isn't an independent git project: the superproject has its
>> _own_ copy of dsp, with its _own_ specific commits and fixes that are
>> not supposed to ever end up in the dsp "mothership". 
>
> Sure. And that's different from any git "branch" exactly how?
>
> So you'd have different branches in the superproject - the way you always 
> have when you have two copies of a git project. And then you merge between 
> the two at will.

My reading of the project David is talking about is that its dsp
project which is a "subproject" part gets non generic commits
within the context of the superproject --- which means (1) you
would have branches in the subproject not superproject, and (2)
once you did that, the subproject is not really a subproject
anymore, as you cannot merge that back to the standalone dsp
project without dragging the non-generic bits along with it.

> There's a special "subtree" merge that does exactly that: it basically is 
> the normal recursive merge, except it merges into a subtree. I think 
> that's how Junio does the "git-gui" merges. Junio?

Yes.  It has exactly the same semantics and limitations with the
gitk merge, but just merges into a sub directory.  Shawn cannot
easily pull the changes done inside git.git repository back to
git-gui.git proper.

Re: How do I manage this setup with git-svn and/or git remotes?

[David Kastrup]

Junio C Hamano <gitster@pobox.com> writes:

> Linus Torvalds <torvalds@linux-foundation.org> writes:
>
>> On Fri, 17 Aug 2007, David Kastrup wrote:
>>> 
>>> But it isn't an independent git project: the superproject has its
>>> _own_ copy of dsp, with its _own_ specific commits and fixes that are
>>> not supposed to ever end up in the dsp "mothership". 
>>
>> Sure. And that's different from any git "branch" exactly how?
>>
>> So you'd have different branches in the superproject - the way you always 
>> have when you have two copies of a git project. And then you merge between 
>> the two at will.
>
> My reading of the project David is talking about is that its dsp
> project which is a "subproject" part gets non generic commits within
> the context of the superproject --- which means (1) you would have
> branches in the subproject not superproject, and (2) once you did
> that, the subproject is not really a subproject anymore, as you
> cannot merge that back to the standalone dsp project without
> dragging the non-generic bits along with it.

Ok, I should perhaps should not make things harder than they are: the
superprojects, being particular to one customer each, don't really
branch (except that git-svn makes a git branch from every Subversion
tag).  The subproject is the one that has considerable branching and
merges.  What usually gets pulled into the superproject is a copy of a
stable subproject branch.  Once this copy is in, only fixes (from the
stable branch) or features (from the development branch) that the
customer definitely needs are merged into the superproject.  While
there might happen some subproject work in the customer branch, this
mostly happens during bugfixing for the customer, and the changes are
typically pulled back into the subproject proper at some point of
time.  Inside of the subproject tree, there is really no superproject
_development_ going on.

>> There's a special "subtree" merge that does exactly that: it
>> basically is the normal recursive merge, except it merges into a
>> subtree. I think that's how Junio does the "git-gui" merges. Junio?
>
> Yes.  It has exactly the same semantics and limitations with the
> gitk merge, but just merges into a sub directory.  Shawn cannot
> easily pull the changes done inside git.git repository back to
> git-gui.git proper.

Well, the other direction would be the most important one: merging or
cherrypicking selected changes in the subproject branches into the
superproject copy of the subproject.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: How do I manage this setup with git-svn and/or git remotes?

[David Kastrup]

David Kastrup <dak@gnu.org> writes:

> Junio C Hamano <gitster@pobox.com> writes:
>>
>> My reading of the project David is talking about is that its dsp
>> project which is a "subproject" part gets non generic commits within
>> the context of the superproject --- which means (1) you would have
>> branches in the subproject not superproject, and (2) once you did
>> that, the subproject is not really a subproject anymore, as you
>> cannot merge that back to the standalone dsp project without
>> dragging the non-generic bits along with it.
>
> Ok, I should perhaps should not make things harder than they are: the
> superprojects, being particular to one customer each, don't really
> branch (except that git-svn makes a git branch from every Subversion
> tag).  The subproject is the one that has considerable branching and
> merges.  What usually gets pulled into the superproject is a copy of a
> stable subproject branch.  Once this copy is in, only fixes (from the
> stable branch) or features (from the development branch) that the
> customer definitely needs are merged into the superproject.  While
> there might happen some subproject work in the customer branch, this
> mostly happens during bugfixing for the customer, and the changes are
> typically pulled back into the subproject proper at some point of
> time.  Inside of the subproject tree, there is really no superproject
> _development_ going on.

I think I got it.  My mistake was focusing all the time what I could
do with the git repository of "great" to facilitate two-way merges.

Instead I need to import great/trunk/dsp into a remote branch in my
_dsp_ git repository.  Since for git-svn, every Subversion directory
is as good to import as any other (there is no concept of a worktree
root in the repository) that should be all it takes.

I'll need to use the dsp repository when doing merge work, but apart
from that, I can work in the great repository r/w even while in the
dsp subdirectory.

If one could tell git in a remote section to
fetch/consider/synchronize/push just a subdirectory as the repo root,
then the same setup for bidirectional merges could be made to work
with projects like gitk.  Though I am fuzzy about the merge
information...  But that is a problem when pushing merges with private
branches, anyway, isn't it?

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: How do I manage this setup with git-svn and/or git remotes?

[Jakub Narebski]

David Kastrup wrote:

> Linus Torvalds <torvalds@linux-foundation.org> writes:
> 
>> On Fri, 17 Aug 2007, David Kastrup wrote:
>>> 
>>> Now is there any chance to set up a git structure that will me
>>> allow to let _git_ perform merges between the standalone dsp
>>> project and the part that has started off as a copy of it in a
>>> subdirectory from projects/great, so that I have a merge history in
>>> my git mirror?
>>
>> Yes. That's what git "submodule" support is all about.  You could
>> create that "dsp" project as its own git project, and then include
>> it within the bigger project as a submodule. Then, that "dsp" thing
>> is really a totally independent git project in its own right, with
>> git support for just "tying" it into the superproject.
> 
> But it isn't an independent git project: the superproject has its
> _own_ copy of dsp, with its _own_ specific commits and fixes that are
> not supposed to ever end up in the dsp "mothership".  There are
> sometimes cross merges, but the stuff in the "dsp" subdirectory of
> "great" is maintained completely together with the branches of
> "great": tags, branches and all.

Independent git project means independent clone of "dsp" repository,
perhaps a fork of "dsp" repository with some (superproject) specific
commits. Which is attached as subritectory of superproject.
 
> But I would like to be able to merge this _subdirectory_ with branches
> from the "mothership" dsp from which it was originally copied.

You would be able to, both from "mothership" to "submodule", and from
"submodule" (perhaps only some selected commits on 'maint' branch) to
"mothership".

Putting files of "dsp" project directly in superproject and merging from
"mothership" using 'subtree' merge strategy as mentioned allows only for
one direction (well, except for sending patches).

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: How do I manage this setup with git-svn and/or git remotes?

[David Kastrup]

Jakub Narebski <jnareb@gmail.com> writes:

> David Kastrup wrote:
>
>> Linus Torvalds <torvalds@linux-foundation.org> writes:
>> 
>>> On Fri, 17 Aug 2007, David Kastrup wrote:
>>>> 
>>>> Now is there any chance to set up a git structure that will me
>>>> allow to let _git_ perform merges between the standalone dsp
>>>> project and the part that has started off as a copy of it in a
>>>> subdirectory from projects/great, so that I have a merge history in
>>>> my git mirror?
>>>
>>> Yes. That's what git "submodule" support is all about.  You could
>>> create that "dsp" project as its own git project, and then include
>>> it within the bigger project as a submodule. Then, that "dsp" thing
>>> is really a totally independent git project in its own right, with
>>> git support for just "tying" it into the superproject.
>> 
>> But it isn't an independent git project: the superproject has its
>> _own_ copy of dsp, with its _own_ specific commits and fixes that are
>> not supposed to ever end up in the dsp "mothership".  There are
>> sometimes cross merges, but the stuff in the "dsp" subdirectory of
>> "great" is maintained completely together with the branches of
>> "great": tags, branches and all.
>
> Independent git project means independent clone of "dsp" repository,
> perhaps a fork of "dsp" repository with some (superproject) specific
> commits. Which is attached as subritectory of superproject.

Well, the point I was trying to make is that as a subdirectory of the
superproject (in Subversion), the tagging and branching structure is
supposed to follow that of the superproject.  I don't understand
submodules well enough to see how much this is the case.

To be fair: merging will only happen on the trunk of the superproject,
but possibly with different branches of the dsp project.  So this is
likely not going to be much of an issue.

On the other hand, it is unclear what git-svn would think about that
sort of setup.  Since a submodule setup would imply creating
appropriate branches in the dsp module, anyway, I can just import the
superproject dsp as one remote branch into the dsp git repository and
do the merging there.  There is no need to make dsp an actual
submodule of the superproject for that as long as I am working with
git-svn.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum