svn:externals using git submodules

svn:externals using git submodules

[Andy Parkins]

Hello,

I've done this by hand as a proof of concept I suspect it would need loads of 
work in git-svn to do it properly.  However, I thought I'd mention as part of 
my "success with submodules" reports.

ffmpeg is managed with svn; I like to track its development with git-svn.  
Works wonderfully except for one problem: they've made use of svn:externals 
for one component, libswscale.  Previously I just regularly updated the 
libswscale subdirectory by checking out the latest copy (which is all that 
subversion does) and committing it to my own branch off upstream.

With submodule support in git, it makes it possible to do a much better job.  
What I did was have two svn-remote sections in the config:

[svn-remote "ffmpeg"]
    url = svn://svn.mplayerhq.hu/ffmpeg
    fetch = trunk:refs/remotes/ffmpeg-svn

[svn-remote "libswscale"]
    url = svn://svn.mplayerhq.hu/mplayer
    fetch = trunk/libswscale:refs/remotes/libswscale-svn

After running git-svn fetch; there are two independent branches in my 
repository:

  -- * -- * -- * -- * -- * (ffmpeg-svn)
  ---- * ----- * ------- * (libswscale-svn)  

Now, we fork from ffmpeg-svn and libswscale-svn to make non-tracking branches 
that can be committed to:

 $ git checkout -b master-ffm ffmpeg-svn
 $ git branch master-sws libswscale-svn

Next, we create a shared clone of the repository as a subdirectory in that 
repository.

 $ git clone -s . libswscale

Now we want that clone to be even more strongly linked to the parent - to the 
extent that they share the same refs, etc:

 $ cd libswscale
 $ rm -rf .git/refs .git/logs .git/info description config
 $ ln -s ../../.git/refs .git/refs
 $ ln -s ../../.git/logs .git/logs
 $ ln -s ../../.git/info .git/info
 $ ln -s ../../.git/config .git/config
 $ ln -s ../../.git/description .git/description

Only HEAD and index are independent.  Next we switch from the ffmpeg branch to 
the libswscale branch in this subdirectory:

 $ git checkout master-sws

Now, we make the subdirectory a submodule in the parent:

 $ cd ..
 $ git add libswscale
 $ git commit -m "libswscale is now a submodule"

How dangerous is this?  I've made the repository it's own submodule and it 
shares the same refs, info and logs.  LIVING ON THE EDGE MAN!

You have to run two git-svn commands to sync with upstream:

 $ git-svn fetch ffmpeg
 $ git-svn fetch libswscale

Then of course you would merge

 $ git merge ffmpeg-svn
 $ cd libswscale; git merge libswscale-svn; cd ..
 $ git commit -m "Sync with upstream"

Personally I think that's pretty cool, this is significantly better than 
svn:externals because the particular revision of libswscale in use is 
recorded.  Seriously - someone show me another VCS that can do that - I think 
git has actual magic powers :-)

I dare say that git-svn could do much better because it could reconstruct the 
submodule history based on the repository dates and create the link in the 
tracking branch rather than having to do it manually at the end as I've done 
here.  That would mean that the recorded submodule was right for all 
history - again, not the case for svn:externals, if you check out a previous 
version the external remains current.



Andy
-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: svn:externals using git submodules

[Chris Shoemaker]

On Tue, May 01, 2007 at 11:21:14AM +0100, Andy Parkins wrote:
> Hello,
> 
> I've done this by hand as a proof of concept I suspect it would need loads of 
> work in git-svn to do it properly.  However, I thought I'd mention as part of 
> my "success with submodules" reports.

That's truly interesting, Andy.  Thanks for the encouraging report.
Please do keep us informed of anything more you conclude about this
approach.  I'm sure some of the experts around here can respond with
the pros and cons of this technique.

For my part, I wonder if it can be simplified somehow; and I suspect
it doesn't work well with svn:externals that specify a particular
revision.

-chris

> ffmpeg is managed with svn; I like to track its development with git-svn.  
> Works wonderfully except for one problem: they've made use of svn:externals 
> for one component, libswscale.  Previously I just regularly updated the 
> libswscale subdirectory by checking out the latest copy (which is all that 
> subversion does) and committing it to my own branch off upstream.
> 
> With submodule support in git, it makes it possible to do a much better job.  
> What I did was have two svn-remote sections in the config:
> 
> [svn-remote "ffmpeg"]
>     url = svn://svn.mplayerhq.hu/ffmpeg
>     fetch = trunk:refs/remotes/ffmpeg-svn
> 
> [svn-remote "libswscale"]
>     url = svn://svn.mplayerhq.hu/mplayer
>     fetch = trunk/libswscale:refs/remotes/libswscale-svn
> 
> After running git-svn fetch; there are two independent branches in my 
> repository:
> 
>   -- * -- * -- * -- * -- * (ffmpeg-svn)
>   ---- * ----- * ------- * (libswscale-svn)  
> 
> Now, we fork from ffmpeg-svn and libswscale-svn to make non-tracking branches 
> that can be committed to:
> 
>  $ git checkout -b master-ffm ffmpeg-svn
>  $ git branch master-sws libswscale-svn
> 
> Next, we create a shared clone of the repository as a subdirectory in that 
> repository.
> 
>  $ git clone -s . libswscale
> 
> Now we want that clone to be even more strongly linked to the parent - to the 
> extent that they share the same refs, etc:
> 
>  $ cd libswscale
>  $ rm -rf .git/refs .git/logs .git/info description config
>  $ ln -s ../../.git/refs .git/refs
>  $ ln -s ../../.git/logs .git/logs
>  $ ln -s ../../.git/info .git/info
>  $ ln -s ../../.git/config .git/config
>  $ ln -s ../../.git/description .git/description
> 
> Only HEAD and index are independent.  Next we switch from the ffmpeg branch to 
> the libswscale branch in this subdirectory:
> 
>  $ git checkout master-sws
> 
> Now, we make the subdirectory a submodule in the parent:
> 
>  $ cd ..
>  $ git add libswscale
>  $ git commit -m "libswscale is now a submodule"
> 
> How dangerous is this?  I've made the repository it's own submodule and it 
> shares the same refs, info and logs.  LIVING ON THE EDGE MAN!
> 
> You have to run two git-svn commands to sync with upstream:
> 
>  $ git-svn fetch ffmpeg
>  $ git-svn fetch libswscale
> 
> Then of course you would merge
> 
>  $ git merge ffmpeg-svn
>  $ cd libswscale; git merge libswscale-svn; cd ..
>  $ git commit -m "Sync with upstream"
> 
> Personally I think that's pretty cool, this is significantly better than 
> svn:externals because the particular revision of libswscale in use is 
> recorded.  Seriously - someone show me another VCS that can do that - I think 
> git has actual magic powers :-)
> 
> I dare say that git-svn could do much better because it could reconstruct the 
> submodule history based on the repository dates and create the link in the 
> tracking branch rather than having to do it manually at the end as I've done 
> here.  That would mean that the recorded submodule was right for all 
> history - again, not the case for svn:externals, if you check out a previous 
> version the external remains current.
> 
> 
> 
> Andy
> -- 
> Dr Andy Parkins, M Eng (hons), MIET
> andyparkins@gmail.com
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: svn:externals using git submodules

[Shawn O. Pearce]

Chris Shoemaker <c.shoemaker@cox.net> wrote:
> On Tue, May 01, 2007 at 11:21:14AM +0100, Andy Parkins wrote:
> > Hello,
> > 
> > I've done this by hand as a proof of concept I suspect it would need loads of 
> > work in git-svn to do it properly.  However, I thought I'd mention as part of 
> > my "success with submodules" reports.
> 
> For my part, I wonder if it can be simplified somehow; and I suspect
> it doesn't work well with svn:externals that specify a particular
> revision.

Actually that is an interesting point that Chris makes.  Isn't the
svn:externals property revision controlled on the parent directory?
So each change to it is actually recorded in the revision history
of the parent project.  And if every svn:externals URL included the
exact version of the other project to include, aren't svn:externals
then more-or-less like the subproject link support, except they
also include the URL?

-- 
Shawn.

Re: svn:externals using git submodules

[Chris Shoemaker]

On Tue, May 01, 2007 at 11:22:28AM -0400, Shawn O. Pearce wrote:
> Chris Shoemaker <c.shoemaker@cox.net> wrote:
> > On Tue, May 01, 2007 at 11:21:14AM +0100, Andy Parkins wrote:
> > > Hello,
> > > 
> > > I've done this by hand as a proof of concept I suspect it would need loads of 
> > > work in git-svn to do it properly.  However, I thought I'd mention as part of 
> > > my "success with submodules" reports.
> > 
> > For my part, I wonder if it can be simplified somehow; and I suspect
> > it doesn't work well with svn:externals that specify a particular
> > revision.
> 
> Actually that is an interesting point that Chris makes.  Isn't the
> svn:externals property revision controlled on the parent directory?
> So each change to it is actually recorded in the revision history
> of the parent project.  

Yes and yes.

> And if every svn:externals URL included the
> exact version of the other project to include, aren't svn:externals
> then more-or-less like the subproject link support, except they
> also include the URL?

Just to clarify, my point was just that Andy's setup seems to assume
that the externals don't specify a revision.  If they do, maybe
git-svn can map the externals into subprojects.  Is this what
you're thinking?

-chris

> 
> -- 
> Shawn.

Re: svn:externals using git submodules

[Shawn O. Pearce]

Chris Shoemaker <c.shoemaker@cox.net> wrote:
> On Tue, May 01, 2007 at 11:22:28AM -0400, Shawn O. Pearce wrote:
> > And if every svn:externals URL included the
> > exact version of the other project to include, aren't svn:externals
> > then more-or-less like the subproject link support, except they
> > also include the URL?
> 
> Just to clarify, my point was just that Andy's setup seems to assume
> that the externals don't specify a revision.  If they do, maybe
> git-svn can map the externals into subprojects.  Is this what
> you're thinking?

Yes.  ;-)

-- 
Shawn.

Re: svn:externals using git submodules

[Andy Parkins]

On Tuesday 2007, May 01, Chris Shoemaker wrote:

> > Actually that is an interesting point that Chris makes.  Isn't the
> > svn:externals property revision controlled on the parent directory?
> > So each change to it is actually recorded in the revision history
> > of the parent project.
>
> Yes and yes.

Yes and no.  Think of svn:externals as a file in the parent repository; 
it contains

 directory-name URL

Now, changes to that file _are_ tracked, in that if I changed the URL 
that change would be recorded in the parent repository.  However, 
nowhere is the revision of the external recorded.  Subversion always 
fetches the latest revision at that URL.

> > And if every svn:externals URL included the
> > exact version of the other project to include, aren't svn:externals
> > then more-or-less like the subproject link support, except they
> > also include the URL?
>
> Just to clarify, my point was just that Andy's setup seems to assume
> that the externals don't specify a revision.  If they do, maybe

They don't.  If they did, they'd be just as useful as git's submodules.

> git-svn can map the externals into subprojects.  Is this what
> you're thinking?

Well, I'm thinking that that information /can/ be reconstructed from the 
revision date information - kind of - the problem is that there is no 
way to know when the parent updated the module.   svn:externals really 
is just a quick way of doing
 $ cd submodule
 $ svn update
That's it.  That's all you get.  We could guess that when the parent 
module was at date YYYY-MM-DD, that the submodule would be at that same 
date - but who knows?



Andy
-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: svn:externals using git submodules

[Andy Parkins]

On Tuesday 2007, May 01, Andy Parkins wrote:

> Now, changes to that file _are_ tracked, in that if I changed the URL
> that change would be recorded in the parent repository.  However,
> nowhere is the revision of the external recorded.  Subversion always
> fetches the latest revision at that URL.

I meant to add as well that this is absolutely NOT the thing that you 
want to be tracked.  There are any number of times while using 
externals that I reorganised a directory only to have to change the 
svn:externals in the parent.  That change is then tracked, so if you 
check out an earlier version not only do you not get a particular 
revision you also don't get the right URL, so subverion doesn't even 
fetch the current version.  Gah!


Andy

-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: svn:externals using git submodules

[Chris Shoemaker]

On Tue, May 01, 2007 at 07:36:11PM +0100, Andy Parkins wrote:
> On Tuesday 2007, May 01, Chris Shoemaker wrote:
> 
> > > Actually that is an interesting point that Chris makes.  Isn't the
> > > svn:externals property revision controlled on the parent directory?
> > > So each change to it is actually recorded in the revision history
> > > of the parent project.
> >
> > Yes and yes.
> 
> Yes and no.  Think of svn:externals as a file in the parent repository; 
> it contains
> 
>  directory-name URL
> 
> Now, changes to that file _are_ tracked, in that if I changed the URL 
> that change would be recorded in the parent repository.  However, 
> nowhere is the revision of the external recorded.  Subversion always 
> fetches the latest revision at that URL.

That's only true when the revision is not specified in the external.
The repo you track may not do that, but it's not uncommon to do so.
And, as I think you're pointing out, it's the only way to get any sort
of reliable information about the relationship between the parent and
the external.

I think it would probably be undesirable for git-svn to attempt to
convert "floating" externals into well-versioned submodules, since
they're not even well-versioned in the svn repo.  However, handling
the "locked-down" externals is quite another thing.

> 
> > > And if every svn:externals URL included the
> > > exact version of the other project to include, aren't svn:externals
> > > then more-or-less like the subproject link support, except they
> > > also include the URL?
> >
> > Just to clarify, my point was just that Andy's setup seems to assume
> > that the externals don't specify a revision.  If they do, maybe
> 
> They don't.  If they did, they'd be just as useful as git's submodules.
>
> > git-svn can map the externals into subprojects.  Is this what
> > you're thinking?
> 
> Well, I'm thinking that that information /can/ be reconstructed from the 
> revision date information - kind of - the problem is that there is no 
> way to know when the parent updated the module.   svn:externals really 
> is just a quick way of doing
>  $ cd submodule
>  $ svn update
> That's it.  That's all you get.  We could guess that when the parent 
> module was at date YYYY-MM-DD, that the submodule would be at that same 
> date - but who knows?

svn users who want the externals to meaningfully define the version
relationship between the parent and the project already have to use
externals that specify a revision.

-chris

Re: svn:externals using git submodules

[Andy Parkins]

On Tuesday 2007, May 01, Chris Shoemaker wrote:

> That's only true when the revision is not specified in the external.
> The repo you track may not do that, but it's not uncommon to do so.

It's been a while since I used subversion, and even longer since I used 
externals - is that a new feature?  I used subversion since before 
version 1.0, so I often missed new features when they arrived. 

> And, as I think you're pointing out, it's the only way to get any
> sort of reliable information about the relationship between the
> parent and the external.

Does subversion automatically update that fixed attachment when you 
update the submodule?  I would have found that quite useful back then.

> I think it would probably be undesirable for git-svn to attempt to
> convert "floating" externals into well-versioned submodules, since
> they're not even well-versioned in the svn repo.  However, handling
> the "locked-down" externals is quite another thing.

Absolutely.  If the information is available, then git is certainly 
capable of recording it.  It sounds like subversion has a facility I 
didn't know exist, so I've been bad mouthing it more than I should.  Oh 
well :-)



Andy

-- 
Dr Andy Parkins, M Eng (hons), MIET
andyparkins@gmail.com

Re: svn:externals using git submodules

[Chris Shoemaker]

On Tue, May 01, 2007 at 08:48:01PM +0100, Andy Parkins wrote:
> On Tuesday 2007, May 01, Chris Shoemaker wrote:
> 
> > That's only true when the revision is not specified in the external.
> > The repo you track may not do that, but it's not uncommon to do so.
> 
> It's been a while since I used subversion, and even longer since I used 
> externals - is that a new feature?  

I don't know, but I would guess that it's no newer than externals in
general, as it's not a particularly special case.

> I used subversion since before 
> version 1.0, so I often missed new features when they arrived. 
> 
> > And, as I think you're pointing out, it's the only way to get any
> > sort of reliable information about the relationship between the
> > parent and the external.
> 
> Does subversion automatically update that fixed attachment when you 
> update the submodule?  I would have found that quite useful back then.

No, you have to manage the revision in the svn:external property
manually.

> > I think it would probably be undesirable for git-svn to attempt to
> > convert "floating" externals into well-versioned submodules, since
> > they're not even well-versioned in the svn repo.  However, handling
> > the "locked-down" externals is quite another thing.
> 
> Absolutely.  If the information is available, then git is certainly 
> capable of recording it.  It sounds like subversion has a facility I 
> didn't know exist, so I've been bad mouthing it more than I should.  Oh 
> well :-)
> 

Making git-svn handle svn:externals with specified revisions would be
_quite_ useful.  There's a special-case of this that I use personally:
svn:externals that point to other paths (and other revisions) of the
parent repo.

I'm curious if people think that teaching git-svn to handle this
special case is more or less difficult than handling the general case.

-chris

Re: svn:externals using git submodules

[Linus Torvalds]

On Tue, 1 May 2007, Chris Shoemaker wrote:
> 
> Making git-svn handle svn:externals with specified revisions would be
> _quite_ useful.  There's a special-case of this that I use personally:
> svn:externals that point to other paths (and other revisions) of the
> parent repo.

Side note: even _without_ a specified revision, I think it's quite sane to 
have the rule that a submodule hash of all zeroes is "unversioned".

Such a submodule is still _useful_: while the tree itself contains no 
information (and it SHOULD NOT do so, since the actual location of the 
external module may not be globally stable or visible!), it would 
basically act like subversion externals together with the ".gitmodules" 
file that contains that information.

So while the git submodule thing was designed to specify specific 
revisions, there's nothing that really technically _requires_ it. The 
exact SHA1 details in the submodule link are going to be up to the 
higher-level user anyway.

(Of course, if you actually have a "all zeroes" gitlink entry, and then 
have a checked-out git tree at that entry, "git status" and "git diff" 
would show it as needing update. I think that's _correct_, but if we want 
to shut it up for the special case of all-zero SHA1, we trivially could).

But while I'm encouraged that the whole gitlink thing seems to be working 
for Andy, and some others are playing with it too, I'm also a bit 
discouraged by the fact that there hasn't been any noise or work on the 
porcelain side. I was obviously optimistic and hoping we'd see support in 
checkout/diff, but I haven't heard anybody talk about actually 
implementing .gitmodules and the porcelain support that uses them..

			Linus

Re: svn:externals using git submodules

[Junio C Hamano]

Linus Torvalds <torvalds@linux-foundation.org> writes:

> On Tue, 1 May 2007, Chris Shoemaker wrote:
>> 
>> Making git-svn handle svn:externals with specified revisions would be
>> _quite_ useful.  There's a special-case of this that I use personally:
>> svn:externals that point to other paths (and other revisions) of the
>> parent repo.
>
> Side note: even _without_ a specified revision, I think it's quite sane to 
> have the rule that a submodule hash of all zeroes is "unversioned".

Yup.

> But while I'm encouraged that the whole gitlink thing seems to be working 
> for Andy, and some others are playing with it too, I'm also a bit 
> discouraged by the fact that there hasn't been any noise or work on the 
> porcelain side. I was obviously optimistic and hoping we'd see support in 
> checkout/diff, but I haven't heard anybody talk about actually 
> implementing .gitmodules and the porcelain support that uses them..

The thing is, almost all the core git people happen to be busy
at the same time at this moment.  Johannes has just moved, Shawn
and I are deep in day-jobs to the neck, ...

Don't worry, it eventually will come.  

Re: svn:externals using git submodules

[Shawn O. Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Linus Torvalds <torvalds@linux-foundation.org> writes:
> > But while I'm encouraged that the whole gitlink thing seems to be working 
> > for Andy, and some others are playing with it too, I'm also a bit 
> > discouraged by the fact that there hasn't been any noise or work on the 
> > porcelain side. I was obviously optimistic and hoping we'd see support in 
> > checkout/diff, but I haven't heard anybody talk about actually 
> > implementing .gitmodules and the porcelain support that uses them..
> 
> The thing is, almost all the core git people happen to be busy
> at the same time at this moment.  Johannes has just moved, Shawn
> and I are deep in day-jobs to the neck, ...

Heh, very true.  My short-term focus (this week) is git-gui,
then pack v4, and probably while working on pack v4 start at least
playing with Linus' gitlink thing and prototyping porcleain over it.
The gitlink work interests me, and I want to spend time on it,
but as Junio said, I'm overbooked...

-- 
Shawn.

Re: svn:externals using git submodules

[Michel Jouvin]

Andy Parkins <andyparkins <at> gmail.com> writes:

> 
> Hello,
> 
> I've done this by hand as a proof of concept I suspect it would need loads of 
> work in git-svn to do it properly.  However, I thought I'd mention as part of 
> my "success with submodules" reports.
> 
> ... 
> Now we want that clone to be even more strongly linked to the parent - to the 
> extent that they share the same refs, etc:
> 
>  $ cd libswscale
>  $ rm -rf .git/refs .git/logs .git/info description config
>  $ ln -s ../../.git/refs .git/refs
>  $ ln -s ../../.git/logs .git/logs
>  $ ln -s ../../.git/info .git/info
>  $ ln -s ../../.git/config .git/config
>  $ ln -s ../../.git/description .git/description
> 
> ...
> Andy

I don't know if it's a good idea to follow-up on such an old entry... but I 
used your trick, Andy, and it works pretty well. With one exception: I am using 
Git on Windows and there is no symlinks on this OS. That means that you need to 
basically "recreate" the symlinks everytime you want to update the submodules. 
Not very handy but acceptable as they are not updated very often.

I have to check the list if there has been some new ways to achieve this. But 
I'm clearly in the case of svn:externals not tighten to a particular revision, 
which seems the difficult case for Git.

Cheers,

Michel