Reconstructing git-svn metadata after a git clone

Reconstructing git-svn metadata after a git clone

[Ævar Arnfjörð Bjarmason]

I maintain a couple of git-svn exports [1][2] on GitHub. I'd like to
instruct my users on how they should clone those projects and then
commit back to Subversion. But in some cases I can't figure out how to
reconstruct the Subversion metadata.

    1. http://github.com/openstreetmap
    2. http://github.com/mediawiki

The git-svn documentation is lacking here. I'd like to send a patch
for it, but first I have to know what to write. I can add Subversion
metadata to some repositories, but not others.

I can e.g. clone the WebKit repository
(http://trac.webkit.org/wiki/UsingGitWithWebKit#Checkout):

    git clone git://git.webkit.org/WebKit.git WebKit
    cd WebKit
    git svn init -T trunk http://svn.webkit.org/repository/webkit
    git update-ref refs/remotes/trunk origin/master

That works just fine and gives me:

    [...]
    r59025 = 542777d30c645a095d5416a74fdf60b2f3446685
    r59026 = 9afc07910ceabcc065fcd4fcd1c68382955d0d7e
    r59027 = c7c8bfaf28e19eab90f68f53dc7523dd07ef794f
    Done rebuilding
.git/svn/refs/remotes/trunk/.rev_map.268f45cc-cd09-0410-ab3c-d52691b4dbfc

However, when I do that with some other repositories I can't make it
do the right thing:

    # Made with `git svn clone
http://svn.openstreetmap.org/applications/utils/tirex/`
    git clone git://github.com/openstreetmap/tirex.git
    git svn init http://svn.openstreetmap.org/applications/utils/tirex/

So far so good, but this doesn't work:

   git update-ref refs/remotes/trunk origin/master
   # or this
   git update-ref refs/remotes/master origin/master
   # or this
   git update-ref refs/remotes/origin/master origin/master

On git-svn fetch I just get:

    $ git svn fetch
    W: Ignoring error from SVN, path probably does not exist:
(160013): Filesystem has no item:
'/!svn/bc/100/applications/utils/tirex' path not found
    W: Do not be alarmed at the above message git-svn is just
searching aggressively for old history.
    This may take a while on large repositories
    Checked through r14400

Followed by it downloading all the revisions again. I get similar
results with the MediaWiki repository:

    git clone git://github.com/mediawiki/mediawiki-svn.git
    # It was cloned with --stdlayout from this path (well, actually a
mirror, but..)
    git svn init --stdlayout svn+ssh://svn.wikimedia.org/svnroot/mediawiki

The branches look like this:

    $ git branch -l -a | head -n 20
    * trunk
      [...]
      remotes/origin/FundraiserPortal-multi@54613
      remotes/origin/HEAD -> origin/trunk
      remotes/origin/LuceneSearch-ajax
      remotes/origin/MWSearch-2.0@21624
      [...]

Trying to update-ref appropriately:

    for i in $(git branch -l -a|ack '^  remotes/origin/(\S+)$'
--output '$1'); do
        git update-ref "refs/remotes/$i" "origin/$i";
    done

Running a subsequent `git svn fetch` fails:

    Rebuilding .git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
...
    Done rebuilding
.git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
    Rebuilding .git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
...
    Done rebuilding
.git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
    Rebuilding .git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
...
    Done rebuilding
.git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
    Rebuilding .git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
...
    Done rebuilding
.git/svn/refs/remotes/trunk/.rev_map.dd0e9695-b195-4be7-bd10-2dea1a65a6b6
    Index mismatch: fc65f221da86f97a43691fb203aed3da67fb2336 !=
4b825dc642cb6eb9a060e54bf8d69288fbee4904
    rereading d226d034df045bfae1804da6c23f75fc821c4d3f
            D       CommonsBrowser/CommonsBrowser.sln
            D       CommonsBrowser/CommonsBrowser.suo
            D       CommonsBrowser/CommonsBrowser/CommonsBrowser.cspr
            [..tries to delete everything in the repo..]
    W: -empty_dir: trunk/zedler
    a.sql
            D       scavenger/tables_pg.sql
            D       scavenger/
    Last fetched revision of refs/remotes/trunk was r66073, but we are
about to fetch: r1!

However, this evil hack works fine for me:

    # Exactly like before
    git clone git://github.com/mediawiki/mediawiki-svn.git
    cd mediawiki-svn
    git svn init --stdlayout svn+ssh://svn.wikimedia.org/svnroot/mediawiki

    # Download a tarball of the .git/svn content from the original
    # repository:

    cd .git
    wget http://git.anyonecanedit.org/~avar/svn.tar.bz2
    tar xjvf svn.tar.bz2
    cd ..

    # Now git svn fetch works!

    $ git svn fetch
            D       extensions/Storyboard/jquery/jquery.ajaxscroll.min.js
            M       extensions/Storyboard/jquery/jquery.ajaxscroll.js
            M       extensions/Storyboard/README
            M       extensions/Storyboard/tags/Storyboard/storyboard.js
            M       extensions/Storyboard/storyboard.js
            M       extensions/Storyboard/INSTALL
            M       extensions/Storyboard/RELEASE-NOTES
    W: -empty_dir: trunk/extensions/Storyboard/jquery/jquery.ajaxscroll.min.js
    r66071 = f33f81701b06abde87ed11d140cfe3ce4e073bf5 (refs/remotes/trunk)
            M       extensions/Storyboard/Storyboard.i18n.php
            M
extensions/Storyboard/tags/Storysubmission/Storysubmission_body.php
            M       extensions/Storyboard/specials/Story/Story_body.php
            M       extensions/Storyboard/storyboard.js
            M       extensions/Storyboard/Storyboard.php
    r66072 = 954997aef3bbadb4c2a3215e7dd031572a6c3d09 (refs/remotes/trunk)
            M       extensions/SemanticMaps/GeoCoords/SM_GeoCoordsValue.php
            M
extensions/SemanticMaps/GeoCoords/SM_GeoCoordsValueDescription.php
    r66073 = 1902743c893d9107a3fe4e1ce4ed109c4c5ca0de (refs/remotes/trunk)
    $ git svn fetch
    $

So:

 * Am I doing something wrong? If so I can't see what it is.

 * Is there something that works for the general case, i.e. you only
   have to know the original `git svn init` options. If there is I'd
   like to document that & submit a patch.

 * Depending on the above; can git-svn itself be friendlier here?
   Maybe by having a `git svn bootstrap` command. E.g.:

  git clone git://$some_url
  # Does all the work of setting up metadata/refs
  git svn bootstrap --stdlayout $remote_svn_url

Re: Reconstructing git-svn metadata after a git clone

[Dmitrijs Ledkovs]

On 8 May 2010 18:58, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> So:
>
>  * Am I doing something wrong? If so I can't see what it is.
>

No

>  * Is there something that works for the general case, i.e. you only
>   have to know the original `git svn init` options. If there is I'd
>   like to document that & submit a patch.
>

In my repo I have a branch with no anestors which has a config file,
setup.sh & fetch.sh

I instruct to clone repo, checkout "utils" branch, run setup.sh (it
overrides .git/config with config file committed to utils branch and
after it copied config it runs git svn init URL), fetch.sh just runs
git svn fetch =)

I have to do this because I have two svn remotes and both are not
standard layout. The only way for me to represet "same init options"
is by editing .git/config cause it's impossible for me to supply
git-init options on the command line =)

>  * Depending on the above; can git-svn itself be friendlier here?
>   Maybe by having a `git svn bootstrap` command. E.g.:
>

It would be nice to have the git init info propogate with the git
clone. But this won't work. You are cloning *all* branches and
providing a git mirror, where as I want to to git init just my svn
branch or a subset of them. To achieve that I will clone just the
branches I need modify my .git/config and get the result I want.

So imho git-svn is ok here.

>  git clone git://$some_url
>  # Does all the work of setting up metadata/refs
>  git svn bootstrap --stdlayout $remote_svn_url


If you are committing to svn regularly you are better of with bzr-svn
in my opinion. Because launchpad can run automatic imports for you
(webkit is already running btw) and the whole bootstrapping thing is
done the way you are expecting it.

#create repository to store revisions efficiently
$ bzr init-repo .

#on the first ever run it will rebuild meta-data
#subsequent runs just fetch missing revisions
$ bzr branch svn://path.to.any.branch

And you can commit from that =) and bzr can operate on your svn
checkouts. And every single clone done by bzr-svn is identical (unlike
git where everyone has to follow the same git-svn mirror to get same
revision-ids).

You have two options when commiting with bzr-svn. Regular bzr ci will
store bzr merge information in revision properties on svn server or
you can use bzr dpush which is like git svn dcommit.

IMHO bzr-svn is the best when you need to commit back to svn and
painlessly commit parts of the feature branch and merge other bits
later.

Re: Reconstructing git-svn metadata after a git clone

[Ævar Arnfjörð Bjarmason]

On Sat, May 8, 2010 at 18:20, Dmitrijs Ledkovs
<dmitrij.ledkov@ubuntu.com> wrote:
> On 8 May 2010 18:58, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>> So:
>>
>>  * Am I doing something wrong? If so I can't see what it is.
>>
>
> No
>
>>  * Is there something that works for the general case, i.e. you only
>>   have to know the original `git svn init` options. If there is I'd
>>   like to document that & submit a patch.
>>
>
> In my repo I have a branch with no anestors which has a config file,
> setup.sh & fetch.sh
>
> I instruct to clone repo, checkout "utils" branch, run setup.sh (it
> overrides .git/config with config file committed to utils branch and
> after it copied config it runs git svn init URL), fetch.sh just runs
> git svn fetch =)
>
> I have to do this because I have two svn remotes and both are not
> standard layout. The only way for me to represet "same init options"
> is by editing .git/config cause it's impossible for me to supply
> git-init options on the command line =)

If I can't resolve this I'll do something similar. Just provide a
tarball of .git/svn for each repo that is.

>>  * Depending on the above; can git-svn itself be friendlier here?
>>   Maybe by having a `git svn bootstrap` command. E.g.:
>>
>
> It would be nice to have the git init info propogate with the git
> clone. But this won't work. You are cloning *all* branches and
> providing a git mirror, where as I want to to git init just my svn
> branch or a subset of them. To achieve that I will clone just the
> branches I need modify my .git/config and get the result I want.
>
> So imho git-svn is ok here.

I still don't see why it can't work. Maybe I'm just hopelessly naïve
to git-svn internals, but:

   * Every commit message in the Git repo has info on the svn
     branch/svn commit.

   * The Git repo has a list of branches that map to SVN branches.

   * If I supply the same `git svn init` options that made the repo,
     it should be able to bootstrap just using the above.

Why isn't that the case? What info is in .git/svn that can't be
inferred from the above?

>>  git clone git://$some_url
>>  # Does all the work of setting up metadata/refs
>>  git svn bootstrap --stdlayout $remote_svn_url
>
> If you are committing to svn regularly you are better of with bzr-svn
> in my opinion. Because launchpad can run automatic imports for you
> (webkit is already running btw) and the whole bootstrapping thing is
> done the way you are expecting it.
>
> #create repository to store revisions efficiently
> $ bzr init-repo .
>
> #on the first ever run it will rebuild meta-data
> #subsequent runs just fetch missing revisions
> $ bzr branch svn://path.to.any.branch
>
> And you can commit from that =) and bzr can operate on your svn
> checkouts. And every single clone done by bzr-svn is identical (unlike
> git where everyone has to follow the same git-svn mirror to get same
> revision-ids).
>
> You have two options when commiting with bzr-svn. Regular bzr ci will
> store bzr merge information in revision properties on svn server or
> you can use bzr dpush which is like git svn dcommit.
>
> IMHO bzr-svn is the best when you need to commit back to svn and
> painlessly commit parts of the feature branch and merge other bits
> later.

That's informative. But from having used Bazaar a bit my experience
with it was that I might as well be using Subversion.

Re: Reconstructing git-svn metadata after a git clone

[Dmitrijs Ledkovs]

On 10 May 2010 22:43, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> On Sat, May 8, 2010 at 18:20, Dmitrijs Ledkovs
> <dmitrij.ledkov@ubuntu.com> wrote:
>> On 8 May 2010 18:58, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>>> So:
>>>
>>>  * Am I doing something wrong? If so I can't see what it is.
>>>
>>
>> No
>>
>>>  * Is there something that works for the general case, i.e. you only
>>>   have to know the original `git svn init` options. If there is I'd
>>>   like to document that & submit a patch.
>>>
>>
>> In my repo I have a branch with no anestors which has a config file,
>> setup.sh & fetch.sh
>>
>> I instruct to clone repo, checkout "utils" branch, run setup.sh (it
>> overrides .git/config with config file committed to utils branch and
>> after it copied config it runs git svn init URL), fetch.sh just runs
>> git svn fetch =)
>>
>> I have to do this because I have two svn remotes and both are not
>> standard layout. The only way for me to represet "same init options"
>> is by editing .git/config cause it's impossible for me to supply
>> git-init options on the command line =)
>
> If I can't resolve this I'll do something similar. Just provide a
> tarball of .git/svn for each repo that is.
>

Always an option.

>>>  * Depending on the above; can git-svn itself be friendlier here?
>>>   Maybe by having a `git svn bootstrap` command. E.g.:
>>>
>>
>> It would be nice to have the git init info propogate with the git
>> clone. But this won't work. You are cloning *all* branches and
>> providing a git mirror, where as I want to to git init just my svn
>> branch or a subset of them. To achieve that I will clone just the
>> branches I need modify my .git/config and get the result I want.
>>
>> So imho git-svn is ok here.
>
> I still don't see why it can't work. Maybe I'm just hopelessly naïve
> to git-svn internals, but:
>
>   * Every commit message in the Git repo has info on the svn
>     branch/svn commit.
>
>   * The Git repo has a list of branches that map to SVN branches.
>

Your local one does. But not the cloned one.

>   * If I supply the same `git svn init` options that made the repo,
>     it should be able to bootstrap just using the above.
>

Yes, but it takes a while..... especially if the svn repo is huge.

> Why isn't that the case? What info is in .git/svn that can't be
> inferred from the above?
>

Is there a git versions mismatch between machine where the first
import was made and the machine where you are testing bootstrapping by
any chance?

on my machine I sometimes see "upgrading metadata" when I'm updating
some of my acient git-svn clones.

>>>  git clone git://$some_url
>>>  # Does all the work of setting up metadata/refs
>>>  git svn bootstrap --stdlayout $remote_svn_url
>>
>> If you are committing to svn regularly you are better of with bzr-svn
>> in my opinion. Because launchpad can run automatic imports for you
>> (webkit is already running btw) and the whole bootstrapping thing is
>> done the way you are expecting it.
>>
>> #create repository to store revisions efficiently
>> $ bzr init-repo .
>>
>> #on the first ever run it will rebuild meta-data
>> #subsequent runs just fetch missing revisions
>> $ bzr branch svn://path.to.any.branch
>>
>> And you can commit from that =) and bzr can operate on your svn
>> checkouts. And every single clone done by bzr-svn is identical (unlike
>> git where everyone has to follow the same git-svn mirror to get same
>> revision-ids).
>>
>> You have two options when commiting with bzr-svn. Regular bzr ci will
>> store bzr merge information in revision properties on svn server or
>> you can use bzr dpush which is like git svn dcommit.
>>
>> IMHO bzr-svn is the best when you need to commit back to svn and
>> painlessly commit parts of the feature branch and merge other bits
>> later.
>
> That's informative. But from having used Bazaar a bit my experience
> with it was that I might as well be using Subversion.
>

Re: Reconstructing git-svn metadata after a git clone

[Ævar Arnfjörð Bjarmason]

On Tue, May 11, 2010 at 04:27, Dmitrijs Ledkovs
<dmitrij.ledkov@ubuntu.com> wrote:
> On 10 May 2010 22:43, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>> Why isn't that the case? What info is in .git/svn that can't be
>> inferred from the above?
>>
>
> Is there a git versions mismatch between machine where the first
> import was made and the machine where you are testing bootstrapping by
> any chance?
>
> on my machine I sometimes see "upgrading metadata" when I'm updating
> some of my acient git-svn clones.

No, they were both running 1.7.1 from Debian.