How to clone git repository with git-svn meta-data included?

How to clone git repository with git-svn meta-data included?

[Grzegorz Kossakowski]

Hello,

Some folks at Apache are experimenting with Git and we are currently seeking for the git-svn
integration that fits our needs and infrastructure.

After some evaluation we decided that our setup could be described using following points:
  a) our svn repository remains our main, official server where every committer is obligated to push
their changes to at some time.
  b) we set up clone of svn repository using git-svn. One of our members, Jukka Zitting, maintains
such a service here[1]. Such repositories should be usable both for our committers (that have rights
to push to svn) and our contributors that want to contribute random patches
  c) we want carefully track who committed/contributed what

Basically, a) implies b) and point b) looks little bit problematic right now.
Jukka has set up his hosting using method described in his e-mail[2] which basically makes use of
git svn. The major problem is that if one clones Jukka's repository then git svn information is not
being cloned so committers have no means to push their changes to main, svn server.

I've tried to play a little bit around with this issue and I tried to copy information from .git
directory found on Jukka's server. This made me able to push my changes but git svn insisted on
rebasing my repository using commits found in svn which is wrong. Basically we want such a setup
that uses git repository (Jukka's clone) for pulling changes and local git svn for pushing changes.
Git svn should never try to rebase local repository because this will lead to two different trees on
two different machines so we won't be able to exchange and merge changesets.

Is it possible with Git right now?


Another point (c) which seems to be brought a couple of times but never a definitive answer has been
given, AFAIK. Let's imagine we have committer C and two contributors A and B.

A and B start to work on some feature and C agreed to help A and B and once their work is finished
to merge their changes into his repository and eventually push them to main, svn repository. Now A
and B work on implementation and from time to time their merge changes from each other. Once they
are finished A asks C to merge their work into C's repository. Everything is fine provided we can
trust both A and B.

What if A was not fair and has rewritten a few commits coming from B so they contain malicious code?
How we can detect something like that and how C be sure that what he merges is really work
attributed by correct names?

Thanks for your answers.

[1] http://jukka.zitting.name/git/
[2] http://markmail.org/message/fzzy7nepk7olx5fl


-- 
Best regards,
Grzegorz Kossakowski

Re: How to clone git repository with git-svn meta-data included?

[Jacob Helwig]

On Sat, Dec 6, 2008 at 04:15, Grzegorz Kossakowski <grek@tuffmail.com> wrote:
> Hello,
>
> Some folks at Apache are experimenting with Git and we are currently seeking for the git-svn
> integration that fits our needs and infrastructure.
>
> After some evaluation we decided that our setup could be described using following points:
>  a) our svn repository remains our main, official server where every committer is obligated to push
> their changes to at some time.
>  b) we set up clone of svn repository using git-svn. One of our members, Jukka Zitting, maintains
> such a service here[1]. Such repositories should be usable both for our committers (that have rights
> to push to svn) and our contributors that want to contribute random patches
>  c) we want carefully track who committed/contributed what
>
> Basically, a) implies b) and point b) looks little bit problematic right now.
> Jukka has set up his hosting using method described in his e-mail[2] which basically makes use of
> git svn. The major problem is that if one clones Jukka's repository then git svn information is not
> being cloned so committers have no means to push their changes to main, svn server.
>
> I've tried to play a little bit around with this issue and I tried to copy information from .git
> directory found on Jukka's server. This made me able to push my changes but git svn insisted on
> rebasing my repository using commits found in svn which is wrong. Basically we want such a setup
> that uses git repository (Jukka's clone) for pulling changes and local git svn for pushing changes.
> Git svn should never try to rebase local repository because this will lead to two different trees on
> two different machines so we won't be able to exchange and merge changesets.
>
> Is it possible with Git right now?
>
>
> Another point (c) which seems to be brought a couple of times but never a definitive answer has been
> given, AFAIK. Let's imagine we have committer C and two contributors A and B.
>
> A and B start to work on some feature and C agreed to help A and B and once their work is finished
> to merge their changes into his repository and eventually push them to main, svn repository. Now A
> and B work on implementation and from time to time their merge changes from each other. Once they
> are finished A asks C to merge their work into C's repository. Everything is fine provided we can
> trust both A and B.
>
> What if A was not fair and has rewritten a few commits coming from B so they contain malicious code?
> How we can detect something like that and how C be sure that what he merges is really work
> attributed by correct names?
>
> Thanks for your answers.
>
> [1] http://jukka.zitting.name/git/
> [2] http://markmail.org/message/fzzy7nepk7olx5fl
>
>
> --
> Best regards,
> Grzegorz Kossakowski
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Grzegorz,

I use git-svn quite a bit at $work, but I haven't seen a way to clone
a git repo, and have it Just Work(TM) with git-svn in the new clone,
unfortunately.

I have been able to use clone to significantly cut down on the setup
time for working with git-svn, however.  I was following the
directions at http://subtlegradient.com/articles/2008/04/22/cloning-a-git-svn-clone
 (Clone the git-svn clone, then re-do the git svn init, and fetch to
re-sync.)

Not sure how much this will actually help you with the first part of
your problem.

-Jacob

Re: How to clone git repository with git-svn meta-data included?

[Nick Andrew]

On Sun, Dec 07, 2008 at 12:43:32AM -0800, Jacob Helwig wrote:
> I use git-svn quite a bit at $work, but I haven't seen a way to clone
> a git repo, and have it Just Work(TM) with git-svn in the new clone,
> unfortunately.

At $work I nightly publish a .tar.gz file of a pristine git repo
synced from svn, with the metadata. So if a developer wants to start
using git they untar the file into their directory and start working,
and after that they have to do "git svn fetch" themselves.

Nick.

Re: How to clone git repository with git-svn meta-data included?

[Peter Harris]

On Sat, Dec 6, 2008 at 7:15 AM, Grzegorz Kossakowski wrote:
>
> Some folks at Apache are experimenting with Git and we are currently seeking for the git-svn
> integration that fits our needs and infrastructure.
>
> After some evaluation we decided that our setup could be described using following points:
>  a) our svn repository remains our main, official server where every committer is obligated to push
> their changes to at some time.
>  b) we set up clone of svn repository using git-svn. One of our members, Jukka Zitting, maintains
> such a service here[1]. Such repositories should be usable both for our committers (that have rights
> to push to svn) and our contributors that want to contribute random patches
>  c) we want carefully track who committed/contributed what
>
> Basically, a) implies b) and point b) looks little bit problematic right now.
> Jukka has set up his hosting using method described in his e-mail[2] which basically makes use of
> git svn. The major problem is that if one clones Jukka's repository then git svn information is not
> being cloned so committers have no means to push their changes to main, svn server.

Make sure you don't use the --no-metadata flag when setting up
git-svn. This will embed the metadata into commit messages, so git-svn
can rebuild it from scratch whenever it needs to. (You probably also
want git 1.6.1rc for incremental rebuild support). This also has the
advantage that you can see the svn revision number when looking at a
commit message.

> I've tried to play a little bit around with this issue and I tried to copy information from .git
> directory found on Jukka's server. This made me able to push my changes but git svn insisted on
> rebasing my repository using commits found in svn which is wrong. Basically we want such a setup
> that uses git repository (Jukka's clone) for pulling changes and local git svn for pushing changes.
> Git svn should never try to rebase local repository because this will lead to two different trees on
> two different machines so we won't be able to exchange and merge changesets.

svn doesn't really know what a merge is (not even 1.5). You MUST
rebase in order to commit to svn. This is a limitation of svn, not
git.

In terms of re-pulling from the git-svn mirror, git-svn will create
the same commits (with the same sha1s) from svn every time, so there
will be no conflicts there.

> Is it possible with Git right now?

Yes, but it might not be possible with svn, depending on which part of
the above "it" is.

> What if A was not fair and has rewritten a few commits coming from B so they contain malicious code?
> How we can detect something like that and how C be sure that what he merges is really work
> attributed by correct names?

If C doesn't trust A, C should not pull from A. C should pull only
from (trusted) B. Presumably B knows who (of A and B) did which work,
and B's repository can be trusted?

If neither of A or B can be trusted, then you have problems that a
computer cannot solve for you.

You could maybe use signed tags ("git help tag") - each contributor
could sign a certain tree state, and if you see commits attributed to
the other contributor after their last tag, you know something is
fishy. But that might be more effort than either you or your
contributors want to go through. And while it might help with
attribution problems, it still doesn't help with all the other
problems you might have with untrusted contributors.

Peter Harris

Re: How to clone git repository with git-svn meta-data included?

[Grzegorz Kossakowski]

Peter Harris pisze:
> Make sure you don't use the --no-metadata flag when setting up
> git-svn. This will embed the metadata into commit messages, so git-svn
> can rebuild it from scratch whenever it needs to. (You probably also
> want git 1.6.1rc for incremental rebuild support). This also has the
> advantage that you can see the svn revision number when looking at a
> commit message.

Not sure what you exactly mean here. Do you mean that if metadata is included in commit messages
then there is an easy way to initialize git-svn after cloning the repo?

By easy I mean:
a) it does not require to much of interactive actions to be performed
b) it does not pull too much from svn server

Point b) is important because we usually have quite large repositories.

Also, could you point me to a place where this rebuild support is described? I would like to know
what our committer has to do after cloning from Jukka's server.

> svn doesn't really know what a merge is (not even 1.5). You MUST
> rebase in order to commit to svn. This is a limitation of svn, not
> git.

Yep, I'm aware of the fact that history has to be flattened but I was more worried about the point
you address below.

> In terms of re-pulling from the git-svn mirror, git-svn will create
> the same commits (with the same sha1s) from svn every time, so there
> will be no conflicts there.

Just to make sure: so if one person pulls from git-svn mirror and another one pulls using git svn
rebase they result in the same tree right?

>> Is it possible with Git right now?
> 
> Yes, but it might not be possible with svn, depending on which part of
> the above "it" is.

It referred mostly to cloning from git svn mirror and then being able to use git svn dcommit to push
changes back to svn. Since git svn data is not being cloned the question is how to recreate it.

>> What if A was not fair and has rewritten a few commits coming from B so they contain malicious code?
>> How we can detect something like that and how C be sure that what he merges is really work
>> attributed by correct names?
> 
> If C doesn't trust A, C should not pull from A. C should pull only
> from (trusted) B. Presumably B knows who (of A and B) did which work,
> and B's repository can be trusted?
> 
> If neither of A or B can be trusted, then you have problems that a
> computer cannot solve for you.

Yep, I was having in mind the case when both A and B are untrusted. I don't want my computer to
check if something coming from A or B is safe or not I just want to know which bits are coming from
A and which from B.

This is really important for us because of legal reasons.

> You could maybe use signed tags ("git help tag") - each contributor
> could sign a certain tree state, and if you see commits attributed to
> the other contributor after their last tag, you know something is
> fishy. But that might be more effort than either you or your
> contributors want to go through. And while it might help with
> attribution problems, it still doesn't help with all the other
> problems you might have with untrusted contributors.

The question is why Git doesn't sign all commits by default but only tags? Creating tags all the
time is rather tedious process and seems to have no sense, right?

Does it mean that with current Git design it's the best to not use advanced features of Git like
tree merging but simply go with posting e-mails with patches instead if contributors cannot be trusted?

Thanks for your reply.

-- 
Best regards,
Grzegorz Kossakowski

Re: How to clone git repository with git-svn meta-data included?

[Peter Harris]

On Sun, Dec 7, 2008 at 2:08 PM, Grzegorz Kossakowski wrote:
> Peter Harris pisze:
>> Make sure you don't use the --no-metadata flag when setting up
>> git-svn. This will embed the metadata into commit messages, so git-svn
>> can rebuild it from scratch whenever it needs to. (You probably also
>> want git 1.6.1rc for incremental rebuild support). This also has the
>> advantage that you can see the svn revision number when looking at a
>> commit message.
>
> Not sure what you exactly mean here. Do you mean that if metadata is included in commit messages
> then there is an easy way to initialize git-svn after cloning the repo?

Yes.

> By easy I mean:
> a) it does not require to much of interactive actions to be performed
> b) it does not pull too much from svn server
>
> Point b) is important because we usually have quite large repositories.

To set up the remotes to mirror the remote svn-remotes, I do the clone manually:
git init
git remote add origin git://svn/mirror
git config --add remote.origin.fetch +refs/remotes/*:refs/remotes/*
git fetch
git reset --hard trunk

After the git clone, I do the following:
git svn init -s svn://repo/sitory
git svn rebase

No data is transferred[1], although 'git svn rebase' does spend a
minute or so reading the commit messages to rebuild its index.

This could all be in a common script you distribute to your users.

> Also, could you point me to a place where this rebuild support is described? I would like to know
> what our committer has to do after cloning from Jukka's server.

"git help svn" mentions the rebuild only in passing. I'm not sure if
it is described in better detail elsewhere.

>> In terms of re-pulling from the git-svn mirror, git-svn will create
>> the same commits (with the same sha1s) from svn every time, so there
>> will be no conflicts there.
>
> Just to make sure: so if one person pulls from git-svn mirror and another one pulls using git svn
> rebase they result in the same tree right?

Yes[2].

>> If C doesn't trust A, C should not pull from A. C should pull only
>> from (trusted) B. Presumably B knows who (of A and B) did which work,
>> and B's repository can be trusted?
>>
>> If neither of A or B can be trusted, then you have problems that a
>> computer cannot solve for you.
>
> Yep, I was having in mind the case when both A and B are untrusted. I don't want my computer to
> check if something coming from A or B is safe or not I just want to know which bits are coming from
> A and which from B.
>
> This is really important for us because of legal reasons.

If something is in A's tree, it is coming from A. Either A has
authority, or A has received authority from someone else, or A is
bringing the legal problem down on himself. When A says "Please Pull"
(or when A pushes) A is effectively saying "These changes are legally
mine to give you".

The Developer's Certificate of Origin 1.0 was designed to address this
issue; see also "Signed-off-by"

Of course, if it's a legal issue, make sure you consult your own lawyer.

>> You could maybe use signed tags ("git help tag")...
>
> The question is why Git doesn't sign all commits by default but only tags? Creating tags all the
> time is rather tedious process and seems to have no sense, right?

Typing in your GPG passphrase for every single little commit would be
even more tedious, IMHO.

> Does it mean that with current Git design it's the best to not use advanced features of Git like
> tree merging but simply go with posting e-mails with patches instead if contributors cannot be trusted?

That would be my policy. At the very least, I would have a human
review the tree before merging it.

Note that git was designed around a "git am" workflow, so it is very
efficient at dealing with large numbers of patches at a time.

Note also that you can do tree merging with an email-patch based
workflow, since git format-patch preserves parent information,
although it does take a little bit more work. See also: "git help am"
under --3way.

Peter Harris

[1] Not strictly true. git-svn does contact the svn server to see if
there are any revisions newer than the latest present in the git repo,
and will transfer those revisions (if any).
[2] Unless (a) someone edited the svn:log (or other) revprop in
between, or (b) you triggered the bug I saw reported (and fixed?) on
this list today. I've never personally triggered (b), but I have seen
(a).

Re: How to clone git repository with git-svn meta-data included?

[Grzegorz Kossakowski]

Peter Harris pisze:
> After the git clone, I do the following:
> git svn init -s svn://repo/sitory
> git svn rebase
> 
> No data is transferred[1], although 'git svn rebase' does spend a
> minute or so reading the commit messages to rebuild its index.

I've tried this method with Cocoon repository
(http://jukka.zitting.name/git/?p=cocoon.git;a=summary) and got this error:

git clone git://jukka.zitting.name/cocoon.git
git svn init -s https://svn.eu.apache.org/repos/asf/cocoon/
git svn rebase
Unable to determine upstream SVN information from working tree history

git --version
git version 1.6.0.2

Any idea what's wrong here?

> This could all be in a common script you distribute to your users.

Good suggestion.

> "git help svn" mentions the rebuild only in passing. I'm not sure if
> it is described in better detail elsewhere.

Ah, I didn't spot this earlier. Thanks.

> If something is in A's tree, it is coming from A. Either A has
> authority, or A has received authority from someone else, or A is
> bringing the legal problem down on himself. When A says "Please Pull"
> (or when A pushes) A is effectively saying "These changes are legally
> mine to give you".
> 
> The Developer's Certificate of Origin 1.0 was designed to address this
> issue; see also "Signed-off-by"
> 
> Of course, if it's a legal issue, make sure you consult your own lawyer.

I see. Thanks for insightful comments.

>>> You could maybe use signed tags ("git help tag")...
>> The question is why Git doesn't sign all commits by default but only tags? Creating tags all the
>> time is rather tedious process and seems to have no sense, right?
> 
> Typing in your GPG passphrase for every single little commit would be
> even more tedious, IMHO.

Yep, that's true.

>> Does it mean that with current Git design it's the best to not use advanced features of Git like
>> tree merging but simply go with posting e-mails with patches instead if contributors cannot be trusted?
> 
> That would be my policy. At the very least, I would have a human
> review the tree before merging it.

Agreed.

> Note that git was designed around a "git am" workflow, so it is very
> efficient at dealing with large numbers of patches at a time.
> 
> Note also that you can do tree merging with an email-patch based
> workflow, since git format-patch preserves parent information,
> although it does take a little bit more work. See also: "git help am"
> under --3way.

Thanks for all your valuable information. As soon as I resolve problem with git svn rebase I'll
start reading on how git am --3way works.

-- 
Best regards,
Grzegorz Kossakowski

Re: How to clone git repository with git-svn meta-data included?

[Peter Harris]

On Sun, Dec 7, 2008 at 5:02 PM, Grzegorz Kossakowski wrote:
> Peter Harris pisze:
>> After the git clone, I do the following:
>> git svn init -s svn://repo/sitory
>> git svn rebase
>>
>> No data is transferred[1], although 'git svn rebase' does spend a
>> minute or so reading the commit messages to rebuild its index.
>
> I've tried this method with Cocoon repository
> (http://jukka.zitting.name/git/?p=cocoon.git;a=summary) and got this error:
>
> git clone git://jukka.zitting.name/cocoon.git
> git svn init -s https://svn.eu.apache.org/repos/asf/cocoon/
> git svn rebase
> Unable to determine upstream SVN information from working tree history

Odd. Usually that indicates a lack of metadata, but that repo appears
to contain the right stuff. Beats me.

Peter Harris

Re: How to clone git repository with git-svn meta-data included?

[Michael J Gruber]

Grzegorz Kossakowski venit, vidit, dixit 07.12.2008 23:02:
> Peter Harris pisze:
>> After the git clone, I do the following:
>> git svn init -s svn://repo/sitory
>> git svn rebase
>>
>> No data is transferred[1], although 'git svn rebase' does spend a
>> minute or so reading the commit messages to rebuild its index.
> 
> I've tried this method with Cocoon repository
> (http://jukka.zitting.name/git/?p=cocoon.git;a=summary) and got this error:
> 
> git clone git://jukka.zitting.name/cocoon.git
> git svn init -s https://svn.eu.apache.org/repos/asf/cocoon/
> git svn rebase
> Unable to determine upstream SVN information from working tree history
> 
> git --version
> git version 1.6.0.2

Could it be as simple as a missing "cd cocoon" between git clone and git
svn rebase? No, you probably did that.

But note that you did not follow Peter's instructions. The point is that
your clone creates "remotes/origin/trunk" whereas Peter's instructions
mirror the source, creating "remotes/trunk", which is what git svn needs
(unless you say "git svn init -s --prefix=origin/" or "git config
svn-remote.svn.fetch trunk:refs/trunk" etc.). The prefix solution should
be the best.

Michael

P.S.: Peter starts off a different layout (standard svn remotes, which
need special instructions to be cloned). Ordinary clone + git svn init
--prefix=origin/ should work fine for the cocoon layout.

P.P.S.: We can't test cocoon unless we have an account on the apache
server...

Re: How to clone git repository with git-svn meta-data included?

[Grzegorz Kossakowski]

Michael J Gruber pisze:
> Could it be as simple as a missing "cd cocoon" between git clone and git
> svn rebase? No, you probably did that.

That would be too easy.

> But note that you did not follow Peter's instructions. The point is that
> your clone creates "remotes/origin/trunk" whereas Peter's instructions
> mirror the source, creating "remotes/trunk", which is what git svn needs
> (unless you say "git svn init -s --prefix=origin/" or "git config
> svn-remote.svn.fetch trunk:refs/trunk" etc.). The prefix solution should
> be the best.
> 
> Michael
> 
> P.S.: Peter starts off a different layout (standard svn remotes, which
> need special instructions to be cloned). Ordinary clone + git svn init
> --prefix=origin/ should work fine for the cocoon layout.

This almost worked. Actually, Cocoon repository hosted on Jukka's server does not have local head
named "trunk" so there is no remotes/origin/trunk created during cloning process.

I had to run:

  git update-ref refs/remotes/origin/trunk refs/heads/master

After doing that git svn rebase resulted in:
[really long list of revisions]
r707379 = f61a2d30b6ac5a5136b46fa2b9b5b91e4763feb1
r710118 = 40997fe552e8581b75b08fed41a6b63a33d58bdf
r720135 = a8160766ec40fd7ebf95bfa7cebfa50dfa2f9c3a
r720180 = b094a222bab3671c8277087e7a96589ec76dd5e4
r720182 = 736b8ed6519c64ad120de2ccf08f135062ee09db
Done rebuilding .git/svn/origin/trunk/.rev_map.13f79535-47bb-0310-9956-ffa450edef68
Current branch master is up to date.

Is this expected output?

> P.P.S.: We can't test cocoon unless we have an account on the apache
> server...

I guess this would be a big problem. Getting write-access access to our repository is rather formal
process and I would like to avoid it.

Since it looks I'm much closer to final solution (and better understanding of how our workflow
should look like) I hope any additional account for testing won't be needed.

Anyway, thanks again for your answer!

-- 
Best regards,
Grzegorz Kossakowski

Re: How to clone git repository with git-svn meta-data included?

[Peter Harris]

On Mon, Dec 8, 2008 at 1:26 PM, Grzegorz Kossakowski wrote:
>
> After doing that git svn rebase resulted in:
> [really long list of revisions]
> r707379 = f61a2d30b6ac5a5136b46fa2b9b5b91e4763feb1
> r710118 = 40997fe552e8581b75b08fed41a6b63a33d58bdf
> r720135 = a8160766ec40fd7ebf95bfa7cebfa50dfa2f9c3a
> r720180 = b094a222bab3671c8277087e7a96589ec76dd5e4
> r720182 = 736b8ed6519c64ad120de2ccf08f135062ee09db
> Done rebuilding .git/svn/origin/trunk/.rev_map.13f79535-47bb-0310-9956-ffa450edef68
> Current branch master is up to date.
>
> Is this expected output?

Yes. The rfoo = sha1hash part is git-svn rebuilding its index.
"Current branch master is up to date" is git-svn calling "git rebase
<svn-branch>", and git saying that there is nothing to do, since there
have been no svn commits to that branch since the last time you ran
git svn rebase (or since you cloned the git mirror, or since the last
time the git mirror pulled from svn).

Peter Harris

Re: How to clone git repository with git-svn meta-data included?

[Grzegorz Kossakowski]

Peter Harris pisze:
> On Mon, Dec 8, 2008 at 1:26 PM, Grzegorz Kossakowski wrote:
>> After doing that git svn rebase resulted in:
>> [really long list of revisions]
>> r707379 = f61a2d30b6ac5a5136b46fa2b9b5b91e4763feb1
>> r710118 = 40997fe552e8581b75b08fed41a6b63a33d58bdf
>> r720135 = a8160766ec40fd7ebf95bfa7cebfa50dfa2f9c3a
>> r720180 = b094a222bab3671c8277087e7a96589ec76dd5e4
>> r720182 = 736b8ed6519c64ad120de2ccf08f135062ee09db
>> Done rebuilding .git/svn/origin/trunk/.rev_map.13f79535-47bb-0310-9956-ffa450edef68
>> Current branch master is up to date.
>>
>> Is this expected output?
> 
> Yes. The rfoo = sha1hash part is git-svn rebuilding its index.
> "Current branch master is up to date" is git-svn calling "git rebase
> <svn-branch>", and git saying that there is nothing to do, since there
> have been no svn commits to that branch since the last time you ran
> git svn rebase (or since you cloned the git mirror, or since the last
> time the git mirror pulled from svn).

Thanks for confirmation and explanation.

The remaining question is who should address this issue with non-existing trunk ref? Should I ask
Jukka, who maintains svn mirrors, to put instruction into his scripts that will add trunk reference?

Would it be the best practice?

-- 
Best regards,
Grzegorz Kossakowski

Re: How to clone git repository with git-svn meta-data included?

[Sam Vilain]

On Mon, 2008-12-08 at 19:43 +0100, Grzegorz Kossakowski wrote:
> > Yes. The rfoo = sha1hash part is git-svn rebuilding its index.
> > "Current branch master is up to date" is git-svn calling "git rebase
> > <svn-branch>", and git saying that there is nothing to do, since there
> > have been no svn commits to that branch since the last time you ran
> > git svn rebase (or since you cloned the git mirror, or since the last
> > time the git mirror pulled from svn).
> 
> Thanks for confirmation and explanation.
> 
> The remaining question is who should address this issue with non-existing trunk ref? Should I ask
> Jukka, who maintains svn mirrors, to put instruction into his scripts that will add trunk reference?

It's up to the git-svn user to make sure that they prepare the refs to
be what git-svn expects.  This is something probably requiring more
documentation and/or git-svn features to be easier.

> Would it be the best practice?

Well, obscure stuff should never really be best practice.  The best practice
is to have a single git repository that is where the svn -> git migration
happens.  And git-svn could perhaps auto-init based on information in the
commit log or something.  Best practice is to enhance the tool to work the
way it Should(tm) :)

Sam

Re: How to clone git repository with git-svn meta-data included?

[Grzegorz Kossakowski]

Sam Vilain pisze:
> It's up to the git-svn user to make sure that they prepare the refs to
> be what git-svn expects.  This is something probably requiring more
> documentation and/or git-svn features to be easier.

What I was asking if we should add trunk ref to svn mirror so others cloning it will have
'origin/trunk' reference created automatically during clone process so no extra steps would be needed.

To be honest, I don't understand how Git exactly handles all this refs mapping and rewriting (e.g.
during cloning). Having said that, I cannot foresee all possible implications of choosing particular
method of solving current issues thus asking you.

>> Would it be the best practice?
> 
> Well, obscure stuff should never really be best practice.  The best practice
> is to have a single git repository that is where the svn -> git migration
> happens.  And git-svn could perhaps auto-init based on information in the
> commit log or something.  Best practice is to enhance the tool to work the
> way it Should(tm) :)

Cannot follow you here. We want single svn mirror but at the same time we want to our committers to
be able to push back to svn. What has been already proposed satisfies my need apart from the fact
that currently there is  small problem with our mirror setup.

-- 
Best regards,
Grzegorz Kossakowski

Re: How to clone git repository with git-svn meta-data included?

[Michael J Gruber]

Grzegorz Kossakowski venit, vidit, dixit 08.12.2008 19:26:
> Michael J Gruber pisze:
>> Could it be as simple as a missing "cd cocoon" between git clone and git
>> svn rebase? No, you probably did that.
> 
> That would be too easy.
> 
>> But note that you did not follow Peter's instructions. The point is that
>> your clone creates "remotes/origin/trunk" whereas Peter's instructions
>> mirror the source, creating "remotes/trunk", which is what git svn needs
>> (unless you say "git svn init -s --prefix=origin/" or "git config
>> svn-remote.svn.fetch trunk:refs/trunk" etc.). The prefix solution should
>> be the best.
>>
>> Michael
>>
>> P.S.: Peter starts off a different layout (standard svn remotes, which
>> need special instructions to be cloned). Ordinary clone + git svn init
>> --prefix=origin/ should work fine for the cocoon layout.
> 
> This almost worked. Actually, Cocoon repository hosted on Jukka's server does not have local head
> named "trunk" so there is no remotes/origin/trunk created during cloning process.
> 
> I had to run:
> 
>   git update-ref refs/remotes/origin/trunk refs/heads/master

Uhm, I misread gitweb output... So, cocoon really has remotes/trunk, so
that Peter's original suggestion would work. If the cocoon git-svn clone
isn't going to change then Peter's way of doing it might be the best: it
ensures that future pulls from origin (cocoon) will update the correct
refs so that you can pull/fetch from the git-svn clone (fast) and then
git-svn rebase rather than git-svn rebasing directly, which would fetch
new commits from the svn repo first (slow).

I think all in all it shows that git-svns default location for branches
is not the best choice: they're not local, but they're no "proper"
remotes either (unless you use --prefix).

Michael

Re: How to clone git repository with git-svn meta-data included?

[Shawn O. Pearce]

Grzegorz Kossakowski <grek@tuffmail.com> wrote:
> Peter Harris pisze:
> >> What if A was not fair and has rewritten a few commits coming from B so they contain malicious code?
> >> How we can detect something like that and how C be sure that what he merges is really work
> >> attributed by correct names?
> > 
> > If C doesn't trust A, C should not pull from A. C should pull only
> > from (trusted) B. Presumably B knows who (of A and B) did which work,
> > and B's repository can be trusted?
> > 
> > If neither of A or B can be trusted, then you have problems that a
> > computer cannot solve for you.
> 
> Yep, I was having in mind the case when both A and B are untrusted. I don't want my computer to
> check if something coming from A or B is safe or not I just want to know which bits are coming from
> A and which from B.
> 
> This is really important for us because of legal reasons.

ASF probably has issues similar to that of the Android project.

In Android we built Gerrit[1] to handle this validation of identity
for us, and to keep track of the contributor agreements each
individual and corporation has signed.  Changes aren't accepted
into Gerrit unless the user has an accepted CLA in the data store.

*1* http://review.source.android.com/

Gerrit 2 is actively under development and is being ported off of
Google App Engine, into a pure Java webapp.  I'm running it under
Jetty, but it should work just as well under Tomcat.  :-)

If the ASF becomes more committed to supporting Git, Gerrit may be
a good way to answer some of the questions you are having about
validating identity of changes.  Plus its a handy source code
review tool.
 
> > You could maybe use signed tags ("git help tag") - each contributor
> > could sign a certain tree state, [...]
> 
> The question is why Git doesn't sign all commits by default but only tags? Creating tags all the
> time is rather tedious process and seems to have no sense, right?

Yea, its tedious to unlock your GnuPG key every time you make a
commit.  Especially if you are just rebasing a series or something
to fix a minor mistake 5 commits back before uploading somewhere.
 
> Does it mean that with current Git design it's the best to not use advanced features of Git like
> tree merging but simply go with posting e-mails with patches instead if contributors cannot be trusted?

Most Git projects rely on patches sent to an email list, with
a single maintainer applying them to to his/her repository, and
publishing the result.  The maintainer is thus forced to keep track
of the CLAs (if the project uses such things) and just trust the
From address of the message.

CLAs in the kernel and in git itself are less enforced than say
what ASF or Android requires.

Some Git projects give write access to the master repository to
multiple trusted parties; SAMBA and X.org are good examples of this
sort of strategy.  But I think in these cases those who have write
access are also very long standing members of the development
community who have known each other in person for many years,
perhaps far longer than a DVCS concept has existed.  So trust
between those with direct write access is slightly less of an
issue for these projects.

So long story short, I think Gerrit may be worth the ASF's time,
if Git is a serious consideration for replacing SVN.  But while a
project is based in SVN I think the best you can do with Git is
publish an automatically updated git-svn mirror and permit only
use of "git svn dcommit" to upload back into the SVN repository.

-- 
Shawn.

Re: How to clone git repository with git-svn meta-data included?

[Grzegorz Kossakowski]

Shawn O. Pearce pisze:
> ASF probably has issues similar to that of the Android project.
> 
> In Android we built Gerrit[1] to handle this validation of identity
> for us, and to keep track of the contributor agreements each
> individual and corporation has signed.  Changes aren't accepted
> into Gerrit unless the user has an accepted CLA in the data store.
> 
> *1* http://review.source.android.com/

I'm not an expert on legal issues at Apache but in general you might be right that ASF and Android
project have similar issues to tackle. I've brought this point because it was previously brought in
ASF discussion on Git. so I try to act more like a bridge between two parties.

> Gerrit 2 is actively under development and is being ported off of
> Google App Engine, into a pure Java webapp.  I'm running it under
> Jetty, but it should work just as well under Tomcat.  :-)

Is there any documentation outlining Gerrit's features and status of Gerrit 2 development?

> If the ASF becomes more committed to supporting Git, Gerrit may be
> a good way to answer some of the questions you are having about
> validating identity of changes.  Plus its a handy source code
> review tool.

I guess it's rather long run before Apache Infrastructure team *officially* starts supporting Git.
This is my personal view and it's not official voice of ASF by any means. SVN to Git transition
(entirely) is complicated process for such a big organization like ASF.

Another option would be to let particular projects choose which SCM they prefer but that would mean
that our infra team has to support two different SCMs. This means some new folks interested in
supporting Git at ASF would have to join infra.

At this point, we chose to let people experiment with Git (so we are allowed to perform rather
expensive clone operations) in order to gain some experience and work out work-flow that suits ASF
model of development. This thread is part of described process.

On the other hand, if we find Gerrit useful for us we might try to set up it somewhere unofficially
and let people play with it.

>> Does it mean that with current Git design it's the best to not use advanced features of Git like
>> tree merging but simply go with posting e-mails with patches instead if contributors cannot be trusted?
> 
> Most Git projects rely on patches sent to an email list, with
> a single maintainer applying them to to his/her repository, and
> publishing the result.  The maintainer is thus forced to keep track
> of the CLAs (if the project uses such things) and just trust the
> From address of the message.

It's rather important to keep in mind that at ASF we don't how practice of single maintainer so we
need solution that works for many committers that push to the same repo but I know Git supports this
rather easily.

> CLAs in the kernel and in git itself are less enforced than say
> what ASF or Android requires.
> 
> Some Git projects give write access to the master repository to
> multiple trusted parties; SAMBA and X.org are good examples of this
> sort of strategy.  But I think in these cases those who have write
> access are also very long standing members of the development
> community who have known each other in person for many years,
> perhaps far longer than a DVCS concept has existed.  So trust
> between those with direct write access is slightly less of an
> issue for these projects.

That is a case for Apache to some extent. Someone is being nominated to become a committer after
some period of cooperation in terms of providing patch, participating in discussions etc. After such
period existing committers can rather reliably judge if they trust particular person or not.

Yet still signed CLA is required but we check this before someone is being granted write access so
this can be done by hand.

> So long story short, I think Gerrit may be worth the ASF's time,
> if Git is a serious consideration for replacing SVN.  But while a
> project is based in SVN I think the best you can do with Git is
> publish an automatically updated git-svn mirror and permit only
> use of "git svn dcommit" to upload back into the SVN repository.

As I said, I don't think ASF will switch to Git any time soon due to many different reasons
including technical (IDE support), social (there are different opinions on DVCS in general) and
other. All of them have to be addressed but in order to have any meaningful discussion we need to
gather some experience.

Thanks for helping me with going through this process!

-- 
Best regards,
Grzegorz Kossakowski