Help using Git(-svn) for specific use case

Help using Git(-svn) for specific use case

[Pico Geyer]

Hi all.

I'd like to start using Git within our company. I'm still trying to
determine if Git can work in our scenario.
May be someone can offer some advice.
Our scenario is as follows:
We have a handful of developers which work at a branch office. This
office has a rather slow internet connection.
At the head office, we are (still) using a Subversion server to host
our source code.
At the branch office, we would like to do the following:
* Fetch the latest code from the subversion server.
* Push changes that we have made at the remote office back to upstream
SVN server.
* Be able to share source code changes locally (at the branch office)
between developers.

The solution that I imagined is that we would setup a server (lets
call it A) at the branch office with a Git repository, that all the
developers can access.
Developers would then clone the server repository A, make mods and
then push changes back to A.
It would be nice if git users could commit to a subversion branch.
A merge master would then (as necessary) use svn dcommit to push the
changes up to the svn server.
Is this scenario possible?

>From the git-svn man page: "git-clone does not clone branches under
the refs/remotes/ hierarchy or any git-svn metadata, or config. So
repositories created and managed with using git-svn should use rsync
for cloning, if cloning is to be done at all."
Does that mean that one should not push changes to a repository that
was created with "git svn clone ..."?

Thanks in advance for any advice.

Pico Geyer

Re: Help using Git(-svn) for specific use case

[Michael J Gruber]

Pico Geyer venit, vidit, dixit 15.09.2008 14:50:
> Hi all.
> 
> I'd like to start using Git within our company. I'm still trying to
> determine if Git can work in our scenario.
> May be someone can offer some advice.
> Our scenario is as follows:
> We have a handful of developers which work at a branch office. This
> office has a rather slow internet connection.
> At the head office, we are (still) using a Subversion server to host
> our source code.
> At the branch office, we would like to do the following:
> * Fetch the latest code from the subversion server.
> * Push changes that we have made at the remote office back to upstream
> SVN server.
> * Be able to share source code changes locally (at the branch office)
> between developers.
> 
> The solution that I imagined is that we would setup a server (lets
> call it A) at the branch office with a Git repository, that all the
> developers can access.
> Developers would then clone the server repository A, make mods and
> then push changes back to A.
> It would be nice if git users could commit to a subversion branch.
> A merge master would then (as necessary) use svn dcommit to push the
> changes up to the svn server.
> Is this scenario possible?
> 
> From the git-svn man page: "git-clone does not clone branches under
> the refs/remotes/ hierarchy or any git-svn metadata, or config. So
> repositories created and managed with using git-svn should use rsync
> for cloning, if cloning is to be done at all."

That is true if you mean by "clone" a complete copy.

> Does that mean that one should not push changes to a repository that
> was created with "git svn clone ..."?

What matters is which branches you push into.

Your developers don't need to push/commit to svn, only into A, if I
understood correctly. Do they need svn-metadata (besides svn revision
numbers)?
If not then the interaction between A and the developers can be a normal
git workflow between git managed branches, and "git clone" will do fine.

Merge master at A could do the following:
- use git svn to keep the svn (remote) branches at A in sync with the
central subversion server
- push them to proper git branches in A, which get picked up by
developers' " git clone"
- maintain integration branches for integrating the changes coming fro
the developers
- dcommit the svn tracking branches back to the central svn server

Disadvantage:
If changes are dcommitted to svn, synced back with git svn fetch, and
then fetched/pull by the developers their commit will look different
(addition git svn info in the commit message). This is similar to a
workflow where you submit by e-mail and your integrated patches get a
different sha1.

Advantage:
For all developers, everything is just git. Only megre master needs to
know git svn.

Cheers,
Michael

Re: Help using Git(-svn) for specific use case

[Dmitry Potapov]

On Mon, Sep 15, 2008 at 4:50 PM, Pico Geyer <picogeyer@gmail.com> wrote:
>
> At the head office, we are (still) using a Subversion server to host
> our source code.
> At the branch office, we would like to do the following:
> * Fetch the latest code from the subversion server.
> * Push changes that we have made at the remote office back to upstream
> SVN server.
> * Be able to share source code changes locally (at the branch office)
> between developers.

I used Git in somewhat similar situation where almost all development
was done in Git mirror of SVN, and then the work was propagated to SVN.
There are some limitations with this approach. The most important is
that SVN does not support merges, so you have to linerialize all history
that you want to commit in SVN. That means that you are going to use git
rebase a lot to rewrite history, which is a PITA. Another problem with
this approach is that the author is not preserved. In SVN, all commits will
look like all changes are done by the person who run "git svn dcommit".

Yet, despite of all these limitations caused by using SVN as backend,
it was far more pleasant to work with Git than with SVN (you can save
your work in progress, you can easily organize the review process,
and many other features such as "git bisect" or "git log -S" just to
name a few...).

>
> The solution that I imagined is that we would setup a server (lets
> call it A) at the branch office with a Git repository, that all the
> developers can access.
> Developers would then clone the server repository A, make mods and
> then push changes back to A.
> It would be nice if git users could commit to a subversion branch.
> A merge master would then (as necessary) use svn dcommit to push the
> changes up to the svn server.
> Is this scenario possible?

Each user has its own copy of the whole repository and they always
commit to their _local_ branches. Then they can push their changes to
the Git repo on your server, but it is a bad idea to push changes to
a local branch of a non-bare repository especially if that branch is
checked out, because push does not (and should not!) update the working
tree and that leads to the situation where all pushed changes are
looked undo in the working tree of that repository. Therefore, you
will probably have a separately integration branch to which users
will push. Oftentimes, users will have to rebase their work on top the
integration branch to be able to push on it (only fast-forward push
should be allowed).

The merge master will merge this branch to the local branch assotiated
with subversion (it will be fast-forward, not a real merge) and then
run git svn dcommit to propagate to subversion.

If there were some conflicting changes in subversion repo, the merge
master will have to resolve them. After that update the merge base
should be updated (using git branch -f) to point out to the top of
the subversion branch.

If users have some changes that were based on the old integration
branches they will have to rebase them to the new position of the
integration branch. As I said above, git rebase is going to be very
often operation in this workflow.

NOTE: branches in git differs from what they are in subversion. In
essence, they are no more than pointers to where to add the next
commit. When I learned Git, I found very useful the article called
"Git for computer scientists" to understand basic ideas of how Git
works: http://eagain.net/articles/git-for-computer-scientists/

>
> From the git-svn man page: "git-clone does not clone branches under
> the refs/remotes/ hierarchy or any git-svn metadata, or config. So
> repositories created and managed with using git-svn should use rsync
> for cloning, if cloning is to be done at all."
> Does that mean that one should not push changes to a repository that
> was created with "git svn clone ..."?

No, what the above description says is that SVN specific information is
not cloned when you clone a repo created by svn. It means that users who
cloned from you will not be able to use git-svn to push their changes in
the original subversion repository. Yet, they can push to your repository
as to any other Git repository.

Dmitry

Re: Help using Git(-svn) for specific use case

[Sverre Rabbelier]

On Mon, Sep 15, 2008 at 21:55, Dmitry Potapov <dpotapov@gmail.com> wrote:
> [...] Another problem with
> this approach is that the author is not preserved. In SVN, all commits will
> look like all changes are done by the person who run "git svn dcommit".

I thought this was fixed in a more recent version of git-svn? Didn't
it even work both ways?

-- 
Cheers,

Sverre Rabbelier

Re: Help using Git(-svn) for specific use case

[Michael J Gruber]

Sverre Rabbelier venit, vidit, dixit 15.09.2008 22:59:
> On Mon, Sep 15, 2008 at 21:55, Dmitry Potapov <dpotapov@gmail.com> wrote:
>> [...] Another problem with
>> this approach is that the author is not preserved. In SVN, all commits will
>> look like all changes are done by the person who run "git svn dcommit".
> 
> I thought this was fixed in a more recent version of git-svn? Didn't
> it even work both ways?

Kind of. You can't fake a different author when committing to svn. But
"--add-author-from" makes dcommit embed the author in the svn commit
message (if there's no from nor sob), and "--use-log-author" makes fetch
look for that info and use it.

Michael

Re: Help using Git(-svn) for specific use case

[Dmitry Potapov]

On Mon, Sep 15, 2008 at 11:15:59PM +0200, Michael J Gruber wrote:
> Sverre Rabbelier venit, vidit, dixit 15.09.2008 22:59:
> > 
> > I thought this was fixed in a more recent version of git-svn? Didn't
> > it even work both ways?
> 
> Kind of. You can't fake a different author when committing to svn.

This is not exactly correct as you can fake author if you have the right
to change the svn:author property (or what is its name?), but by default
this is not allowed to anyone, and git-svn does not support this method
anyway.

> But
> "--add-author-from" makes dcommit embed the author in the svn commit
> message (if there's no from nor sob), and "--use-log-author" makes fetch
> look for that info and use it.

Yes, it works. And it is very useful especially you are going eventually
convert your SVN repository in Git (so all authorship information will
be retain). Probably, I should have mentioned this possibility.

Thanks,
Dmitry

Re: Help using Git(-svn) for specific use case

[Pico Geyer]

Thanks guys for your replies and advice.

I'm going to be trying to configure such a setup tomorrow.
I'll probably have more questions :-)

Regards,
Pico Geyer

On Mon, Sep 15, 2008 at 11:47 PM, Dmitry Potapov <dpotapov@gmail.com> wrote:
> On Mon, Sep 15, 2008 at 11:15:59PM +0200, Michael J Gruber wrote:
>> Sverre Rabbelier venit, vidit, dixit 15.09.2008 22:59:
>> >
>> > I thought this was fixed in a more recent version of git-svn? Didn't
>> > it even work both ways?
>>
>> Kind of. You can't fake a different author when committing to svn.
>
> This is not exactly correct as you can fake author if you have the right
> to change the svn:author property (or what is its name?), but by default
> this is not allowed to anyone, and git-svn does not support this method
> anyway.
>
>> But
>> "--add-author-from" makes dcommit embed the author in the svn commit
>> message (if there's no from nor sob), and "--use-log-author" makes fetch
>> look for that info and use it.
>
> Yes, it works. And it is very useful especially you are going eventually
> convert your SVN repository in Git (so all authorship information will
> be retain). Probably, I should have mentioned this possibility.
>
> Thanks,
> Dmitry
>

Re: Help using Git(-svn) for specific use case

[Pico Geyer]

Hi Dmitry

I'm going to give example commands to confirm my understanding of the
git-svn workflow you described.
Any comments or corrections are welcome.

On the local server, get a clone of the repository:
srvA# git svn clone http://server.com/repo/proj/trunk proj
srvA# cd proj

Create a local branch, where developers will push to
srvA# git branch integration

Now a developer clones the repository on the server, and makes a local
branch for developement:
dev1# git clone ssh://svrA/path/to/repo
dev1# git checkout -b new_featureX
(Developer makes some changes and commits to the new_featureX branch)

Developer now pushes the changes to the integration branch
dev1# git push origin new_featureX:integration

The merge master now rebases the changes on to the master branch:
srvA# git rebase integration

Now push the changes to the upstream subversion server:
srvA# git svn dcommit

At some stage svn users will commit to the subversion repository, and
we will need to pull changes into srvA:
srvA# git svn rebase

Now the integration branch does not include the changes in master, so
rebase the changes to integration:
srvA# git rebase master integration

Finally the developer needs those changes too, so on his master branch:
dev1# git rebase origin

Is this the process that you were trying to describe?
Any additional steps that need to be added?

And how do I make sure that when developers push to the server that it
results in a fast-forward merge? Do I do that with commit hooks?

Thanks again for the advice.
Pico

On Mon, Sep 15, 2008 at 9:55 PM, Dmitry Potapov <dpotapov@gmail.com> wrote:
> On Mon, Sep 15, 2008 at 4:50 PM, Pico Geyer <picogeyer@gmail.com> wrote:
>>
>> At the head office, we are (still) using a Subversion server to host
>> our source code.
>> At the branch office, we would like to do the following:
>> * Fetch the latest code from the subversion server.
>> * Push changes that we have made at the remote office back to upstream
>> SVN server.
>> * Be able to share source code changes locally (at the branch office)
>> between developers.
>
> I used Git in somewhat similar situation where almost all development
> was done in Git mirror of SVN, and then the work was propagated to SVN.
> There are some limitations with this approach. The most important is
> that SVN does not support merges, so you have to linerialize all history
> that you want to commit in SVN. That means that you are going to use git
> rebase a lot to rewrite history, which is a PITA. Another problem with
> this approach is that the author is not preserved. In SVN, all commits will
> look like all changes are done by the person who run "git svn dcommit".
>
> Yet, despite of all these limitations caused by using SVN as backend,
> it was far more pleasant to work with Git than with SVN (you can save
> your work in progress, you can easily organize the review process,
> and many other features such as "git bisect" or "git log -S" just to
> name a few...).
>
>>
>> The solution that I imagined is that we would setup a server (lets
>> call it A) at the branch office with a Git repository, that all the
>> developers can access.
>> Developers would then clone the server repository A, make mods and
>> then push changes back to A.
>> It would be nice if git users could commit to a subversion branch.
>> A merge master would then (as necessary) use svn dcommit to push the
>> changes up to the svn server.
>> Is this scenario possible?
>
> Each user has its own copy of the whole repository and they always
> commit to their _local_ branches. Then they can push their changes to
> the Git repo on your server, but it is a bad idea to push changes to
> a local branch of a non-bare repository especially if that branch is
> checked out, because push does not (and should not!) update the working
> tree and that leads to the situation where all pushed changes are
> looked undo in the working tree of that repository. Therefore, you
> will probably have a separately integration branch to which users
> will push. Oftentimes, users will have to rebase their work on top the
> integration branch to be able to push on it (only fast-forward push
> should be allowed).
>
> The merge master will merge this branch to the local branch assotiated
> with subversion (it will be fast-forward, not a real merge) and then
> run git svn dcommit to propagate to subversion.
>
> If there were some conflicting changes in subversion repo, the merge
> master will have to resolve them. After that update the merge base
> should be updated (using git branch -f) to point out to the top of
> the subversion branch.
>
> If users have some changes that were based on the old integration
> branches they will have to rebase them to the new position of the
> integration branch. As I said above, git rebase is going to be very
> often operation in this workflow.
>
> NOTE: branches in git differs from what they are in subversion. In
> essence, they are no more than pointers to where to add the next
> commit. When I learned Git, I found very useful the article called
> "Git for computer scientists" to understand basic ideas of how Git
> works: http://eagain.net/articles/git-for-computer-scientists/
>
>>
>> From the git-svn man page: "git-clone does not clone branches under
>> the refs/remotes/ hierarchy or any git-svn metadata, or config. So
>> repositories created and managed with using git-svn should use rsync
>> for cloning, if cloning is to be done at all."
>> Does that mean that one should not push changes to a repository that
>> was created with "git svn clone ..."?
>
> No, what the above description says is that SVN specific information is
> not cloned when you clone a repo created by svn. It means that users who
> cloned from you will not be able to use git-svn to push their changes in
> the original subversion repository. Yet, they can push to your repository
> as to any other Git repository.
>
> Dmitry
>

Re: Help using Git(-svn) for specific use case

[Dmitry Potapov]

Hi Pico,

I will try answer your question, but unfortunately I don't have much
time for detail explanations, so maybe someone else will add a few
more comments.

On Wed, Sep 17, 2008 at 04:09:51PM +0200, Pico Geyer wrote:
> 
> I'm going to give example commands to confirm my understanding of the
> git-svn workflow you described.
> Any comments or corrections are welcome.
> 
> On the local server, get a clone of the repository:
> srvA# git svn clone http://server.com/repo/proj/trunk proj
> srvA# cd proj
> 
> Create a local branch, where developers will push to
> srvA# git branch integration
> 
> Now a developer clones the repository on the server, and makes a local
> branch for developement:
> dev1# git clone ssh://svrA/path/to/repo
> dev1# git checkout -b new_featureX
> (Developer makes some changes and commits to the new_featureX branch)

I believe developers should start from the integration branch, i.e.:

git checkout -b new_featureX origin/integration

> 
> Developer now pushes the changes to the integration branch
> dev1# git push origin new_featureX:integration

If someone already push there, push will refuse saying that it is
non-fast forward update. So, the developer will have to rebase his/her
branch on top of origin/integration:

  git fetch && git rebase origin/integration

or provided that you created your branch from origin/integration,
you can run:

  git pull --rebase

which does git fetch and rebase on the top of the original branch.

> 
> The merge master now rebases the changes on to the master branch:
> srvA# git rebase integration

What this commands does is to rebase your current branch on top
of the integration branch. I suppose you have something like
this before running it:

master            integration
 o------o-----o-------o

If you are staying at the master, it will advance master to
integration. So, yes, it does what you need, but your wording:
"rebases the changes on to the master branch" is not correct,
because it is master is rebased on top of integration, which
in your case is just fast-forward update of it to integration:

                  integration=master
o------o-----o-------o

you can have the same effect by "git merge integration".

> 
> Now push the changes to the upstream subversion server:
> srvA# git svn dcommit

I usually run "git svn dcommit --dry-run" first to make sure that it
will commit what I want. It is very useful when you try a new setup.

> 
> At some stage svn users will commit to the subversion repository, and
> we will need to pull changes into srvA:
> srvA# git svn rebase

Yes, but you usually do that before git svn dcommit...

> 
> Now the integration branch does not include the changes in master, so
> rebase the changes to integration:
> srvA# git rebase master integration

That is okay but may lead to the situation that conflicts that were
already resolved during git svn rebase will be re-emerge again.

You can force integration branch to be the same master after you 
have dcommited changes to SVN using the following command:
   git branch -f integration master
but if someone push into repository after the point where 'master' was
equal to 'integration' then those changes may be lost. To prevent this
from happening, you can lock repository (or only this particular branch)
from being push into when you dcommit changes to svn. Then when you have
finished, you can safely use the above command. The lock can be
implemented using the 'update' hook (for instance, the hook verifies
whether the "integration.lock" tag exists, if it does then the hook
exists with non-zero value).

> 
> Finally the developer needs those changes too, so on his master branch:
> dev1# git rebase origin

No, it is
  git fetch && git rebase origin/integration
or just
  git pull --rebase
provided that you started your branch from origin/integration.

> 
> Is this the process that you were trying to describe?
> Any additional steps that need to be added?

Yes, basically, it is. It could be some variations though. Instead of
having the integration branch in the repository that you use to sync
with SVN, you can have a separate bare repository where 'master' will
represent the integration branch, while git-svn repository is private to
the merge master. The advantage is that merge master does not need the
shell access to the server. Also, developers see only one branch 'master'
(less confusion about what to use as the starting point). In this setup,
you need to allow to the merge master non-fast forward update to the
bare repository (to update 'master' after commiting changes to SVN),
while the rest developers should have only fast-forward update.

> 
> And how do I make sure that when developers push to the server that it
> results in a fast-forward merge? Do I do that with commit hooks?

No, all commits and merges are purely local things. git push never
merges anything, it only propogates changes and update the branch
references. The only thing you should guarantee that no one (except
the merge master if necessary) can do non-fast forward push to server.
To disable to all users non-fast forward update, you can set
receive.denyNonFastForwards = true

If you disable to some users but not to others then you can use
either $GIT_DIR/hooks/pre-receive or $GIT_DIR/hooks/update.
See man git-receive-pack for more information, and examples
of the update hook included in Git.


Dmitry