Git push over git protocol for corporate environment

Git push over git protocol for corporate environment

[Eugene Sajine]

First of all thanks to everybody for an amazing tool! I’m a big fan of
it, so I’m trying to get rid of CVS in my company and migrate to Git.

We are working in, I would say, standard corporate environment, so all
development is in internal network. We have multiple offices in
different countries. Currently as I said we are using CVS (don’t
ask!;))

My problem is that I need the simplest, easiest and fastest solution
from setup and maintenance point of view in a situation when we have a
huge CVS repo with hundreds of modules (projects) in it. My current
understanding is that we are going to pull out project by project from
CVS and create corresponding git repos.
So, this brings us to hundreds of git repos and over 200 hundred
committers. In this circumstances we don’t want to manage each repo
separately as well as we don’t want to manage each person write access
rights to each repo.
As I understand the best solution here is git protocol (one port only
on dedicated server and no security as we are in trusted network) with
read and write access configured for all repos on a dedicated server.
What do you think I should do? How to enable push over git protocol?

I would appreciate any recommendation about such set up and any links
to corresponding docs.

Thank you,
Eugene

Re: Git push over git protocol for corporate environment

[David Brown]

On Wed, Sep 30, 2009 at 04:13:23PM -0700, Eugene Sajine wrote:

> As I understand the best solution here is git protocol (one port only
> on dedicated server and no security as we are in trusted network) with
> read and write access configured for all repos on a dedicated server.
> What do you think I should do? How to enable push over git protocol?

You can pass --enable=receive-pack but it probably isn't what you
want.  Anybody can write anything, anywhere with that, and more
importantly, anybody can delete anything.

When we started with git, we had a single machine that housed the
repos.  It ran a read-only git server, and people used ssh to
push to it.  It doesn't require accounts on the machine, but you
can use git-shell to restrict access.  This is probably a good
way to start out.

Eventually, it's possible to realize that there doesn't need to
be _the_ central server.  There can be several, and different
people in charge of different parts.  Here at least, people never
really adapted to this model.

We're now primarily using Gerrit, but that's a larger step from
process change from CVS.

David Brown

Re: Git push over git protocol for corporate environment

[Jakub Narebski]

Eugene Sajine <euguess@gmail.com> writes:

> My problem is that I need the simplest, easiest and fastest solution
> from setup and maintenance point of view in a situation when we have a
> huge CVS repo with hundreds of modules (projects) in it. My current
> understanding is that we are going to pull out project by project from
> CVS and create corresponding git repos.
>
> So, this brings us to hundreds of git repos and over 200 hundred
> committers. In this circumstances we don’t want to manage each repo
> separately as well as we don’t want to manage each person write access
> rights to each repo.
>
> As I understand the best solution here is git protocol (one port only
> on dedicated server and no security as we are in trusted network) with
> read and write access configured for all repos on a dedicated server.
> What do you think I should do? How to enable push over git protocol?

No, I don't think it is a good solution, as git protocol is by design
anonymous and unauthenticated.

To enable push via git protocol, you have to enable 'receive-pack'
service for git-daemon (the --enable=<service> option).

> 
> I would appreciate any recommendation about such set up and any links
> to corresponding docs.

You would probably want to use some tool to manage git repositories, 
like
 * Gitosis (in Python, requires setuptools),
 * Gitolite (in Perl),
 * SCuMD (in Java),
or even
 * ssh_acl

I think Gitosis is most commonly used tool, see links in
http://git.or.cz/gitwiki/InterfacesFrontendsAndTools and 
http://git.or.cz/gitwiki/BlogPosts pages on git wiki.

There are also full-fledged git hosting solutions, usually with web
interface to git repositories administration:
 * GitHub:FI (proprietary, non-free)
 * Gitorious (Ruby on Rails)
 * InDefero (PHP, clone of Google Code)
 * Girocco (Perl + bash, used by http://repo.or.cz)


There are also tools such as repo and Gerrit from Android project
(Gerrit is a review board).


Also, depending on workflow used, you might not need for anyone beside
project maintainer to have push access to public repository;
maintainer would process pull requests from co-developers, from their
per-developer forks.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Re: Git push over git protocol for corporate environment

[Eugene Sajine]

I'm sorry if it will be a dup again... My first email got stuck or
deleted by server, although i didn't use any html...

Thanks to everybody for prompt answers!

There is one thing I’m still missing though. Do I understand correctly
that if a person has an ssh access (account) to the host in internal
network, then this won’t be enough for him to be able to push to the
repo? Should we still go through the hassle of managing the ssh keys
for each particular user who is supposed to have push access?

I believe the answer is yes and that's why I'm leaning towards pulls
and pushes over git protocol. There is no solution yet which would be
as effective and simple to maintain. Using git protocol will not add
security, but it won't be worse than existing CVS or any other
centralized version control security model. As soon as security comes
into play, then we will need some other solution, but currently i
didn't see anything that would be easy to sell to the company.

Github is cool, but FI is way too expensive and very hard to sell.

Gitorious is even better!! for corporate use, i think, because of its
team oriented approach, but... man... I would kill for java
implementation or anything as simple as that!! As i see It is
impossible to install in network without internet access, and the
amount of dependencies which you have install/pre-install is enormous.
I read somewhere ruby on rails is fun to develop with, but is a
nightmare to deploy and maintain, and it seems to be true. Come on,
guys!! Look at the Hudson CI - one war file containing everything you
need, application starts from command line "java -jar hudson.war" and
runs on any port you specify. Time to start from download to having
first build is less the 10 min!!! If there are gitorious guys -
please, think about it and don't forget to share the profit;)!

I think Cgit can be something competitive - although i failed to run
it yet, having some issues with build...and as all other web based
stuff, you should implement something in order to create and set up
bare repos on the server automatically (even probably edit the config
file via script) to avoid a mess and to avoid one guy spending his
time adding and configuring repos... Probably we will and up using
gitweb as it at least knows to scan a folder for git repos...although
it also gives me troubles installing... both with cgit and gitweb are
conducted under cygwin, so probably this is the real problem with
them;)

I think that this is what is missing right now in order for git to get
rocket start and spread inside companies: secure and easy to maintain
mainline hosting.

Probably my lack of experience with git causes these thoughts - so,
while i will continue to work on it, i would really appreciate any
advice, especially about experience using git not for open source and
not in 3 person's team.

Thanks a lot,
Eugene

Re: Re: Git push over git protocol for corporate environment

[Shawn O. Pearce]

Eugene Sajine <euguess@gmail.com> wrote:
> Gitorious is even better!! for corporate use, i think, because of its
> team oriented approach, but... man... I would kill for java
> implementation or anything as simple as that!!

If you want a Java based server, look at either:

* SCuMD               http://github.com/gaffo/scumd
* Gerrit Code Review  http://code.google.com/p/gerrit/

I think SCuMD might be easier to install, I don't think it depends
upon a database or a servlet container like Gerrit does.  But both
are a SSH+Git implementation with some access control capabilities,
and are implemented in Java.

I don't think either is (yet) as easy to install as Hudson CI.
Both projects have a much smaller team of developers behind them,
and are still focusing on basic functionality rather than ease of
new system setup.

-- 
Shawn.

Re: Re: Git push over git protocol for corporate environment

[Eugene Sajine]

Thanks Shawn!

I saw info about Scumd and Gerrit in previous emails, but
unfortunately haven't enough time to spend with those tools yet.
Reading about Gerrit right now.


On Fri, Oct 2, 2009 at 10:47 AM, Shawn O. Pearce <spearce@spearce.org> wrote:
> Eugene Sajine <euguess@gmail.com> wrote:
>> Gitorious is even better!! for corporate use, i think, because of its
>> team oriented approach, but... man... I would kill for java
>> implementation or anything as simple as that!!
>
> If you want a Java based server, look at either:
>
> * SCuMD               http://github.com/gaffo/scumd
> * Gerrit Code Review  http://code.google.com/p/gerrit/
>
> I think SCuMD might be easier to install, I don't think it depends
> upon a database or a servlet container like Gerrit does.  But both
> are a SSH+Git implementation with some access control capabilities,
> and are implemented in Java.
>
> I don't think either is (yet) as easy to install as Hudson CI.
> Both projects have a much smaller team of developers behind them,
> and are still focusing on basic functionality rather than ease of
> new system setup.
>
> --
> Shawn.
>

Re: Git push over git protocol for corporate environment

[Ismael Luceno]

[-- Attachment #1: Type: text/plain, Size: 1077 bytes --]

Eugene Sajine escribió:
> I think that this is what is missing right now in order for git to get
> rocket start and spread inside companies: secure and easy to maintain
> mainline hosting.
> 

It looks like your problem is using cygwin. It's more complicated on a
MS-Windows environment, and personally I think it's a _very bad idea_.

Git is really easy to use in fact, you just set up the repo with:

  mkdir repo.git
  cd repo.git
  git init --bare --shared=all

--shared=all makes the repo readable to anyone, and ensures push rights
to users under the same group as the user setting up the repo. You can
change the group with chmod of course.

SSH access will be needed to push, unless the users can remotely mount
the repo via NFS or any other protocol.

Pulling is possible over http too, you just need to make
hooks/post-update executable. To export via git protocol you must create
an empty file named "git-daemon-export-ok".

Besides setting a web repo browser and git-server there's nothing else
specific to git.

-- 
Ismael Luceno


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Git push over git protocol for corporate environment

[Jakub Narebski]

On Thu, 1 Oct 2009, Eugene Sajine wrote:

> Thanks to everybody for prompt answers!

You are welcome!

> There is one thing I'm still missing though. Do I understand correctly that  
> if a person has an ssh access (account) to the host in internal network,  
> then this won't be enough for him to be able to push to the repo? Should we  
> still go through the hassle of managing the ssh keys for each particular  
> user who is supposed to have push access?

Yes, it is enough to push (and fetch) via SSH protocol.

Nevertheless it is better to use key-based authentication, or to be more
exact passwordless authentication, so that you are asked for ssh key
password (if there is one set for given key) only once when adding it
to ssh agent (c.f. ssh-add, ssh-agent, and keychain).  Otherwise you 
would need to give password over and over (well, I think git uses one
or two connections for fetch / push, so it shouldn't be much of an 
issue).

Also you can have SSH key shared via networked filesystem...


Conversely, on the other hand side many git management tool require to
setup only single SSH account, and distinguish between users based on 
keys they use.

> 
> I believe the answer is yes and that's why I'm leaning towards pulls and  
> pushes over git protocol. There is no solution yet which would be as  
> effective and simple to maintain. Using git protocol will not add security,  
> but it won't be worse than existing CVS or any other centralized version  
> control security model. As soon as security comes into play, then we will  
> need some other solution, but currently i didn't see anything that would be  
> easy to sell to the company.

Git protocol is anonymous and unauthenticated, so you can't (I think)
make any ACL with it.  CVS pserver was not secure, but CVS tunnelled over
SSH, or used over SSH was.


Gitosis, Gitolite, SCuMD, repo are all git management tools.  
GitHub:FI, Gitosis, Girocco (with github), InDefero are all git hosting
tools (with web interface both for repo browsing and for administration).
Gerrit is git based review board.


> Github is cool, but FI is way too expensive and very hard to sell.
> 
> Gitorious is even better!! for corporate use, i think, because of its team  
> oriented approach, but... man... I would kill for java implementation or  
> anything as simple as that!!

Gitorious is roughly GitHub equivalent, as it is also git hosting
software (a web application).

Both SCuMD and (I think) Gerrit are in Java; SCuMD is SSH server and/or
git repository management tool in early stages of development,  Gerrit
is multi-repo management web app and (mainly) review board.

> As i see It is impossible to install in   
> network without internet access, and the amount of dependencies which you  
> have install/pre-install is enormous. I read somewhere ruby on rails is fun  
> to develop with, but is a nightmare to deploy and maintain, and it seems to  
> be true. Come on, guys!! Look at the Hudson CI - one war file containing  
> everything you need, application starts from command line "java -jar  
> hudson.war" and runs on any port you specify. Time to start from download  
> to having first build is less the 10 min!!! If there are gitorious guys -  
> please, think about it and don't forget to share the profit;)!

I don't know what are tricks that Rubyists use to ease deployment, but
take a look at things such as Gems and Capistrano.

> 
> I think Cgit can be something competitive - although i failed to run it  
> yet, having some issues with build...and as all other web based stuff, you  
> should implement something in order to create and set up bare repos on the  
> server automatically (even probably edit the config file via script) to  
> avoid a mess and to avoid one guy spending his time adding and configuring  
> repos... Probably we will and up using gitweb as it at least knows to scan  
> a folder for git repos...although it also gives me troubles installing...  
> both with cgit and gitweb are conducted under cygwin, so probably this is  
> the real problem with them;)

Both cgit (which is written in C) and gitweb (which is single Perl script,
plus CSS and two images) are git web interfaces.  They do not have
features for managing repositories, nor for managing access to git 
repositories.

Girocco, which is git hosting software that http://repo.or.cz uses to
manage git repositories, uses (enhanced) gitweb for web interface.

> 
> I think that this is what is missing right now in order for git to get  
> rocket start and spread inside companies: secure and easy to maintain  
> mainline hosting.

Paraphrasing known quote: Git development community have no plans for
world domination; it would be purely accidental side-effect ;-))

Spread inside companies is not a goal; best possible tool for OSS
development is.

> 
> Probably all my frustration comes from lack of experience - so, while i  
> will continue to work on it, i would really appreciate any advice,  
> especially about experience using git not for open source and not in 3  
> person's team.

You didn't mention trying Gitosis or Gitolite, which are I think most
commonly used tool for managing git repositories (well, Gitosis is 
anyway).  Gitosis is even mentioned in "Pro Git" book (http://progit.org).

See e.g. "Hosting Git repositories, The Easy (and Secure) Way"
http://scie.nti.st/2007/11/14/hosting-git-repositories-the-easy-and-secure-way
(from http://git.or.cz/gitwiki/BlogPosts)


[cut rest of reply; please do not toppost, and remove parts of reply
you are not responding to].
-- 
Jakub Narebski
Poland

Re: Git push over git protocol for corporate environment

[Matthieu Moy]

Jakub Narebski <jnareb@gmail.com> writes:

> On Thu, 1 Oct 2009, Eugene Sajine wrote:
>
>> Thanks to everybody for prompt answers!
>
> You are welcome!
>
>> There is one thing I'm still missing though. Do I understand correctly that  
>> if a person has an ssh access (account) to the host in internal network,  
>> then this won't be enough for him to be able to push to the repo? Should we  
>> still go through the hassle of managing the ssh keys for each particular  
>> user who is supposed to have push access?
>
> Yes, it is enough to push (and fetch) via SSH protocol.

To be a bit more precise: roughly, there are two ways to manage access
to a Git repo via SSH:

* One unix user (typically called "git") managing the repository, and
  eveybody connecting to the repo via ssh://git@.... Then, if you want
  any access control within the owned repositories for this user, you
  need a key-based authentication to be able to distinguish who's
  connecting. This is what gitorious does.

* Everyone has its own unix account, and the repository is shared (via
  ACLs or simple group-based permissions, see git init --shared).
  Then, each user can choose the way he prefers for authentication,
  and if the user has an unrestricted account (i.e. can write
  ~/.ssh/authorized_keys), then it's the job of the users to manage
  this, not the one of the sysadmin.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: Git push over git protocol for corporate environment

[Michael Poole]

Eugene Sajine writes:

[snip]
> My problem is that I need the simplest, easiest and fastest solution
> from setup and maintenance point of view in a situation when we have a
> huge CVS repo with hundreds of modules (projects) in it. My current
> understanding is that we are going to pull out project by project from
> CVS and create corresponding git repos.
> So, this brings us to hundreds of git repos and over 200 hundred
> committers. In this circumstances we don’t want to manage each repo
> separately as well as we don’t want to manage each person write access
> rights to each repo.
> As I understand the best solution here is git protocol (one port only
> on dedicated server and no security as we are in trusted network) with
> read and write access configured for all repos on a dedicated server.
> What do you think I should do? How to enable push over git protocol?

How do you manage permissions now?  How would you like to manage
rights under the new system?

I am a git amateur, but I would suggest using git+ssh (git over ssh)
and use group or ACL permissions based on the SSH user account.  The
standard git-daemon does not provide authentication or authorization,
so you would have to roll your own -- but git+ssh lets you leverage
the operating system's built-in access controls.

For example, some developers might belong to group A, and others
developers belong to group B.  With standard Unix permissions, you
could grant global read but only group-A commit rights to any number
of permissions (by appropriate use of git init --shared).

I have not tried using POSIX ACLs to grant more complicated access
rights for git repositories, but setting default ACL entries on the
directory before running "git init" *should* give good results.

(Others have mentioned Gerrit.  I use that at work, and my only major
wish is that it had per-branch rather than per-project access
controls.  It is a vast improvement over the Subversion system we had
before.)

Michael Poole

Re: Git push over git protocol for corporate environment

[Shawn O. Pearce]

Michael Poole <mdpoole@troilus.org> wrote:
> (Others have mentioned Gerrit.  I use that at work, and my only major
> wish is that it had per-branch rather than per-project access
> controls.  It is a vast improvement over the Subversion system we had
> before.)

You'll be happy to hear _everyone_ is demanding per-branch controls,
I have to do it before the end of the year, maybe even before the
end of the month...

-- 
Shawn.

Re: Git push over git protocol for corporate environment

[Marius Storm-Olsen]

Shawn O. Pearce said the following on 01.10.2009 02:06:
> Michael Poole <mdpoole@troilus.org> wrote:
>> (Others have mentioned Gerrit.  I use that at work, and my only
>> major wish is that it had per-branch rather than per-project
>> access controls.  It is a vast improvement over the Subversion
>> system we had before.)
> 
> You'll be happy to hear _everyone_ is demanding per-branch
> controls, I have to do it before the end of the year, maybe even
> before the end of the month...

Ugh, any pointers on this one? Does this mean that you're planning to 
add this sort of control in git itself, or just some way to facilitate 
the setting of owner/group on individual ref files? What about packed 
refs?

--
.marius

Re: Git push over git protocol for corporate environment

[Shawn O. Pearce]

Marius Storm-Olsen <mstormo@gmail.com> wrote:
> Shawn O. Pearce said the following on 01.10.2009 02:06:
>> Michael Poole <mdpoole@troilus.org> wrote:
>>> (Others have mentioned Gerrit.  I use that at work, and my only
>>> major wish is that it had per-branch rather than per-project
>>> access controls.  It is a vast improvement over the Subversion
>>> system we had before.)
>>
>> You'll be happy to hear _everyone_ is demanding per-branch
>> controls, I have to do it before the end of the year, maybe even
>> before the end of the month...
>
> Ugh, any pointers on this one? Does this mean that you're planning to  
> add this sort of control in git itself, or just some way to facilitate  
> the setting of owner/group on individual ref files? What about packed  
> refs?

I guess you don't know how Gerrit Code Review works, or missed that
I was talking about Gerrit and not git.

Gerrit behaves like Gitosis, it owns the repositories under its care,
and (in general) nobody else is allowed to read or write to them
except the Gerrit daemon process.  That process is running JGit,
not git.git, which means I have full control over the entire code
that serves that repository.

We already have write level control to branches in that JGit has
per-ref hook support similar to what the update hook provides in git.
It doesn't actually use the update hook, its an interface API the
server implements and pushes down into the JGit library, and it has
more control over the response issued to the client, but we get the
same result.  I'm just missing a UI that allows an administrator to
configure that implementation's decision making on a per-ref basis.

We don't yet have read level control to read branches, but this
is fairly trivial to implement.  I just need an interface API
that can filter the refs before we advertise them to the client.
Given my expand refs protocal extension that I started working on
(but have not yet finished) I'd need something like that in JGit
anyway just to implement the expand refs behavior.  Teaching it to
further filter refs by who can read what is then trivial.

-- 
Shawn.