Multiple clients accessing git over NFS

Multiple clients accessing git over NFS

[Khawaja Shams]

  Is it a recommended practice to share a repository over NFS, where
multiple clients can be pushing changes simultaneously?  In our
production environment, we have a Git repository setup behind
git-http-backend. We would like to place multiple Apache servers
behind a load balancer to maximize availability and performance.
Before we proceed, we wanted to check to see if this practice has a
potential to cause repository corruption. If there are other ways
others have solved this problem, we would be very interested in
learning about those as well. Thank you.

Re: Multiple clients accessing git over NFS

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 1243 bytes --]


Khawaja Shams <kshams@usc.edu> writes:

> Is it a recommended practice to share a repository over NFS, where
> multiple clients can be pushing changes simultaneously?  In our
> production environment, we have a Git repository setup behind
> git-http-backend. We would like to place multiple Apache servers
> behind a load balancer to maximize availability and performance.
> Before we proceed, we wanted to check to see if this practice has a
> potential to cause repository corruption. If there are other ways
> others have solved this problem, we would be very interested in
> learning about those as well. Thank you.

NFS locking has historically been problematic, and my impression is that
most people avoid it.  Perhaps it's ok on Solaris, but without serious
testing, I'd be worried.

Can you explain what you have set up, and what your performance
situation is, and why you think adding a second or third apache over NFS
will help?  How many users?  How many pushes/day?

One option is to have a multi-core box with tons of RAM running apache;
I've done that for trac (8 core, 16G, RAID5) because trac/python is so
piggy, and buying a $3K box was cheaper than making trac go faster.
That doesn't get you into remote FS locking issues.

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: Multiple clients accessing git over NFS

[Khawaja Shams]

Hi Greg,
   Thank you for the insightful response. We have multiple automated
clients pushing and pulling changes from git as events occur. We have
not hit any real performance issues just yet. Our main goal is to
improve the availability of the repository in case the box running the
apache server has an outage during a mission critical period. Any
other ideas on how to accomplish this? From your remarks, it sounds
like putting the git repository on NFS, even with a single client, can
be problematic due to the locking issues. Is that what you meant?

   I am still interested in knowing if git can handle multiple
simultaneous pushes on the same repository without encountering
corruption issues. Thank you.

Re: Multiple clients accessing git over NFS

[Jonathan Nieder]

Khawaja Shams wrote:

>    I am still interested in knowing if git can handle multiple
> simultaneous pushes on the same repository without encountering
> corruption issues.

Yes, concurrent attempts to update a branch are serialized.  (But
please don't ask me to answer about NFS semantics.  See

http://stackoverflow.com/questions/750765/concurrency-in-a-git-repo-on-a-network-shared-folder

for some notes.)  See the note about fast-fowards in the git push
manual for how integrity is preserved.

After reading that, you might wonder: if there are many, many clients
pushing to the same branch, how is starvation avoided?  Good question!
It isn't.  If you have so many clients wanting to push to a single
branch, I would suggest having a single person or a few people
maintaining it, pulling from others.  Life will be better for many
reasons, especially quality control.

Hope that helps.
Jonathan

Re: Multiple clients accessing git over NFS

[Jan Hudec]

On Sun, Nov 14, 2010 at 15:42:29 -0800, Khawaja Shams wrote:
> Hi Greg,
>    Thank you for the insightful response. We have multiple automated
> clients pushing and pulling changes from git as events occur. We have
> not hit any real performance issues just yet. Our main goal is to
> improve the availability of the repository in case the box running the
> apache server has an outage during a mission critical period.

If you are out for availability, NFS isn't an answer, because the NFS server
remains a single point of failure. There are distributed filesystems
(Gluster, Lustre etc.) that can provide redundancy of storage nodes too or
you could have shared storage array with appropriate filesystem (GlobalFS,
OCFS2, etc.), but that requires special hardware. These will probably give
you better performance too -- git network protocol is optimized to send
minimal data, but that often means a lot more needs to be read from the disk.

I don't have personal experience with them though, so I can't give you more
specific recommendation.

-- 
						 Jan 'Bulb' Hudec <bulb@ucw.cz>

Re: Multiple clients accessing git over NFS

[Drew Northup]

On Mon, 2010-11-15 at 20:56 +0100, Jan Hudec wrote:
> On Sun, Nov 14, 2010 at 15:42:29 -0800, Khawaja Shams wrote:
> > Hi Greg,
> >    Thank you for the insightful response. We have multiple automated
> > clients pushing and pulling changes from git as events occur. We have
> > not hit any real performance issues just yet. Our main goal is to
> > improve the availability of the repository in case the box running the
> > apache server has an outage during a mission critical period.
> 
> If you are out for availability, NFS isn't an answer, because the NFS server
> remains a single point of failure. There are distributed filesystems
> (Gluster, Lustre etc.) that can provide redundancy of storage nodes too or
> you could have shared storage array with appropriate filesystem (GlobalFS,
> OCFS2, etc.), but that requires special hardware. These will probably give
> you better performance too -- git network protocol is optimized to send
> minimal data, but that often means a lot more needs to be read from the disk.
> 
> I don't have personal experience with them though, so I can't give you more
> specific recommendation.

Khawaja,
I haven't tried setting a server up with it yet, but perhaps DRDB
mirrored devices may be of use? At that point then you have a way of
making all of your HTTPd instances "see" the same filesystem (and will
have notification options for when they do not). It probably isn't
perfect, but may be worth looking into if your SAN cannot provide
downtime-less NFS. As an added benefit, there is no longer a requirement
that all of your front-ends be co-located (physically or logically).

-- 
-Drew Northup N1XIM
   AKA RvnPhnx on OPN
________________________________________________
"As opposed to vegetable or mineral error?"
-John Pescatore, SANS NewsBites Vol. 12 Num. 59

Re: Multiple clients accessing git over NFS

[Greg Troxel]

[-- Attachment #1: Type: text/plain, Size: 699 bytes --]


If only one computer is accessing the repository, then failure to lock
may be ok.  But you'll still need atomic rename etc. to work.

I may be overly conservative, but I would not (and do not) allow anyone
to access a repository (cvs, svn, git, whatever) over NFS, ever.

My expectation is that multiple git processes on one machine with a repo
on local disk works fine.  If not it's a bug.  When you add a remote FS
you have to wonder if the unix filesystem sematics are preserved.

Another approach would be cloned repositories that constantly pull from
the main one or each other, and have people use those.  That will get
you delayed merge conflicts, but they might be useful as RO replicas.



[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: Multiple clients accessing git over NFS

[J. Bruce Fields]

On Sun, Nov 14, 2010 at 06:11:41PM -0500, Greg Troxel wrote:
> 
> Khawaja Shams <kshams@usc.edu> writes:
> 
> > Is it a recommended practice to share a repository over NFS, where
> > multiple clients can be pushing changes simultaneously?  In our
> > production environment, we have a Git repository setup behind
> > git-http-backend. We would like to place multiple Apache servers
> > behind a load balancer to maximize availability and performance.
> > Before we proceed, we wanted to check to see if this practice has a
> > potential to cause repository corruption. If there are other ways
> > others have solved this problem, we would be very interested in
> > learning about those as well. Thank you.
> 
> NFS locking has historically been problematic, and my impression is that
> most people avoid it.  Perhaps it's ok on Solaris, but without serious
> testing, I'd be worried.

Does git actually do file locking when people push to a bare repo?

If all it needs is for rename and/or O_EXCL to be atomic--that should be
fine over NFS.

--b.

> 
> Can you explain what you have set up, and what your performance
> situation is, and why you think adding a second or third apache over NFS
> will help?  How many users?  How many pushes/day?
> 
> One option is to have a multi-core box with tons of RAM running apache;
> I've done that for trac (8 core, 16G, RAID5) because trac/python is so
> piggy, and buying a $3K box was cheaper than making trac go faster.
> That doesn't get you into remote FS locking issues.

Re: Multiple clients accessing git over NFS

[Sitaram Chamarty]

On Mon, Nov 15, 2010 at 2:54 AM, Khawaja Shams <kshams@usc.edu> wrote:
>   Is it a recommended practice to share a repository over NFS, where
> multiple clients can be pushing changes simultaneously?  In our

http://permalink.gmane.org/gmane.comp.version-control.git/122670

may be useful...

> production environment, we have a Git repository setup behind
> git-http-backend. We would like to place multiple Apache servers
> behind a load balancer to maximize availability and performance.
> Before we proceed, we wanted to check to see if this practice has a
> potential to cause repository corruption. If there are other ways
> others have solved this problem, we would be very interested in
> learning about those as well. Thank you.
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Sitaram

Re: Multiple clients accessing git over NFS

[Alex]

Khawaja Shams <kshams <at> usc.edu> writes:

> 
>   Is it a recommended practice to share a repository over NFS, where
> multiple clients can be pushing changes simultaneously?  In our
> production environment, we have a Git repository setup behind
> git-http-backend. We would like to place multiple Apache servers
> behind a load balancer to maximize availability and performance.
> Before we proceed, we wanted to check to see if this practice has a
> potential to cause repository corruption. If there are other ways
> others have solved this problem, we would be very interested in
> learning about those as well. Thank you.
> 


Others have commented on the git aspects of this, but FYI there is a handy
program here: http://www.unixcoding.org/NFSCoding#NFS_Cache_Tester
that tests aspects of your NFS implementation. (Sadly the one we have
at work is crap, or at least it was last time I ran the program).

Alex