Centralized processes in git

Centralized processes in git

[Robert Boone]

Hello,
     My company is looking at git to replace subversion as our scm.  
We have a small team of developers which need to publish to a  
centralized repository. One thing we would like is to have an  
automated bug fix merge ability. So if I make a bug fix to master we  
want that fix to be merged into other branches that we set. If there  
is a conflict we want that person  to be notified by email or some  
other way. I would like to know if anyone on the list has a good way  
to implement this?

Re: Centralized processes in git

[Jan Hudec]

[-- Attachment #1: Type: text/plain, Size: 2327 bytes --]

On Mon, Aug 20, 2007 at 13:30:47 -0500, Robert Boone wrote:
> Hello,
>     My company is looking at git to replace subversion as our scm. We have 
> a small team of developers which need to publish to a centralized 
> repository. One thing we would like is to have an automated bug fix merge 
> ability. So if I make a bug fix to master we want that fix to be merged 
> into other branches that we set. If there is a conflict we want that person 
>  to be notified by email or some other way. I would like to know if anyone 
> on the list has a good way to implement this?

Normal way of pushing changes to the central repo is over the ssh protocol,
which invokes git (specifically git-receive-pack) on the server and that in
turn invokes pre-receive, update, post-receive and post-update hooks. The
first two can prevent the refs to be updated, the later two are for arbitrary
post-processing.

To do the automatic merging like you describe, you'd probably put up
a script to run from post-receive and start whatever merges you want to do.

That script gets a list of updated refs along with new and old version names
on standard input, so you just look for your mainline name (refs/heads/master
probably) and if you find, run the appropriate merges. You can of course
program whatever notifications you want in that script.

Commit would undo any changes pushed between checking out a branch and
commiting, so you should probably do the merges on a separate repository,
that will only be managed by the scripts. You can use the alternates
mechanism to avoid duplicating the data if that repository will be on the
central server.

I'd recommend the hook script to write the things to just write the
instructions what to merge somewhere (into a named pipe, probably -- or
socket, which could be extended to separate server) and process that with
a continually running "daemon" script, to avoid synchronization issues.

In git distribution is a pair of scripts in contrib/continuous directory,
that use the notifying hook + daemon approach to run tests against each
commit, so you can use that for inspiration. It's in perl and being universal
complicates it quite a bit though; you can get a lot shorter with shell and
everything hardcoded.

-- 
						 Jan 'Bulb' Hudec <bulb@ucw.cz>

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Centralized processes in git

[Steven Grimm]

Jan Hudec wrote:
> Commit would undo any changes pushed between checking out a branch and
> commiting, so you should probably do the merges on a separate repository,
> that will only be managed by the scripts. You can use the alternates
> mechanism to avoid duplicating the data if that repository will be on the
> central server.
>   

That makes this kind of operation ten times more complicated than it 
ought to be, IMO.

I wonder if it makes sense to expose a repository locking mechanism for 
this kind of application. The builtin git commands would test for the 
lock and block (waiting up to some configurable timeout) until it went 
away, but wouldn't necessarily ever actually lock things themselves. Or 
maybe a shared/exclusive lock (aka an rwlock) would be appropriate here; 
the repository-altering commands would grab a shared lock.

A lock-and-block primitive eliminates the need for a separate work queue 
manager for stuff like this: you just make sure you exclusive-lock the 
repo before you start your postprocessing (and make sure your 
postprocessing handles the case where another commit landed before you 
got launched, of course). Then you know that nothing else will screw 
with the repo while you're working, and that your execution will be 
serialized. If you don't need serialized operation like that, you just 
never grab the exclusive lock and things continue to work as today.

Stupid idea? The wrinkle, of course, is that you need to run git 
commands from within your script, so *those* can't block. I can think of 
a few easy ways around that, though, e.g., use an environment variable 
to identify yourself as the holder of the lock, perhaps by putting your 
PID in the lockfile and setting the variable to your PID.

-Steve

Re: Centralized processes in git

[Steven Grimm]

Steven Grimm wrote:
> the repository-altering commands would grab a shared lock.

I guess that's a bit confusing -- what I mean to say is that the 
repository-altering commands that today do not lock the repository would 
grab a shared lock such that, in the absence of something else grabbing 
the exclusive lock, they will continue to effectively not lock the 
repository. IOW, by grabbing the shared lock they will continue to 
exhibit their current behavior under normal conditions.

-Steve

Re: Centralized processes in git

[Jeff King]

On Tue, Aug 21, 2007 at 08:36:40AM +0800, Steven Grimm wrote:

> I wonder if it makes sense to expose a repository locking mechanism for this 
> kind of application. The builtin git commands would test for the lock and 
> block (waiting up to some configurable timeout) until it went away, but 
> wouldn't necessarily ever actually lock things themselves. Or maybe a 
> shared/exclusive lock (aka an rwlock) would be appropriate here; the 
> repository-altering commands would grab a shared lock.

For this use case, I don't think you need to touch the core git commands
at all. Since we're just talking about a repo that people are pushing
into, why not just grab the lock in a hook before accepting the push?

That serializes the push, but each push can do arbitrary work while
holding the lock.

-Peff