Solve continuous integration (pending head / commit queue) problem using git

Solve continuous integration (pending head / commit queue) problem using git

[Jan Koprowski]

Hi !

  This is my first mail on the list so hello everyone :)
  I'am currently write my MA. Part of my thesis is looking for some
way to stay "master" stable.
  First assumption is simple: Control version system make all dirty
job - programmers can't "stop" or "break" all procedure and work in
natural ways. We can't assume that programmer use some custom hooks on
their side or add some addition parameters to git commands.
  Second assumption  is that programmer can't compile source code one
their machine for some reasons. Only way to compile is some other way
- for example use CruiseControl or Hudson or some CI tools. This isn't
really matter in my question now.
  Third assumption: Code cloned from repository is stable = compiling
well and pass all tests.
  Forth assumption: compiling is testing are very very very fast.

What I meen "natural way of work" by programmer. They *clone*
repository (or *pull* changes) from some central repository. Then do
some stuff with code on their working copy and *push* their changes.
Ok but - their don't know is code working well. And this is a problem.
I know there is some options: XP pair programming, automated static
code analysis, code review and others ... but this is not the point.
In my "configuration" there are some frequently scheduled build of
system automated by some tool. Tool just get all stuff from repo,
compile all stuff, running tests (if compiling successes) and if
sending e-mail.
But SCM should "somehow" distinct unstable commits from stable commits
(after compiling).

Now. My idea. There is some revision tagged as "stable". *Clone* and
*pull* operations is somehow "overloaded" from server side and always!
return last revision tagged as stable. After compiling external tool
just move tag to another revision which pass all tests. Of course
there is some additional parameter (for example --last or --unstable)
which can clone fine way of repository.

Two questions.
1) Maybe I try to invent the wheel again. Is there any way to take the
effect without overloading standard git behaviours.
2) If not how overload git behaviors on git "server side" repo?

Thanks in advance!
-- 
><> Jan Koprowski [696775174] GSM

Re: Solve continuous integration (pending head / commit queue) problem using git

[Avery Pennarun]

On Fri, Feb 12, 2010 at 11:37 AM, Jan Koprowski <jan.koprowski@gmail.com> wrote:
> Now. My idea. There is some revision tagged as "stable". *Clone* and
> *pull* operations is somehow "overloaded" from server side and always!
> return last revision tagged as stable. After compiling external tool
> just move tag to another revision which pass all tests. Of course
> there is some additional parameter (for example --last or --unstable)
> which can clone fine way of repository.
>
> Two questions.
> 1) Maybe I try to invent the wheel again. Is there any way to take the
> effect without overloading standard git behaviours.
> 2) If not how overload git behaviors on git "server side" repo?

In general, code that lies to you about what's the most revision is
evil.  Sometimes you *do* want to fetch that revision it's lying to
you and saying doesn't exist, precisely because you'd like to help fix
it before integration.

What you really want is:

- nobody can push to the "integration branch" except the "integration manager"

- the "integration manager" should be a computer program, so that you
can have "continuous integration"

This isn't actually that hard.  Give each user their own repository;
no user can write to any other user's repository.  (This is the
default setup on github.com, for example.)  Alternatively, just tell
people to never, ever push to the master branch by themselves.  People
are easily capable of following rules like that unless they're
actively trying to screw you.

Then set up something like gitbuilder
(http://github.com/apenwarr/gitbuilder) (Full disclosure: I wrote it)
to build *all* the branches from *all* the users.  This sounds like it
would create exponential work for the build machine, but it doesn't,
since most users will have mostly the same commits anyway.

When gitbuilder tags a particular commit as having built and passed
all tests, then it becomes a candidate for merging into the
integration branch.  Write a little script that goes through candidate
branches, checks their gitbuilder status, and if they've passed,
pushes them into the integration branch.  The push will only succeed
if the integration branch can be fast-forwarded to match the branch
you're trying to push; if you can't, it'll be rejected, which is what
you want, since merging (even conflict-free merging) might break
tests.

That mechanism works pretty well at my company, with one exception: we
didn't bother with an automatic tool that merges into master.  We
prefer to have a release manager do that.

Have fun,

Avery

Re: Solve continuous integration (pending head / commit queue) problem using git

[Jan Koprowski]

On Fri, Feb 12, 2010 at 6:42 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On Fri, Feb 12, 2010 at 11:37 AM, Jan Koprowski <jan.koprowski@gmail.com> wrote:
>> Now. My idea. There is some revision tagged as "stable". *Clone* and
>> *pull* operations is somehow "overloaded" from server side and always!
>> return last revision tagged as stable. After compiling external tool
>> just move tag to another revision which pass all tests. Of course
>> there is some additional parameter (for example --last or --unstable)
>> which can clone fine way of repository.
>>
>> Two questions.
>> 1) Maybe I try to invent the wheel again. Is there any way to take the
>> effect without overloading standard git behaviours.
>> 2) If not how overload git behaviors on git "server side" repo?
>
> In general, code that lies to you about what's the most revision is
> evil.  Sometimes you *do* want to fetch that revision it's lying to
> you and saying doesn't exist, precisely because you'd like to help fix
> it before integration.
>
> What you really want is:
>
> - nobody can push to the "integration branch" except the "integration manager"
>
> - the "integration manager" should be a computer program, so that you
> can have "continuous integration"
>
> This isn't actually that hard.  Give each user their own repository;
> no user can write to any other user's repository.  (This is the
> default setup on github.com, for example.)  Alternatively, just tell
> people to never, ever push to the master branch by themselves.  People
> are easily capable of following rules like that unless they're
> actively trying to screw you.
>
> Then set up something like gitbuilder
> (http://github.com/apenwarr/gitbuilder) (Full disclosure: I wrote it)
> to build *all* the branches from *all* the users.  This sounds like it
> would create exponential work for the build machine, but it doesn't,
> since most users will have mostly the same commits anyway.
>
> When gitbuilder tags a particular commit as having built and passed
> all tests, then it becomes a candidate for merging into the
> integration branch.  Write a little script that goes through candidate
> branches, checks their gitbuilder status, and if they've passed,
> pushes them into the integration branch.  The push will only succeed
> if the integration branch can be fast-forwarded to match the branch
> you're trying to push; if you can't, it'll be rejected, which is what
> you want, since merging (even conflict-free merging) might break
> tests.
>
> That mechanism works pretty well at my company, with one exception: we
> didn't bother with an automatic tool that merges into master.  We
> prefer to have a release manager do that.
>
> Have fun,
>
> Avery
>

Probably I don't have a problem (or it is a lateness). Because only
tagging as stable and making two compile loops: one per management
always compiling stable tag and second compiling latest repo... And
that is all :D

-- 
><> Jan Koprowski [696775174] GSM

Re: Solve continuous integration (pending head / commit queue) problem using git

[Jan Koprowski]

On Fri, Feb 12, 2010 at 7:07 PM, Jan Koprowski <jan.koprowski@gmail.com> wrote:
> On Fri, Feb 12, 2010 at 6:42 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
>> On Fri, Feb 12, 2010 at 11:37 AM, Jan Koprowski <jan.koprowski@gmail.com> wrote:
>>> Now. My idea. There is some revision tagged as "stable". *Clone* and
>>> *pull* operations is somehow "overloaded" from server side and always!
>>> return last revision tagged as stable. After compiling external tool
>>> just move tag to another revision which pass all tests. Of course
>>> there is some additional parameter (for example --last or --unstable)
>>> which can clone fine way of repository.
>>>
>>> Two questions.
>>> 1) Maybe I try to invent the wheel again. Is there any way to take the
>>> effect without overloading standard git behaviours.
>>> 2) If not how overload git behaviors on git "server side" repo?
>>
>> In general, code that lies to you about what's the most revision is
>> evil.  Sometimes you *do* want to fetch that revision it's lying to
>> you and saying doesn't exist, precisely because you'd like to help fix
>> it before integration.
>>
>> What you really want is:
>>
>> - nobody can push to the "integration branch" except the "integration manager"
>>
>> - the "integration manager" should be a computer program, so that you
>> can have "continuous integration"
>>
>> This isn't actually that hard.  Give each user their own repository;
>> no user can write to any other user's repository.  (This is the
>> default setup on github.com, for example.)  Alternatively, just tell
>> people to never, ever push to the master branch by themselves.  People
>> are easily capable of following rules like that unless they're
>> actively trying to screw you.
>>
>> Then set up something like gitbuilder
>> (http://github.com/apenwarr/gitbuilder) (Full disclosure: I wrote it)
>> to build *all* the branches from *all* the users.  This sounds like it
>> would create exponential work for the build machine, but it doesn't,
>> since most users will have mostly the same commits anyway.
>>
>> When gitbuilder tags a particular commit as having built and passed
>> all tests, then it becomes a candidate for merging into the
>> integration branch.  Write a little script that goes through candidate
>> branches, checks their gitbuilder status, and if they've passed,
>> pushes them into the integration branch.  The push will only succeed
>> if the integration branch can be fast-forwarded to match the branch
>> you're trying to push; if you can't, it'll be rejected, which is what
>> you want, since merging (even conflict-free merging) might break
>> tests.
>>
>> That mechanism works pretty well at my company, with one exception: we
>> didn't bother with an automatic tool that merges into master.  We
>> prefer to have a release manager do that.
>>
>> Have fun,
>>
>> Avery
>>
>
> Probably I don't have a problem (or it is a lateness). Because only
> tagging as stable and making two compile loops: one per management
> always compiling stable tag and second compiling latest repo... And
> that is all :D
>
> --
>><> Jan Koprowski [696775174] GSM
>

Here is a thing :)
After I install Hudson CI and equip them of Git plugin I saw two ways
supported by Hudson:
1) Compile specific branch of code which suggest merging stable
changes to this branch
2) Merging witch "something" before building and rollback changes
after which suggest some unstable "branch" or "repository" where all
programmers commit and changes from are "merged" before build to the
stable branch. I don't check details because I choose first option.

Sad thing there is no support for tagging in Git plugin.


-- 
><> Jan Koprowski [696775174] GSM

Re: Solve continuous integration (pending head / commit queue) problem using git

[Daniel Barkalow]

On Fri, 12 Feb 2010, Avery Pennarun wrote:

> On Fri, Feb 12, 2010 at 11:37 AM, Jan Koprowski <jan.koprowski@gmail.com> wrote:
> > Now. My idea. There is some revision tagged as "stable". *Clone* and
> > *pull* operations is somehow "overloaded" from server side and always!
> > return last revision tagged as stable. After compiling external tool
> > just move tag to another revision which pass all tests. Of course
> > there is some additional parameter (for example --last or --unstable)
> > which can clone fine way of repository.
> >
> > Two questions.
> > 1) Maybe I try to invent the wheel again. Is there any way to take the
> > effect without overloading standard git behaviours.
> > 2) If not how overload git behaviors on git "server side" repo?
> 
> In general, code that lies to you about what's the most revision is
> evil.  Sometimes you *do* want to fetch that revision it's lying to
> you and saying doesn't exist, precisely because you'd like to help fix
> it before integration.

I think a more suitable detail here would be to have the remote system 
respond to pushes by stating that it's taking your push request under 
advisement, but cannot give an immediate verdict for that request (and it 
may want to let you know that it's updated a different ref of its choice 
that you didn't intentionally request).

$ git push
   f99642a..e70de97  HEAD -> master (proposed, not updated)

$ git log --oneline origin/master
f99642a Original commit

(wait for external signal, like getting a confirmation email)

$ git fetch
   f99642a..e70de97  maaster    -> origin/master

$ git log --oneline origin/master
f99642a Your commit

I think the only thing that would be needed would be a way for the remote 
server to report that it's not updating the ref, but it is planning to act 
on your request, so that your local git can give a non-error without 
updating the remote branch inappropriately. (Presumably, the server would 
have used a pre-update hook to give this response, which would have 
enqueued the request in the CI system; when the CI system likes a change, 
it can push and the hook would detect that it's actually the CI system and 
let the update happen).

	-Daniel
*This .sig left intentionally blank*