Test that every revision builds before pushing changes?

Test that every revision builds before pushing changes?

[Daniel Pittman]

G'day.

I would like to ensure that my commits are fully bisectable before I
commit them to an upstream repository, at least to the limits of an
automatic tool for testing them.

'git bisect run' is similar: it can automatically locate the breaking in
a test suite, for example, but that doesn't help me in the case of three
commits, A (good), B (bad) and C (good, fixing B).

I would much rather, in this case, use rebase to fix B so that it, too,
builds before I push the changes and pollute a public repository with a
broken changeset — and make bisect that much harder to use in future.

Regards,
        Daniel

Re: Test that every revision builds before pushing changes?

[Andreas Ericsson]

Daniel Pittman wrote:
> G'day.
> 
> I would like to ensure that my commits are fully bisectable before I
> commit them to an upstream repository, at least to the limits of an
> automatic tool for testing them.
> 
> 'git bisect run' is similar: it can automatically locate the breaking in
> a test suite, for example, but that doesn't help me in the case of three
> commits, A (good), B (bad) and C (good, fixing B).
> 
> I would much rather, in this case, use rebase to fix B so that it, too,
> builds before I push the changes and pollute a public repository with a
> broken changeset — and make bisect that much harder to use in future.
> 

You can do that, but it requires manual work too. The trick is to make
the release branch immutable on the public repository and use topic
branches with per-developer namespaces. The per-developer namespace
thing is actually important, as it leaves the freedom to rewind and
recreate topics to the developers (which shared branches do not).

The manual step comes at merge-time; Someone has to be responsible for
merging all the topics that are to be included in the release branch
and make sure it builds and passes all tests after each merge.

This workflow is a bit cumbersome. NASA uses something like this but
with an extra step for multiple peer reviews on every feature/fix for
software they send to satellites. Or so I've heard anyways.

If you're thinking of a staging area that should queue all commits
and lock the repo while testing is in progress, you need to think
again, I'm afraid, as that locks up developer time in such huge
amounts that it isn't really worth it. Without the locking, you
get the problem of trying to automate merges (which may conflict
and need a manual resolution). Although I guess a merge conflict
could result in a delayed push error, so perhaps that could work
too. If you manage to cook up a solution for this, please make a
small writeup on this list how you went about achieving it.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Test that every revision builds before pushing changes?

[Daniel Pittman]

Andreas Ericsson <ae@op5.se> writes:
> Daniel Pittman wrote:
>>
>> I would like to ensure that my commits are fully bisectable before I
>> commit them to an upstream repository, at least to the limits of an
>> automatic tool for testing them.
>>
>> 'git bisect run' is similar: it can automatically locate the breaking in
>> a test suite, for example, but that doesn't help me in the case of three
>> commits, A (good), B (bad) and C (good, fixing B).
>>
>> I would much rather, in this case, use rebase to fix B so that it, too,
>> builds before I push the changes and pollute a public repository with a
>> broken changeset — and make bisect that much harder to use in future.
>
> You can do that, but it requires manual work too. The trick is to make
> the release branch immutable on the public repository and use topic
> branches with per-developer namespaces. The per-developer namespace
> thing is actually important, as it leaves the freedom to rewind and
> recreate topics to the developers (which shared branches do not).
>
> The manual step comes at merge-time; Someone has to be responsible for
> merging all the topics that are to be included in the release branch
> and make sure it builds and passes all tests after each merge.

Ah.  You have not quite grasped what I was looking for: I was after a
tool to help automate that step, rather than a workflow around it.

For example, the responsible person for that testing could use the
hypothetical (until someone tells me where to find it):

    git test public..test make test

Which would then effectively wrap:

    for each revision between public and private:
         git checkout revision
         make test
         # report if that fails, allow fixing the commit or whatever
         # then 'git test continue' to carry on...

That turn the process from a manual one to an automated one: it runs
that command for every revision until it fails, or until they all pass.

Regards,
        Daniel

Re: Test that every revision builds before pushing changes?

[Andreas Ericsson]

Daniel Pittman wrote:
> Andreas Ericsson <ae@op5.se> writes:
>> Daniel Pittman wrote:
>>> I would like to ensure that my commits are fully bisectable before I
>>> commit them to an upstream repository, at least to the limits of an
>>> automatic tool for testing them.
>>>
>>> 'git bisect run' is similar: it can automatically locate the breaking in
>>> a test suite, for example, but that doesn't help me in the case of three
>>> commits, A (good), B (bad) and C (good, fixing B).
>>>
>>> I would much rather, in this case, use rebase to fix B so that it, too,
>>> builds before I push the changes and pollute a public repository with a
>>> broken changeset — and make bisect that much harder to use in future.
>> You can do that, but it requires manual work too. The trick is to make
>> the release branch immutable on the public repository and use topic
>> branches with per-developer namespaces. The per-developer namespace
>> thing is actually important, as it leaves the freedom to rewind and
>> recreate topics to the developers (which shared branches do not).
>>
>> The manual step comes at merge-time; Someone has to be responsible for
>> merging all the topics that are to be included in the release branch
>> and make sure it builds and passes all tests after each merge.
> 
> Ah.  You have not quite grasped what I was looking for: I was after a
> tool to help automate that step, rather than a workflow around it.
> 

Oh right. Sorry, I'm stuck in continuous-integration land where people
tend to want the server to take care of such things.

> For example, the responsible person for that testing could use the
> hypothetical (until someone tells me where to find it):
> 
>     git test public..test make test
> 
> Which would then effectively wrap:
> 
>     for each revision between public and private:
>          git checkout revision
>          make test
>          # report if that fails, allow fixing the commit or whatever
>          # then 'git test continue' to carry on...
> 
> That turn the process from a manual one to an automated one: it runs
> that command for every revision until it fails, or until they all pass.
> 

Something like this?
--%<--%<--
#!/bin/sh

git stash
revspec="$1"
shift
for rev in $(git rev-list "$revspec"); do
	git checkout $rev
	"$@" || break
done
--%<--%<--

Run it as such:
./git-test.sh public..test make test

Adding support for "continue" in there shouldn't be hard, since
you'd just replace the first rev with the first parent of the
last tested revision, although if you want to fix up a commit
in the middle, you'd have to manually rebase the other ones
onto the newly created commit (again, not very hard but I'll
leave both as an exercise to you).

It doesn't handle merges very nicely, btw, but I guess this
should be run prior to merging anyways.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Test that every revision builds before pushing changes?

[Daniel Pittman]

Andreas Ericsson <ae@op5.se> writes:
> Daniel Pittman wrote:
>> Andreas Ericsson <ae@op5.se> writes:
>>> Daniel Pittman wrote:
>>>> I would like to ensure that my commits are fully bisectable before I
>>>> commit them to an upstream repository, at least to the limits of an
>>>> automatic tool for testing them.

[...]

>>> The manual step comes at merge-time; Someone has to be responsible for
>>> merging all the topics that are to be included in the release branch
>>> and make sure it builds and passes all tests after each merge.
>>
>> Ah.  You have not quite grasped what I was looking for: I was after a
>> tool to help automate that step, rather than a workflow around it.
>
> Oh right. Sorry, I'm stuck in continuous-integration land where people
> tend to want the server to take care of such things.

We have that also; I am primarily motivated by avoiding trivial breakage
in the CI server, as well as bisection.

>> For example, the responsible person for that testing could use the
>> hypothetical (until someone tells me where to find it):
>>
>>     git test public..test make test

[...]

> Something like this?
> --%<--%<--
> #!/bin/sh
>
> git stash
> revspec="$1"
> shift
> for rev in $(git rev-list "$revspec"); do
> 	git checkout $rev
> 	"$@" || break
> done
> --%<--%<--
>
> Run it as such:
> ./git-test.sh public..test make test

Thank you, that points me in the right direction, and I can obviously
season the rest of it to taste — reverse that revision list, for
example.

Thank you also to Wincent Colaiuta, who provided a similar script albiet
with significantly more detail.

[...]

> It doesn't handle merges very nicely, btw, but I guess this should be
> run prior to merging anyways.

Once I have the framework I can quietly work my way through fixing nasty
issues one way or another.  Thanks.

Regards,
        Daniel

Re: Test that every revision builds before pushing changes?

[Jeff King]

On Thu, Mar 26, 2009 at 08:10:52PM +1100, Daniel Pittman wrote:

> For example, the responsible person for that testing could use the
> hypothetical (until someone tells me where to find it):
> 
>     git test public..test make test
> 
> Which would then effectively wrap:
> 
>     for each revision between public and private:
>          git checkout revision
>          make test
>          # report if that fails, allow fixing the commit or whatever
>          # then 'git test continue' to carry on...
> 
> That turn the process from a manual one to an automated one: it runs
> that command for every revision until it fails, or until they all pass.

I don't think such a script exists. There are continuous integration
systems that support git, but I don't know if they actually check every
commit.

-Peff

Re: Test that every revision builds before pushing changes?

[Wincent Colaiuta]

El 26/3/2009, a las 10:49, Jeff King escribió:

> On Thu, Mar 26, 2009 at 08:10:52PM +1100, Daniel Pittman wrote:
>
>> For example, the responsible person for that testing could use the
>> hypothetical (until someone tells me where to find it):
>>
>>    git test public..test make test
>>
>> Which would then effectively wrap:
>>
>>    for each revision between public and private:
>>         git checkout revision
>>         make test
>>         # report if that fails, allow fixing the commit or whatever
>>         # then 'git test continue' to carry on...
>>
>> That turn the process from a manual one to an automated one: it runs
>> that command for every revision until it fails, or until they all  
>> pass.
>
> I don't think such a script exists. There are continuous integration
> systems that support git, but I don't know if they actually check  
> every
> commit.

It shouldn't be too hard to put together a script that does exactly  
what you want.

A couple of years ago I whipped up a simple script (see below) to test  
all the commits on a topic branch before submitting them upstream.

It's designed for that specific case, and I'm neither a shell nor a  
plumbing expert, but it worked for my simple workflow at least.

Cheers,
Wincent

die () {
   echo "$1"
   exit 1
}

git diff --quiet || die "Unstaged changes; won't start"
git diff --cached --quiet || die "Staged changes; won't start"

# parse HEAD (something like "ref: refs/heads/my_local_branch") to get  
remote and merge
GIT_DIR=$(git rev-parse --git-dir)
TOPIC="$GIT_DIR/HEAD"
test -f "$TOPIC" || die "No HEAD found at '$TOPIC'"
BRANCH_REF=$(< "$TOPIC")
if [ "${BRANCH_REF%%: */*}" != "ref" ]; then
   die "expected HEAD '$BRANCH_REF' to be of the form 'ref: .../...'"
fi
BRANCH=${BRANCH_REF##*/}
REMOTE=$(git config branch.$BRANCH.remote) || die "failed to get  
branch.$BRANCH.remote"
MERGE=$(git config branch.$BRANCH.merge) || die "failed to get branch. 
$BRANCH.merge"
MERGE=${MERGE##*/}

# remember which branch we were on prior to starting
trap 'git checkout $BRANCH' exit

# will check all commits in topic branch not present in origin
# (note: may be more appropriate to use "git merge-base" here)
echo "On branch $BRANCH"
echo "Commits to be checked:"
git rev-list --pretty=oneline HEAD ^$REMOTE/$MERGE
for COMMIT in $(git rev-list HEAD ^$REMOTE/$MERGE); do
   echo "Checking $COMMIT"
   git checkout $COMMIT
   make clean
   make test || die "Commit $COMMIT is bad (make test failed)"
done

echo "All revisions passed!"