[RFC] Auto bi-sect build server idea

[RFC] Auto bi-sect build server idea

[Leon Woestenberg]

Hello all,


I typically run a from scratch build once a day, typically a
combination of {uc,eg,g}libc and targets {ixp4xx, powerpc, avr32}.

Today my avr32/uclibc build stopped with bug #1380 which has been
reported 2 months ago but never occurred to me.

When I fixed that, I then ran into bug #4065 which has been
opened/fixed/closed and reopened.

So, I think we have a regression at hand (which might be already fixed
when I send this email, but that's not the point).

Then I thought about the idea of having a build server automatically
doing an automatic bi-sect towards the commit that breaks things.

Define a few local.conf's that are known-good.
Build each of them,
if the build completes, tag the revision as "last known good".
if the build fails, tag the revision as "latest known bad", and
bi-sect, selecting a "candidate" and iterate to the breaking commit.

I'm mainly interested in a helloworld-image, I'm not much interested
in package recipes, but mostly in the toolchain/libc
stuff/rootfs/image stuff, for now.

Has this been done earlier? Ideas?

Regards,
-- 
Leon

Re: [RFC] Auto bi-sect build server idea

[Koen Kooi]

Leon Woestenberg wrote:
> Hello all,
>
>
> I typically run a from scratch build once a day, typically a
> combination of {uc,eg,g}libc and targets {ixp4xx, powerpc, avr32}.
>
> Today my avr32/uclibc build stopped with bug #1380 which has been
> reported 2 months ago but never occurred to me.
>
> When I fixed that, I then ran into bug #4065 which has been
> opened/fixed/closed and reopened.
>
> So, I think we have a regression at hand (which might be already fixed
> when I send this email, but that's not the point).
>
> Then I thought about the idea of having a build server automatically
> doing an automatic bi-sect towards the commit that breaks things.
>
> Define a few local.conf's that are known-good.
> Build each of them,

What is different between those confs? I haven't edited my local.conf in 
months and still build different configurations every day.

> if the build completes, tag the revision as "last known good".
> if the build fails, tag the revision as "latest known bad", and

Tags would be a bad idea, we can use certs with monotone, which allow a 
free-from key:value structure, so you can do

mtn cert 8759384591ada9304319403 bug4530 bad
mtn cert 3123424905903affeaf0232 bug4530 good

This way you can do multiple bisects in parallel, and you don't clutter 
the tags list (which is db global and permanent).

> bi-sect, selecting a "candidate" and iterate to the breaking commit.
>
> I'm mainly interested in a helloworld-image, I'm not much interested
> in package recipes, but mostly in the toolchain/libc
> stuff/rootfs/image stuff, for now.
>
> Has this been done earlier? Ideas?

It would be easy to extend seppuku.bbclass to add a 'bug#:bad' cert each 
time it falls over. That would catch buildproblems. For bisecting we 
would probably need a small wrapper script.

Any volunteers for extending seppuku.bbclass and/or writing such a 
wrapper script?

regards,

Koen

Re: [RFC] Auto bi-sect build server idea

[Leon Woestenberg]

Hello Koen,

On Thu, May 8, 2008 at 10:02 AM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
> > Define a few local.conf's that are known-good.
>  What is different between those confs? I haven't edited my local.conf in
> months and still build different configurations every day.
>
Ah yes, you specify stuff on the command line prior to building?

MACHINE=foobar bitbake etc.?

> > if the build completes, tag the revision as "last known good".
> > if the build fails, tag the revision as "latest known bad", and
> >
>
>  Tags would be a bad idea, we can use certs with monotone, which allow a
> free-from key:value structure, so you can do
>
I didn't mean tagging in the SCM context, just local to the build
server (i.e. local to the bisect script). Should have made this more
clear, as I don't intend to clutter the OE mtn db.

Should check if mtn supports bi-sect, otherwise cook up a script that does this.

Regards,
 --
Leon

Re: [RFC] Auto bi-sect build server idea

[Richard Purdie]

On Thu, 2008-05-08 at 12:15 +0200, Leon Woestenberg wrote:
> Hello Koen,
> 
> On Thu, May 8, 2008 at 10:02 AM, Koen Kooi <k.kooi@student.utwente.nl> wrote:
> > > Define a few local.conf's that are known-good.
> >  What is different between those confs? I haven't edited my local.conf in
> > months and still build different configurations every day.
> >
> Ah yes, you specify stuff on the command line prior to building?
> 
> MACHINE=foobar bitbake etc.?
> 
> > > if the build completes, tag the revision as "last known good".
> > > if the build fails, tag the revision as "latest known bad", and
> > >
> >
> >  Tags would be a bad idea, we can use certs with monotone, which allow a
> > free-from key:value structure, so you can do
> >
> I didn't mean tagging in the SCM context, just local to the build
> server (i.e. local to the bisect script). Should have made this more
> clear, as I don't intend to clutter the OE mtn db.
> 
> Should check if mtn supports bi-sect, otherwise cook up a script that does this.

There is another bisect which would be useful too. You know a package
foo from svn failed to compile somewhere between revisions 345 and 876.
Can bitbake tell you which commit broke things?

Both should be possible with our current technology, its just a question
of implementing them!

Cheers,

Richard