[ANNOUNCE] automated linux kernel testing results

[ANNOUNCE] automated linux kernel testing results

[Martin J. Bligh]

OK, I've finally got this to the point where I can publish it.

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html

Currently it builds and boots any mainline, -mjb, -mm kernel within
about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx.
Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32)
PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system.
The config files it uses are linked by the machine names in the column 
headers.

Andrew, you should be able to see build failures in -mm more easily now ;-)
I'll work on some automatic email triggers to you if you want them.

Thanks to all the other IBM people who've worked on the ABAT test system 
that this stuff relies on - too many to list, but especially Andy, Adam,
and Enrique, who have fixed endless bugs, and put up with my incessant 
bitching about it all not working as it should ;-)

It will do various other bits and pieces ... more profiling, more tests,
it'll do patches, etc as well. I don't want to push much volume up to
kernel.org (I just copy a small subset of the generated files right now),
but there are more runs queued with some fixup patches on top of -mm tree,
for example. It did also do -bk and -git nightly builds, they're not in 
this matrix, but I'll add them back soon.

Andrew, if you want some other test added to the mix, please let me 
know ... though theoretically we can do multi-machine tests (eg networked),
I want to stick with single-machine ones for now. There's a huge pile of
tests we intend to add ... but I thought I'd throw the general results
mechanism out there for people to look at.

Clicking on the failure ones error codes should take you to somewhere
vaguely helpful to diagnose it. Clicking on the job number just below
that takes you to the info I'm publishing right now, which should 
include perf results and profiles, etc. I'll add graphs, etc later,
comparing performance across kernels (I have them ... just not automated).

M.

Re: [ANNOUNCE] automated linux kernel testing results

[Martin J. Bligh]

--Greg KH <greg@kroah.com> wrote (on Thursday, June 02, 2005 22:51:57 -0700):

> On Thu, Jun 02, 2005 at 03:03:18PM -0700, Martin J. Bligh wrote:
>> OK, I've finally got this to the point where I can publish it.
>> 
>> http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html
>> 
>> Currently it builds and boots any mainline, -mjb, -mm kernel within
>> about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx.
>> Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32)
>> PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system.
>> The config files it uses are linked by the machine names in the column 
>> headers.
> 
> Nice, very nice, congrats to all involved.
> 
> Now, any chance you can do this on the nightly -git snapshots too? :)
> 
> And I don't see the -stable releases in there...

It does do both. I just didn't pull in all the historical data, I just
repopulated the external set with a brief snapshot (I have way more
internally, but it has some crap unpublishable benchmarks in it).
If you look at the latest rev, about 3 up it has 2.6.12-rc5-git7, and
I think I've fixed it to monitor for new ones automatically now.
I guess we'll see if it worked in the morning ;-)

M.

Re: [ANNOUNCE] automated linux kernel testing results

[Greg KH]

On Thu, Jun 02, 2005 at 03:03:18PM -0700, Martin J. Bligh wrote:
> OK, I've finally got this to the point where I can publish it.
> 
> http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html
> 
> Currently it builds and boots any mainline, -mjb, -mm kernel within
> about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx.
> Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32)
> PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system.
> The config files it uses are linked by the machine names in the column 
> headers.

Nice, very nice, congrats to all involved.

Now, any chance you can do this on the nightly -git snapshots too? :)

And I don't see the -stable releases in there...

thanks,

greg k-h

Re: [ANNOUNCE] automated linux kernel testing results

[Greg KH]

On Thu, Jun 02, 2005 at 10:45:13PM -0700, Martin J. Bligh wrote:
> 
> 
> --Greg KH <greg@kroah.com> wrote (on Thursday, June 02, 2005 22:51:57 -0700):
> 
> > On Thu, Jun 02, 2005 at 03:03:18PM -0700, Martin J. Bligh wrote:
> >> OK, I've finally got this to the point where I can publish it.
> >> 
> >> http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html
> >> 
> >> Currently it builds and boots any mainline, -mjb, -mm kernel within
> >> about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx.
> >> Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32)
> >> PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system.
> >> The config files it uses are linked by the machine names in the column 
> >> headers.
> > 
> > Nice, very nice, congrats to all involved.
> > 
> > Now, any chance you can do this on the nightly -git snapshots too? :)
> > 
> > And I don't see the -stable releases in there...
> 
> It does do both. I just didn't pull in all the historical data, I just
> repopulated the external set with a brief snapshot (I have way more
> internally, but it has some crap unpublishable benchmarks in it).
> If you look at the latest rev, about 3 up it has 2.6.12-rc5-git7, and
> I think I've fixed it to monitor for new ones automatically now.

Ah, sorry about that, you are correct, I missed it while seeing all of
the -mm releases in there :)

thanks,

greg k-h

Re: [ANNOUNCE] automated linux kernel testing results

[Denis Vlasenko]

On Friday 03 June 2005 01:03, Martin J. Bligh wrote:
> OK, I've finally got this to the point where I can publish it.
> 
> http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html
>
> Currently it builds and boots any mainline, -mjb, -mm kernel within
> about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx.
> Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32)
> PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system.
> The config files it uses are linked by the machine names in the column 
> headers.

Wow. 8]

Some (but not all) green links like [GOOD 4550] lead to 404 land. Is this intended?
--
vda

Re: [ANNOUNCE] automated linux kernel testing results

[Martin J. Bligh]

--Denis Vlasenko <vda@ilport.com.ua> wrote (on Friday, June 03, 2005 16:12:30 +0300):

> On Friday 03 June 2005 01:03, Martin J. Bligh wrote:
>> OK, I've finally got this to the point where I can publish it.
>> 
>> http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html
>> 
>> Currently it builds and boots any mainline, -mjb, -mm kernel within
>> about 15 minutes of release. runs dbench, tbench, kernbench, reaim and fsx.
>> Currently I'm using a 4x AMD64 box, a 16x NUMA-Q, 4x NUMA-Q, 32x x440 (ia32)
>> PPC64 Power 5 LPAR, PPC64 Power 4 LPAR, and PPC64 Power 4 bare metal system.
>> The config files it uses are linked by the machine names in the column 
>> headers.
> 
> Wow. 8]
> 
> Some (but not all) green links like [GOOD 4550] lead to 404 land. Is this intended?

Nope, it's buggered. Some of the new ones are not replicating out properly.
Fixing it ...

M.

Re: [ANNOUNCE] automated linux kernel testing results

[Martin J. Bligh]

> OK, I've finally got this to the point where I can publish it.
> 
> http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/regression_matrix.html

...

> Clicking on the failure ones error codes should take you to somewhere
> vaguely helpful to diagnose it. Clicking on the job number just below
> that takes you to the info I'm publishing right now, which should 
> include perf results and profiles, etc. I'll add graphs, etc later,
> comparing performance across kernels (I have them ... just not automated).

OK, there's performance graphs out there for tbench, dbench, kernbench,
and reaim now.

http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/perf_matrix.html

-mm seems to be rather sucky on kernbench recently, I'll have a drill down
into why, and try to send out some more data later on. The vertical bars
on the kernbench graph are std deviation between sets of runs (a rough
error margin).

Suggestions for improvements welcome ...

M.

Re: [ANNOUNCE] automated linux kernel testing results

[Jeff Garzik]

Martin J. Bligh wrote:
> OK, there's performance graphs out there for tbench, dbench, kernbench,
> and reaim now.
> 
> http://ftp.kernel.org/pub/linux/kernel/people/mbligh/abat/perf/perf_matrix.html


Very nice.  All this stuff is helpful, thanks much.

	Jeff