Re: [BENCHMARK] Corrected gcc3.2 v gcc2.95.3 contest results

Con Kolivas
2002-09-23T14:19:40Z

Quoting "Richard B. Johnson" <root@chaos.analogic.com>:

> On Mon, 23 Sep 2002, Ryan Anderson wrote:
>
> > On Mon, Sep 23, 2002 at 08:30:21PM +1000, Con Kolivas wrote:
> > > Quoting Ingo Molnar <mingo@elte.hu>:
> > > > On Mon, 23 Sep 2002, Con Kolivas wrote:
> > > >
> > > > how many times are you running each test? You should run them at
> least
> > > > twice (ideally 3 times at least), to establish some sort of
> statistical
> > > > noise measure. Especially IO benchmarks tend to fluctuate very
> heavily
> > > > depending on various things - they are also very dependent on the
> initial
> > > > state - ie. how the pagecache happens to lay out, etc. Ie. a
> meaningful
> > > > measurement result would be something like:
> > >
> > > Yes you make a very valid point and something I've been stewing over
> privately
> > > for some time. contest runs benchmarks in a fixed order with a "priming"
> compile
> > > to try and get pagecaches etc back to some sort of baseline (I've been
> trying
> > > hard to make the results accurate and repeatable).
> >
> > Well, run contest once, discard the results. Run it 3 more times, and
> > you should have started the second, third and fourth runs with similar
> initial conditions.
> >
> > Or you could run the contest 3 times, rebooting between each run....
> > (automating that is a little harder, of course.)
> >
> > IANAS, however.
> >
>
> (1) Obtain statistics from a number of runs.
> (2) Throw away the smallest and largest.
> (3) Average whatever remains.
>
> This works for many "real-world" things because it removes noise-spikes
> that could unfairly poison the average.

That is the system I was considering. I just need to run enough benchmarks to
make this worthwhile though. That means about 5 for each it seems - which may
take me a while. A basic mean will suffice for a measure of central tendency. I
also need to quote some measure of variability. Standard deviation?

Con