Re: lmbench results for 2.4 and 2.5 -- updated results

[Larry McVoy <lm@bitmover.com>]

In general, LMbench optimizes for fast results over exactness.  You can
definitely get more accurate results by doing longer runs.  My view
at the time of writing it was that I was looking for the broad stroke
results because I was trying to measure differences between various
operating systems.  There was more than enough to show so the results
didn't need to be precise, getting people to run the benchmark and
report results was more important.

If people are doing release runs to see if there are regressions, I
think that setting ENOUGH up to something longer is a good idea.
If there is enough interest, I could spend some time on this and
try and make a more accurate way to get results.  Let me know.

On Tue, Mar 25, 2003 at 10:23:50AM -0800, Martin J. Bligh wrote:
> > work ~/LMbench2/bin/i686-pc-linux-gnu ENOUGH=1000000 time bw_pipe
> > Pipe bandwidth: 655.37 MB/sec
> > real    0m23.411s
> > user    0m0.480s
> > sys     0m1.180s
> > 
> > work ~/LMbench2/bin/i686-pc-linux-gnu time bw_pipe
> > Pipe bandwidth: 809.81 MB/sec
> > 
> > real    0m2.821s
> > user    0m0.480s
> > sys     0m1.180s
> 
> OK, is a bit more stable now ... before:
> 
> Process fork+exit: 294.4118 microseconds
> Process fork+exit: 279.1500 microseconds
> Process fork+exit: 280.0000 microseconds
> Process fork+exit: 280.0000 microseconds
> Process fork+exit: 277.2222 microseconds
> Process fork+exit: 286.0000 microseconds
> Process fork+exit: 277.6231 microseconds
> Process fork+exit: 307.1176 microseconds
> Process fork+exit: 295.4706 microseconds
> Process fork+exit: 294.3529 microseconds
> 
> after:
> 
> Process fork+exit: 298.4124 microseconds
> Process fork+exit: 298.6746 microseconds
> Process fork+exit: 297.7784 microseconds
> Process fork+exit: 294.8297 microseconds
> Process fork+exit: 299.6249 microseconds
> Process fork+exit: 297.6771 microseconds
> Process fork+exit: 297.9801 microseconds
> Process fork+exit: 293.1421 microseconds
> Process fork+exit: 281.9868 microseconds
> 
> I can probably butcher that around by taking a few derived medians and
> averages to get pretty consistent numbers out of it (std dev < 1% for 99%
> of the time). Though 10 runs with ENOUGH=1000000 is kinda slow for all
> tests, so I probably won't be able to do this by default for every version.
> If there are any more suggestions on added stability, I'd love to hear them.
> 
> Is cool to have something big enough to profile too ;-)
> 
> Thanks very much,
> 
> M.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm