Re: Linux Gazette benchmark Reiser 4

[Hans Reiser <reiser@namesys.com>]

Pysiak Satriani wrote:

>Hello Edward,
>
>Sunday, January 8, 2006, 11:07:46 PM, you wrote:
>  
>
>>Let's consider this important aspect of benchmarking more carefully.
>>So there is an interesting question: how much should be a difference
>>in order to approve that some fs really wins at this statistics? Is
>>there any guarantee you won't get, say, 0.05 and 0.02 after next run?
>>Sorry, but I didn't find any answer in Justin's notes, NOTE5 (Tests
>>Performed) says that questionable tests were re-run, but it seems we
>>need something kinda research here instead of re-run.
>>    
>>
>Exactly. By the way, Justin writes he did only 3 tests and calculated
>the average out of these 3. In statistics this is a very small sample.
>We would need at least 30 or so. If the results would have a big
>variance, they should be treated with exponential smoothening.
>And then we can go off with the calculations. Also It would be nice
>to have data from the exact tests made regularly to test for regressions
>and see what's the trend.
>  
>
I can just tell you from experience that benchmarks that take less than
a minute have a high tendency to be poor measures  He should increase
the size of the benchmark until each thing he measures takes more than 2
minutes.  If it is reproduceable it can be still meaningless.

Hans