Re[2]: Linux Gazette benchmark Reiser 4

[Pysiak Satriani <pysiak.satriani@wp.pl>]

Hello Edward,

Sunday, January 8, 2006, 11:07:46 PM, you wrote:
> Let's consider this important aspect of benchmarking more carefully.
> So there is an interesting question: how much should be a difference
> in order to approve that some fs really wins at this statistics? Is
> there any guarantee you won't get, say, 0.05 and 0.02 after next run?
> Sorry, but I didn't find any answer in Justin's notes, NOTE5 (Tests
> Performed) says that questionable tests were re-run, but it seems we
> need something kinda research here instead of re-run.
Exactly. By the way, Justin writes he did only 3 tests and calculated
the average out of these 3. In statistics this is a very small sample.
We would need at least 30 or so. If the results would have a big
variance, they should be treated with exponential smoothening.
And then we can go off with the calculations. Also It would be nice
to have data from the exact tests made regularly to test for regressions
and see what's the trend.

> etc..) 10 times. We will obtain for the same statistics X a set of
> different (because of errors) values x1, x2, ..., x10. Suppose that
> X has a normal distribution (any objections?)
Well, for serious reasoning a smirnof-kolmogorov test should be used
or Shapiro-Wilk or any other that applies to check the normal
distribution. If it is, we can go ahead with all the calculations.
If not.. Well, I'm not enough of a statistics guru to say :-)

Regards,
Maciej