Re: Linux Gazette benchmark Reiser 4

[Edward Shishkin <edward@namesys.com>]

Hans Reiser wrote:

>PFC wrote:
>
>  
>
>>    Hehe. Wow. Sure, a benchmark that runs in 0.03 seconds for the
>>fastest  one and 0.07 seconds for the slowest one looks pretty
>>reliable to me. How  much time does it take to spawn the "touch"
>>process 10k times ? Hm... I'd  guess most of the benchmark time ?
>>    
>>
>
>  
>

Let's consider this important aspect of benchmarking more carefully.
So there is an interesting question: how much should be a difference
in order to approve that some fs really wins at this statistics? Is
there any guarantee you won't get, say, 0.05 and 0.02 after next run?
Sorry, but I didn't find any answer in Justin's notes, NOTE5 (Tests
Performed) says that questionable tests were re-run, but it seems we
need something kinda research here instead of re-run.

Below are some comments for how this problem is resolved (1*) in mongo
benchmark. Look for example at this table:
http://www.namesys.com/benchmarks.html#mongo.2.6.11
Fractions like 0.982 (D/A), 1.017 (C/A) are in black color, it means
that we _can not_ do any assumptions about winner because
|1 - X/A| < 0.02. What the magic M = 0.02 is?
Let's run the same phase for the same settings (file system, file set,
etc..) 10 times. We will obtain for the same statistics X a set of
different (because of errors) values x1, x2, ..., x10. Suppose that
X has a normal distribution (any objections?). It means that we can
calculate its trusted interval for a single measurement (2*) as
[X - d(P), X + d(P)], where d(P) = D*U(P), D is dispersion and U(P)
should be found from the standard table by any nominated value of
trusted probability P (3*).
Now we have the following simple criterion (*4):

|A - X| >= 2d(P), i.e. |1 - X/A| >= 2D*U(P)/A

|           |<-d->|    |<-d->|
------<-----|----->----<-----|----->------
            A                X

The magic M = 0.02 for mongo benchmark was calculated as 2D*U(P)/A
for the trusted probability P=0.85 (5*).
Now it is clear from the formula above why statistics shouldn't be
too small: because the criterion becomes false. I am sure (and it
is easy to check) 2d(P=0.85) is much more then |0.07 - 0.03| as it
is in the case of find 10000 files. By the way, some settings, which
provide a small values (~5 sec) of the mongo STATS statistics also
make this criterion false.


(1*) Maybe this is not a perfect way, but it is better then nothing
(2*) For N measurements the expression for boundaries becomes a bit
     complicated.
(3*) For P=0.85 (as we can found in any scientific book) U(P)=1.44
(4*) One more assumption here about identical distributions of A and X
(5*) Actually D = max(D_create, D_copy, D_read, D_delete, D_dd), where
     D_each_phase was estimated once by 10 measurements with some fixed
     settings by the standard way:
     D^2 = ((x - x1)^2 + ... + (x - x10)^2)/(10 - 1), where
     x = (x1 + ... + x10)/10 is an average value.

Edward.